文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > Integrating POMDP and reinforcement learning for a two layer simulated robot architecture
侵权投诉

Integrating POMDP and reinforcement learning for a two layer simulated robot architecture

Two layer control systems are common in robot architectures. The lower level is designed to provide fast, fine grained control while the higher level plans longer term sequences of actions to achieve some goal. Our approach uses reinforcement learning (RL)

IntegratingPOMDPandReinforcementLearningforaTwo

LayerSimulatedRobotArchitecture

LarryD.Pyeatt

AdeleE.Howe

ComputerScienceDepartmentColoradoStateUniversityFortCollins,CO80523

pyeatt,howe@cs.colostate.edudoc.guandang.net pyeatt,howe

Abstract

Twolayercontrolsystemsarecommoninrobotarchitectures.Thelowerlevelisdesignedtoprovidefast, negrainedcontrolwhilethehigherlevelplanslongertermsequencesofactionstoachievesomegoal.Ourapproachusesreinforcementlearning(RL)forthelowlevelandPartiallyObservableMarkovDecisionProcess(POMDP)planningforthehighlevel.Becausebothlevelscanadapttheirbe-haviorwithinthescopeoftheirtasks,thecombinationisexpectedtoberobusttodegradationsinsensorandactuatorfailuresandsotoen-hanceoverallsystemreliability.WeimplementedourarchitectureforuseintheKheperarobotsimulator.Inasetofexperiments,weshowthatgoodperformancecanbedif culttoachievewithhandcodedlowlevelcontrolandthatperformanceofourRL/POMDPsystemdegradesslowlywithincreasingsensorandactuatorfailure.

Thetwolayercontrolarchitectureoffersreliability.Forexample,autonomousrobotsmustbeabletodealwithfailureofsensorsandactuatorsinarobustwaywhentheymaybeoperatingoutofdirecthumancontrol(e.g.,spaceexploration,hazardousenvironments).Inthesesituations,hardwarefailurecanresultinfailureofthemis-sionunlessthecontrolsystemissuf cientlyadaptive.Inlesscriticalsituations,robustoperationisalsonecessary.Evenanof cerobotshouldbecapableofdealingwithsomesensorfailuresormiscali-brationsandactuatorfailures.

Wewishtodeveloparobustcontrolsystemforanautonomousrobotthatcanoperateinanof ce-likeenvironment.Byrobust,wemeanthattherobotshouldbeabletodealwithsensorandactuatormal-functioncausedbyhardwarefailureorlowbatteryconditions.Ourcurrentsystemhastwolevelsofcontrol.Thelowerlevelcontrolstheactuatorsthatmovetherobotaroundandprovidesasetofbe-haviorsthatcanbeusedbythehigherlevelcontrolandplanningsystem.Theupperlevelplansasequenceofbehaviorstoruninor-dertomovetherobotfromitscurrentlocationtothegoal.Inourarchitecture,thebottomlevelisaccomplishedbyreinforce-mentlearning(RL);thetoplevelisaPartiallyObservableMarkovDecisionProcess(POMDP)planner.ItisthiscombinationofRLLearningandPOMDPthatdistinguishesourapproachfromprevi-ouswork.WebelievethatRLisagoodchoiceforimplementinglowlevelcontrolbecauseitisabletolearnonlineandcanadapttochangesintheenvironment.RLsimultaneouslylearnsthevaluefunctionandthepolicy,reducingtheinterventionoftheprogram-merandaccommodatingfurtherlearningasenvironmentconditionschange.POMDPplanningisagoodframeworkforthehigherlevelcontrolbecauseitoperatesquicklyonceapolicyisgeneratedandcanprovidethereinforcementneededbythelowerlevelbehaviors.WehavebeentestingourarchitectureontheKheperarobotsimu-lator[14].Kheperaisaverysmallrobotwithlimitedsensorsandawell-de nedenvironment.Thismakessimulationanattractiveop-tionfortestingourideas.Thesimulatorcanberunmuchfasterthanreal-timeanddoesnotrequirehumaninterventionforlowbatteryconditionsandsensorfailuresasaphysicalrobotcould.

1Introduction

Twolayercontrolsystemsarecommoninrobotarchitectures.Twolayercontrolisapragmaticsolutiontodifferenttimeandcontrolgranularitiesofactivity.Thelowerlayerprovidesfast,shorthori-zondecisionaboutquicklyexecutedactions;theupperlayeradoptsamoregoalorientedviewandplansoveralongerscope.Thus,thelowerlayerisdesignedtokeeptherobotoutoftrouble(e.g.,run-ningintoobstacles)andmakeminormodi cationstoanotherwisegoodcourseofactionbasedonsensoryfeedback(e.g.,coursecor-rections).Theupperlayerensuresthattherobotcontinuallyworkstowarditstargettaskorgoal(e.g.,doesnotpaintitselfintoacornerorgoincircles).

第1页

你可能喜欢

热门文档

相关文档

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email