Two layer control systems are common in robot architectures. The lower level is designed to provide fast, fine grained control while the higher level plans longer term sequences of actions to achieve some goal. Our approach uses reinforcement learning (RL)
IntegratingPOMDPandReinforcementLearningforaTwo
LayerSimulatedRobotArchitecture
LarryD.Pyeatt
AdeleE.Howe
ComputerScienceDepartmentColoradoStateUniversityFortCollins,CO80523
pyeatt,howe@cs.colostate.edudoc.guandang.net pyeatt,howe
Abstract
Twolayercontrolsystemsarecommoninrobotarchitectures.Thelowerlevelisdesignedtoprovidefast, negrainedcontrolwhilethehigherlevelplanslongertermsequencesofactionstoachievesomegoal.Ourapproachusesreinforcementlearning(RL)forthelowlevelandPartiallyObservableMarkovDecisionProcess(POMDP)planningforthehighlevel.Becausebothlevelscanadapttheirbe-haviorwithinthescopeoftheirtasks,thecombinationisexpectedtoberobusttodegradationsinsensorandactuatorfailuresandsotoen-hanceoverallsystemreliability.WeimplementedourarchitectureforuseintheKheperarobotsimulator.Inasetofexperiments,weshowthatgoodperformancecanbedif culttoachievewithhandcodedlowlevelcontrolandthatperformanceofourRL/POMDPsystemdegradesslowlywithincreasingsensorandactuatorfailure.
Thetwolayercontrolarchitectureoffersreliability.Forexample,autonomousrobotsmustbeabletodealwithfailureofsensorsandactuatorsinarobustwaywhentheymaybeoperatingoutofdirecthumancontrol(e.g.,spaceexploration,hazardousenvironments).Inthesesituations,hardwarefailurecanresultinfailureofthemis-sionunlessthecontrolsystemissuf cientlyadaptive.Inlesscriticalsituations,robustoperationisalsonecessary.Evenanof cerobotshouldbecapableofdealingwithsomesensorfailuresormiscali-brationsandactuatorfailures.
Wewishtodeveloparobustcontrolsystemforanautonomousrobotthatcanoperateinanof ce-likeenvironment.Byrobust,wemeanthattherobotshouldbeabletodealwithsensorandactuatormal-functioncausedbyhardwarefailureorlowbatteryconditions.Ourcurrentsystemhastwolevelsofcontrol.Thelowerlevelcontrolstheactuatorsthatmovetherobotaroundandprovidesasetofbe-haviorsthatcanbeusedbythehigherlevelcontrolandplanningsystem.Theupperlevelplansasequenceofbehaviorstoruninor-dertomovetherobotfromitscurrentlocationtothegoal.Inourarchitecture,thebottomlevelisaccomplishedbyreinforce-mentlearning(RL);thetoplevelisaPartiallyObservableMarkovDecisionProcess(POMDP)planner.ItisthiscombinationofRLLearningandPOMDPthatdistinguishesourapproachfromprevi-ouswork.WebelievethatRLisagoodchoiceforimplementinglowlevelcontrolbecauseitisabletolearnonlineandcanadapttochangesintheenvironment.RLsimultaneouslylearnsthevaluefunctionandthepolicy,reducingtheinterventionoftheprogram-merandaccommodatingfurtherlearningasenvironmentconditionschange.POMDPplanningisagoodframeworkforthehigherlevelcontrolbecauseitoperatesquicklyonceapolicyisgeneratedandcanprovidethereinforcementneededbythelowerlevelbehaviors.WehavebeentestingourarchitectureontheKheperarobotsimu-lator[14].Kheperaisaverysmallrobotwithlimitedsensorsandawell-de nedenvironment.Thismakessimulationanattractiveop-tionfortestingourideas.Thesimulatorcanberunmuchfasterthanreal-timeanddoesnotrequirehumaninterventionforlowbatteryconditionsandsensorfailuresasaphysicalrobotcould.
1Introduction
Twolayercontrolsystemsarecommoninrobotarchitectures.Twolayercontrolisapragmaticsolutiontodifferenttimeandcontrolgranularitiesofactivity.Thelowerlayerprovidesfast,shorthori-zondecisionaboutquicklyexecutedactions;theupperlayeradoptsamoregoalorientedviewandplansoveralongerscope.Thus,thelowerlayerisdesignedtokeeptherobotoutoftrouble(e.g.,run-ningintoobstacles)andmakeminormodi cationstoanotherwisegoodcourseofactionbasedonsensoryfeedback(e.g.,coursecor-rections).Theupperlayerensuresthattherobotcontinuallyworkstowarditstargettaskorgoal(e.g.,doesnotpaintitselfintoacornerorgoincircles).
China Towards Direct Policy Search Reinforcement Learning for Robot Control ...In this preliminary work, we demonstrate its feasibility with simulated ...
We demonstrate a simulated robot in a three-dimensional, texture-mapped ...The credit assignment problem associated with reinforcement learning is greatly ...
DESCENT LEARNING RATE Reinforcement learning with light-seeking robot Kary Fr?...In robotics, experiments are also longer than simulated ones, so learning ...
Reinforcement Learning in Large State Spaces Simulated Robotic Soccer as a testbed_专业资料。Abstract. Large state spaces and incomplete information are two ...
Abstract: Reinforcement learning methods are useful for robot learning, but ... enabling the robots to perform run-time trials in a simulated world, ...
Poincare-map-based reinforcement learning for biped walking_专业资料。Abstract...We applied this approach to both a simulated robot model and an actual ...
Reinforcement Learning in Large State Spaces Simulated Robotic Soccer as a testbed Abstract. Bayesian networks (BNs) are a compact representation of a ...
Poincare-map-based reinforcement learning for biped walking_专业资料。Abstract...We applied this approach to both a simulated robot model and an actual ...
热门文档