文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > Contextual Recognition of Head Gestures
侵权投诉

Contextual Recognition of Head Gestures

Head pose and gesture offer several key conversational grounding cues and are used extensively in face-to-face interaction among people. We investigate how dialog context from an embodied conversational agent (ECA) can improve visual recognition of user ge

ContextualRecognitionofHeadGestures

Louis-PhilippeMorency

CandaceSidner

ChristopherLee

TrevorDarrell

ComputerSciencesandArti cialIntelligenceLaboratory

MassachussettsInstituteofTechnology

Cambridge,MA02139,USA

{lmorency,trevor}@csail.mit.edu

ABSTRACT

Headposeandgestureofferseveralkeyconversationalground-ingcuesandareusedextensivelyinface-to-faceinteractionamongpeople.Weinvestigatehowdialogcontextfromanembodiedcon-versationalagent(ECA)canimprovevisualrecognitionofusergestures.Wepresentarecognitionframeworkwhich(1)extractscontextualfeaturesfromanECA’sdialogmanager,(2)computesapredictionofheadnodandheadshakes,and(3)integratesthecon-textualpredictionswiththevisualobservationofavision-basedheadgesturerecognizer.Wefoundasubsetoflexical,punctuationandtimingfeaturesthatareeasilyavailableinmostECAarchitec-turesandcanbeusedtolearnhowtopredictuserfeedback.Usingadiscriminativeapproachtocontextualpredictionandmulti-modalintegration,wewereabletoimprovetheperformanceofheadges-turedetectionevenwhenthetopicofthetestsetwassigni cantlydifferentthanthetrainingset.

1.INTRODUCTION

Duringface-to-faceconversation,peopleusevisualfeedbacktocommunicaterelevantinformationandtosynchronizerhythmbe-tweenparticipants.Agoodexampleofnonverbalfeedbackisheadnoddinganditsuseforvisualgrounding,turn-takingandanswer-ingyes/noquestions.Whenrecognizingvisualfeedback,peopleusemorethantheirvisualperception.Knowledgeaboutthecur-renttopicandexpectationsfrompreviousutteranceshelpguideourvisualperceptioninrecognizingnonverbalcues.Ourgoalistoequipanembodiedconversationalagent(ECA)withtheabilitytousecontextualinformationforperformingvisualfeedbackrecog-nitionmuchinthesamewaypeopledo.

Inthelastdecade,manyECAshavebeendevelopedforface-to-faceinteraction.Akeycomponentofthesesystemsisthedialoguemanager,usuallyconsistingofahistoryofthepastevents,thecur-rentstate,andanagendaoffutureactions.Thedialoguemanagerusescontextualinformationtodecidewhichverbalornonverbalactiontheagentshouldperformnext.Thisiscalledcontext-basedsynthesis.

Contextualinformationhasprovenusefulforaidingspeechrecog-nition[9].In[9],thegrammarofthespeechrecognizerdynam-

Permissiontomakedigitalorhardcopiesofallorpartofthisworkforpersonalorclassroomuseisgrantedwithoutfeeprovidedthatcopiesarenotmadeordistributedforpro torcommercialadvantageandthatcopiesbearthisnoticeandthefullcitationonthe rstpage.Tocopyotherwise,torepublish,topostonserversortoredistributetolists,requirespriorspeci cpermissionand/orafee.

ICMI’05,October4–6,2005,Trento,Italy.

Copyright2005ACM1-59593-028-0/05/0010...$

Contextual Recognition of Head Gestures

Contextual Recognition of Head Gestures

Contextual Recognition of Head Gestures

5.00.

MitsubishiElectricResearchLaboratories(MERL)

Cambridge,MA02139,USA{sidner,lee}@merl.com

Figure1:Contextualrecognitionofheadgesturesduringface-to-faceinteractionwithanembodiedagent.

icallychangesdependingontheagent’spreviousactionorutter-ance.Inasimilarfashion,wewanttodevelopacontext-basedvisualrecognitionmodulethatbuildsuponthecontextualinforma-tionavailableinthedialoguemanagertoimproveperformance.Theuseofdialoguecontextforvisualgesturerecognitionhas,toourknowledge,notbeenexploredbeforeforconversationalinter-action.Inthispaperwepresentapredictionframeworkforincorpo-ratingdialoguecontextwithvision-basedheadgesturerecognition.ThecontextualfeaturesarederivedfromtheutterancesoftheECA,whichisreadilyavailablefromthedialoguemanager.Wehighlightthreetypesofcontextualfeatures:lexical,punctuation,andtiming,andselectedasubsetforourexperimentthatweretopicindepen-dent.Weuseadiscriminativeapproachtopredictheadnodsandheadshakesfromasmallsetofrecordedinteractions.Wethencombinethecontextualpredictionswithavision-basedrecognitionalgorithmbasedonthefrequencypatternoftheuser’sheadmotion.Ourcontext-basedrecognitionframeworkallowsustopredict,forexample,thatincertaincontextsaglanceisnotlikelywhereasaheadshakeornodis(asinFigure1),orthataheadnodisnotlikelyandaheadnodmisperceivedbythevisionsystemcanbeignored.

ThefollowingsectiondescribesrelatedworkongestureswithECAs.Section3describesthecontextualinformationavailableinmostembodiedagentarchitectures.Section4showshowweautomaticallyextractasubsetofthiscontexttocomputelexical,punctuation,andtimingfeatures.Section5demonstrateshowweusethecontextualfeaturestopredictheadnodsandheadshakes.Section6describeshowweintegratethecontextualpredictionwiththeresultsfromavision-onlyheadgesturerecognizer.Finally,wedescribeourexperiments,performedon16videorecordingsofhu-manparticipantsinteractingwitharobot.

上一页第3页

你可能喜欢

热门文档

相关文档

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email