SYNTHESISDISTORTIONESTIMATIONIN3DVIDEOUSINGFREQUENCYANDSPATIALANALYSIS
LuFang
Ngai-ManCheung
DongTian AnthonyVetro
HuifangSun LuYu
Univ.ofSci.andTech.ofChinaSingaporeUniv.ofTech.andDesignMitsubishiElectricResearchLabsZhejiangUniv.
ABSTRACT
Weproposeananalyticalmodeltoestimatethesynthesizedviewqualityin3Dvideo.Speci cally,weestimatethedepth-errorinduceddistortionusinganapproachthatcombinesfrequencyandspatialdomainanalysis.Wealsoproposetodecomposethespatial-variantvideosignalsintogradient-basedrepresentationstocapturetheinteractionbetweenimagegradients,deptherrorsandsynthesisdistortion.Experimentresultswithvideosequencesandcoding/renderingtoolsusedinMPEG3DVactivitiesshowthatouranalyticalmodelcanaccuratelyestimatethesynthesisnoisepower.IndexTerms—3DVideo,SynthesisDistortionEstimation,FrequencyandSpatialAnalysis,DepthError
I.INTRODUCTION
3Dvideo(3DV)hasattractedmuchattentionrecently[1]–[5].3Ddatasetsusuallyconsistofmultiplevideosequences(texturedata)capturedbycamerasatdifferentpositions,alongwiththeassociateddepthimages.Thequalityofthesynthesizedviewisimperativein3DVapplications[6],[7].Thesynthesisquality,however,dependsonseveralfactorsandcomplicatedinteractionsbetweenthem.Inparticular,textureanddepthimagesmaycontainerrorsduetoimperfectsensingorlossycompression[8],[9],anditisnotclearhowtheseerrorsinteractandaffecttherenderingquality.Unliketextureerrors,whichcausedistortionintheluminance/chrominancelevel,deptherrorscausepositionerrorsinsynthesis[10],i.e.,pixelsarewarpedtoslightlyshiftedpositionsduringsynthesis.Theeffectofdeptherrorsisverysubtle.Forinstance,theimpactofdeptherrorswouldvarywiththeimagecontents,andimageswithlesstexturestendtobemoreresilienttothedeptherrors.
Anaccurateanalyticalmodeltoestimatethesynthesisqualityisveryvaluableforthedesignof3DVsystems.NguyenandDo[11]analyzedtherenderingqualityofimage-basedrendering(IBR)algorithmsandusedTaylorseriesexpansiontoderivetheupperboundofthemeanabsoluteerror(MAE)inthesynthesisoutput.Liuetal.[12]approximatederrorsduetodepthmapartifactsusingalinearmodelofaveragemagnitudeofmean-squareddisparityerrorsoveranentireframeandamotionsensitivityfactorcomputedfromtheenergydensity.AnautoregressivemodelwasproposedbyKimetal.[13]toestimatethesynthesisdistortionattheblocklevelandwasshowntobeeffectiveforrate-distortionoptimizedmodeselection.AdistortionmodelasafunctionoftheviewlocationwasalsoproposedbyVelisavljevicetal.[14]forbitallocation.Takahashi[15]proposedanoptimizedviewinterpolationschemebasedonfrequencydomainanalysisofdepthmaperror.
Inourpreviouswork[16],weproposedtoestimatethesynthesisqualityusingpowerspectraldensity(PSD).Thisassumedthattheunderlyingvideoframesignalsarespatialinvariant.However,signalsalongstrongtextureedgeschangemuchquicklythanthoseinthenon-edgeregions,i.e.,autocovariancefunctiondecaysmuch
fasterinedgepixels.Differentfrompreviousapproaches,weproposeinthisworkamodelthatcombinesfrequencyanalysisandspatialanalysistoaccountforthenon-stationaryinthesignals.Speci cally,ourcontributionsare:1)Weproposeananalyticalmodelcombiningfrequencyandspatialdomainanalysistoestimatethesynthesisviewquality,givendepthmaperrors,textureimagecharacteristics(smoothortextural),textureimagequalityandthecameracon gurationastheinputs;2)Inparticular,weproposetodecomposethespatialvariantsignalsintogradient-basedrep-resentationstofacilitateanalysis.Theanalysisresultsshowthatlinearapproximationsofthespatialvariantsignalscanleadtoacomputationally-ef cientandyetaccurateestimation;3)WeverifyourmodelwithsubstantialexperimentsusingvideosequencesandcodingandrenderingtoolsfromtheMPEG3DVactivities[17],[18].Therestofthepaperisasfollows.SectionsIIdiscussesouranalyticalmodel.SectionIIIpresentsexperimentresultsandSectionIVconcludesthepaper.
II.ESTIMATENOISEPOWERDUETODEPTHCODINGWe rstdiscussourviewsynthesismodel,whichconsistsofframewarpingfollowedbyblending.Tworeferencetextureframescapturedbytheleftandrightcameras(denotedbyXl(m,n)andXr(m,n)respectively)alongwiththeirassociateddepthimages(denotedbyDl(m,n)andDr(m,n)respectively)areusedtogeneratethesynthesizedframeU(m,n)atacertainvirtualcameraposition.First,inframewarping,pixelsarecopiedfromXltoformanintermediateframeUl,fromposition(m′,n)to(m,n).Weassumethecamerasarerecti edandarrangedlinearly,andthereexistsonlyhorizontaldisparitygivenbym m′determinedbythedepthimages,cameraparametersandcameradistance[16].Likewise,pixelsarecopiedfromXrtoformtheintermediateframeUr.Then,UlandUraremerged(blended)togenerateU.Weassumemergingbylinearcombination:U(m,n)=αUl(m,n)+(1 α)Ur(m,n).Heretheweightαisdeterminedbythedistancesbetweenthevirtualcamerapositionandtheleft/rightreferencecamerapositions.
Inpractice,thetextureanddepthimagesarelossyencoded.Weassumethatwhenthereconstructedtexture/depthimages l,X r,D l,D r)arefedintothesynthesispipeline,weobtain(X
renderingoutputW.LetV=U Wbethenoiseintherenderingoutputduetocodingerrorsintexture/depthimages.In[16],weshowthatunderreasonableassumptionsthetotalsynthesisnoisepower(E[V2])canbeestimatedbysummingtwocomponents:oneisthesynthesisnoisepowerduetotextureimagecoding(E[N2]),theotheristhesynthesisnoisepowerduetodepthimagecoding(E[Z2]),i.e.,
E[V2]=E[N2]+E[Z2].(1)WediscussedtheestimationofE[N2]in[16]andthefocusinthis
paperisontheestimationofE[Z2].
热门文档