文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > Estimating divergence functionals and the likelihoodratio by penalized convex risk minimiza
侵权投诉

Estimating divergence functionals and the likelihoodratio by penalized convex risk minimiza

Abstract We develop and analyze an algorithm for nonparametric estimation of divergencefunctionals and the density ratio of two probability distributions. Our method is based on a variational characterization of f-divergences, which turns the estima-tion i

Estimatingdivergencefunctionalsandthelikelihood

ratiobypenalizedconvexriskminimizationXuanLongNguyen

SAMSI&DukeUniversityMartinJ.WainwrightUCBerkeleyMichaelI.JordanUCBerkeley

Abstract

Wedevelopandanalyzeanalgorithmfornonparametricestimationofdivergencefunctionalsandthedensityratiooftwoprobabilitydistributions.Ourmethodisbasedonavariationalcharacterizationoff-divergences,whichturnstheestima-tionintoapenalizedconvexriskminimizationproblem.Wepresentaderivationofourkernel-basedestimationalgorithmandananalysisofconvergenceratesfortheestimator.Oursimulationresultsdemonstratetheconvergencebehaviorofthemethod,whichcomparesfavorablywithexistingmethodsintheliterature.

1Introduction

Animportantclassof“distances”betweenmultivariateprobabilitydistributionsPandQaretheAli-Silveyorf-divergences[1,6].Thesedivergences,tobede nedformallyinthesequel,areallofthe formDφ(P,Q)=φ(dQ/dP)dP,whereφisaconvexfunctionofthelikelihoodratio.Thisfamily,includingtheKullback-Leibler(KL)divergenceandthevariationaldistanceasspecialcases,playsanimportantroleinvariouslearningproblems,includingclassi cation,dimensionalityreduction,featureselectionandindependentcomponentanalysis.Foralloftheseproblems,iff-divergencesaretobeusedascriteriaofmerit,onehastobeabletoestimatethemef cientlyfromdata.Withthismotivation,thefocusofpaperistheproblemofestimatinganf-divergencebasedoni.i.d.samplesfromeachofthedistributionsPandQ.Ourstartingpointisavariationalcharacterizationoff-divergences,whichallowsourproblemtobetackledviaanM-estimationprocedure.Speci -cally,thelikelihoodratiofunctiondP/dQandthedivergencefunctionalDφ(P,Q)canbeestimatedbysolvingaconvexminimizationproblemoverafunctionclass.Inthispaper,weestimatethelike-lihoodratioandtheKLdivergencebyoptimizingapenalizedconvexrisk.Inparticular,werestricttheestimatetoaboundedsubsetofareproducingkernelHilbertSpace(RKHS)[17].TheRKHSissuf cientlyrichformanyapplications,andalsoallowsforcomputationallyef cientoptimizationprocedures.Theresultingestimatorisnonparametric,inthatitentailsnostrongassumptionsontheformofPandQ,exceptthatthelikelihoodratiofunctionisassumedtobelongtotheRKHS.Thebulkofthispaperisdevotedtothederivationofthealgorithm,andatheoreticalanalysisoftheperformanceofourestimator.Thekeytoouranalysisisabasicinequalityrelatingaperformancemetric(theHellingerdistance)ofourestimatortothesupremaoftwoempiricalprocesses(withrespecttoPandQ)de nedonafunctionclassofdensityratios.ConvergenceratesarethenobtainedusingtechniquesforanalyzingnonparametricM-estimatorsfromempiricalprocesstheory[20].Relatedwork.Thevariationalrepresentationofdivergenceshasbeenderivedindependentlyandexploitedbyseveralauthors[5,11,14].BroniatowskiandKeziou[5]studiedtestingandestimationproblemsbasedondualrepresentationsoff-divergences,butworkinginaparametricsettingasop-posedtothenonparametricframeworkconsideredhere.Nguyenetal.[14]establishedaone-to-onecorrespondencebetweenthefamilyoff-divergencesandthefamilyofsurrogatelossfunctions[2],throughwhichthe(optimum)“surrogaterisk”isequaltothenegativeofanassociatedf-divergence.Anotherlinkistotheproblemofestimatingintegralfunctionalsofasingledensity,withtheShan-nonentropybeingawell-knownexample,whichhasbeenstudiedextensivelydatingbacktoearly

第1页

你可能喜欢

热门文档

相关文档

  • XuanLong Nguyen

    Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization XuanLong Nguyen SAMSI & Duke University Martin J. Wainwright ...

  • XuanLong Nguyen

    Estimating divergence functionals and the likelihood ratio by penalized convex risk minimization XuanLong Nguyen SAMSI & Duke University Martin J. Wainwright ...

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email