Handwritten Character Recognition by Alternately Trained Relaxation Convolutional Neural Network

深度学习 最新(2014)卷积神经网络的改进,用于中文字符的识别! State of art.

2014 14th International Conference on Frontiers in Handwriting Recognition

HandwrittenCharacterRecognitionby

AlternatelyTrainedRelaxationConvolutionalNeuralNetwork

ChunpengWu,WeiFan,YuanHe,JunSun,SatoshiNaoi

FujitsuResearch&DevelopmentCenterCo.Ltd.,Beijing,100025,China

{wuchunpeng,fanwei,heyuan,sunjun,naoi}@cn.fujitsu.com

Abstract—Deeplearningmethodshaverecentlyachievedimpressiveperformanceintheareaofvisualrecognitionandspeechrecognition.Inthispaper,weproposeahand-writingrecognitionmethodbasedonrelaxationconvolutionalneuralnetwork(R-CNN)andalternatelytrainedrelaxationconvolutionalneuralnetwork(ATR-CNN).PreviousmethodsregularizeCNNatfull-connectedlayerorspatial-poolinglayer,however,wefocusonconvolutionallayer.TherelaxationconvolutionlayeradoptedinourR-CNN,unliketraditionalconvolutionallayer,doesnotrequireneuronswithinafeaturemaptosharethesameconvolutionalkernel,endowingtheneuralnetworkwithmoreexpressivepower.Asrelaxationconvolutionsharplyincreasethetotalnumberofparameters,weadoptalternatetraininginATR-CNNtoregularizetheneuralnetworkduringtrainingprocedure.OurpreviousC-NNtookthe1stplaceinICDAR’13ChineseHandwritingCharacterRecognitionCompetition,whileourlatestATR-CNNoutperformsourpreviousoneandachievesthestate-of-the-artaccuracywithanerrorrateof3.94%,furthernarrowingthegapbetweenmachineandhumanobservers(3.87%).Keywords-handwrittencharacterrecognition;convolutionalneuralnetwork;relaxationconvolution;alternatetraining

I.INTRODUCTION

Despiteconsiderableeffortshavebeendevotedtohand-writingrecognition,itisstillachallengingproblemduetothepresenceofcursivewriting,touchingstrokes,andconfusioninshapes.Forhandwrittendigitrecognition,meth-odsbasedondifferentclassi ersareproposed,suchasKNN[1],[2],SVM[3]andBoosting[4].AstohandwrittenChinesecharacterrecognition,modi edquadraticdiscrimi-nantfunction(MQDF)anditsvariantsarethemainstreamclassi cationmethodsforthepastdecades[5],[6],[7].Hand-craftedfeatures(e.g.,8-directiongradient[8]),andfeatureselectionalgorithmsarekeycomponentsoftheabovemethods[9].Recently,deepneuralnetwork(DNN)hasbeenproposedforhandwrittenEnglishwordrecognitionbyautomaticfeaturelearning[10].DNNalsoyieldsthebestrecognitionaccuracyonhandwrittendigitdataset[11]andhandwrittenChinesecharacterdataset[12].

DNNsarebreakingrecordsandimprovingstate-of-the-artrecognitionperformanceformanydif cultvisualtasks.Bissaccoetal.[13]proposeaHoG-inputDNNforrecogniz-ingcharactersinnaturalscenes,andachievethebestperfor-manceinICDAR’13RobustReadingCompetition.Sermanetetal.[14]presentanintegratedframeworkforadopting

DNNforimageclassi cation,localizationanddetection,takingthe1stplaceinImageNet’13Competition.Wangetal.[15]trainastackedauto-encodertolearngenericimagefeaturesforvideotracking,achievingthestate-of-the-artonsomehardbenchmarkvideosequences.Ciresanetal.[16]areseveralchampionsofneuronmembranesegmentation,traf csignclassi cation,handwritingrecognitionandetc.,basedontheirmulti-columnDNNs(MCDNN).TheaboveencouragingprogressofDNNcanbeattributedtofactorsasfollows:(1)Highperformancecomputingtechniques(e.g.,CPUclustersandGPUfarms);(2)Flexiblestructureofneuralnetworks(e.g.Lin.[17]);(3)Availabilityoflargertrainingdatasets;(4)Effectivelearningalgorithm,especiallymodelregularizationmethod(e.g.,dropout[18]).

Asigni cantdifferencebetweenpreviousneuralnetworksandcurrentDNNsisthescaleofthemodel.Recentsuccessonvariousvisualandacoustictasksshowthatscalingupneuralnetworkscoulddramaticallyimprovetherecognitionperformance[19].However,twoproblemsbroughtbythescalingupareslowconvergenceandover- tting.

Toacceleratethetrainingconvergence,itispopulartotrainDNNswithunsupervisedlayer-wisepre-trainingfol-lowedbyback-propagation ne-tuning,accordingtoHintonetal.’sbreakthrough[20].Butmorerecentexperiments[21],[22]showthatunsupervisedpre-trainingmaybelessneces-sarywhenmoretrainingdataandcarefulweightinitializa-tionareavailable[23],[24].Inpractice,highperformancecomputingtechniques,i.e.,CPUclustersandGPUfarms,havemadeitpossibletotrainlarge-scaleneuralnetworkswithstochasticgradientdescent(SGD)inseveraldays,insteadofmonthsorevenyears[25],[26],[27].ButbecauseofSGD’sinherentproblem—largevariance[28],themulti-coreCPUandGPUwillnotsharplyspeedupthetrainingprocedureunlesstheDNNstructure(e.g.,convolutionallayerandspatial-poolinglayer)andthescaleoftrainingdatasetareappropriatefortherelatedtask[19],[23].

Astothesecondproblem,over- tting,large-scaleneuralnetworkswithmillionsorbillionsofparametersarepronetoover- tting,especiallywhentheyaretrainedonsmalltrainingsets.ThereforeregularizingDNNshasbecomeapopularresearchsubject[29].Dropout[18]randomlydropsactivations,anddropconnect[11]randomlydropsweightsaretwotypicalstochasticregularizingtechniques.These