Google, Yahoo and MSN all provide both web user interfaces (WUIs) and application programming interfaces (APIs) to their collections. Whether building collections of resources or studying the search engines themselves, the search engines request that resea
AgreeingtoDisagree:SearchEnginesandTheir
PublicInterfaces
OldFrankDominionMcCown
UniversityComputerScienceDepartmentNorfolk,Virginia,USA23529
fmccown@cs.odu.edu
ABSTRACT
Google,YahooandMSNallprovidebothwebuserinterfaces(WUIs)andapplicationprogramminginterfaces(APIs)totheircollections.Whetherbuildingcollectionsofresourcesorstudyingthesearchenginesthemselves,thesearchenginesrequestthatresearchersusetheirAPIsandnot“scrape”theWUIs.However,anecdotalevidencesuggeststheinterfacesproducedi erentresults.Weprovidethe rstindepthquan-titativeanalysisoftheresultsproducedbytheGoogle,MSNandYahooAPIandWUIinterfaces.Wehavequeriedbothinterfacesfor vemonthsandfoundsigni cantdiscrepanciesbetweentheinterfacesinseveralcategories.Ingeneral,wefoundMSNtoproducethemostconsistentresultsbetweentheirtwointerfaces.Our ndingssuggestthattheAPIin-dexesarenotolder,buttheyareprobablysmallerforGoogleandYahoo.Wealsoexaminedhowsearchresultsdecayovertimeandbuiltpredictivemodelsbasedontheobservedde-cayrates.Basedonour ndings,itcantakeoverayearforhalfofthetop10resultstoapopularquerytobereplacedinGoogleandYahoo;forMSNitmaytakeonly2-3months.CategoriesandSubjectDescriptors:H.3.5[Informa-tionStorageandRetrieval]OnlineInformationServices:Web-basedservices
GeneralTerms:Experimentation,Measurement
Keywords:API,distancemeasurement,searchenginere-sults,searchengineinterfaces
1.INTRODUCTION
Commercialsearchengineshavelongbeenusedinacad-emicstudies.Sometimesthesearchenginesthemselvesarebeingstudied,andsometimestheyareusedtostudytheWeborwebphenomena.Inthepast,researchershaveei-thermanuallysubmittedqueriestothesearchengine’swebuserinterface(WUI),ortheyhavecreatedprogramsthatautomatethetaskofsubmittingqueries.Thereturnedre-
Thisistheauthor’sversionofthework.ItispostedherebypermissionofACMforyourpersonaluse.Notforredistribution.
JCDL’07,June17–22,2007,Vancouver,BritishColumbia,Canada.Copyright2007ACM978-1-59593-644-8/07/0006...$5.00.
OldMichaelDominionL.UniversityNelson
ComputerScienceDepartmentNorfolk,Virginia,USA23529
mln@cs.odu.edu
sultshavebeenprocessedmanuallyorbyprogramsthatrelyonbrittlescreen-scrapingtechniques.
Butdatacollectionmechanismsforsearchengineshavechangedinthepastfewyears.Threeofthemostpopularcommercialsearchengines(Google,MSNandYahoo)havedevelopedfreelyavailableAPIsforaccessingtheirindex,andresearcherscannowusetheseAPIsintheirautomateddatacollectionprocesses.Unfortunately,theAPIsdonotalwaysgivethesameresultsastheWUI.Thelistservesandnews-groupsthatcatertotheAPIcommunitiesarefullofques-tionsregardingtheperceiveddi erencesinresultsbetweenthetwo.Noneofthesearchenginespubliclydisclosethein-nerworkingsoftheirAPIs,sousersareleftwonderingiftheAPIsaregivingsecond-ratedata.Thisanecdotalevidencehasledsomeresearcherstoquestionthevalidityoftheir ndings.Forexample,Bar-YossefandGurevich[8]statethat“duetolegallimitationsonautomaticqueries,weusedtheGoogle,MSN,andYahoo!websearchAPIs,whichare,reportedly,servedfromolderandsmallerindicesthantheindicesusedtoservehumanusers.”Otherresearchersmaybetotallyunawareofthedi erences.Whenwritingabouta2004experimentusingtheGooglesearchengine,Thelwall[47]statedthat“theGoogleAPI...couldhavebeenusedtoautomatetheinitialcollectionoftheresultspages,whichshouldhavegiventhesameoutcome”asusingtheWUI.Themainpurposeofthisstudyistoexaminethedi er-encesbetweenwhatisreportedbetweentheWUIandAPIwhenqueriedwithavarietyofqueriesthatresearchersandusersfrequentlyuse.Our ndingsallowustoaddressthequestionofwhethertheAPIsarepullingfromolderandsmallerindexes.Asecondarypurposeistoexaminehowsearchresultsdecayovertimeandprovidepredictivemod-elsfordeterminingthehalf-livesofsearchresults.
2.BACKGROUNDANDRELATEDWORK2.1
SearchEngineAPIs
Inearly2002,Googlegeneratedagreatdealoffanfarewhenitbecamethe rstsearchenginetoreleaseafreeSOAP-basedAPIforaccessingitsindex.Googlerequireduserstoregisterforalicensekeythatallowed1000requeststobeissuedperday.YahooandMSNreleasedpublicAPIsinearlyandlate2005,respectively,butinane orttoout-doGoogle,theygreatlyreducedtheusagerestrictionsoftheirAPIs.Yahoo’sREST-basedAPIallows5000dailyrequestsperIPaddress,perapplicationID[51].MSN’sSOAPAPIallows10,000dailyrequestsperIPaddress,perapplication
Agreeing to Disagree_经济学_高等教育_教育专区。2005年诺贝尔经济学奖获奖论文英文版今日推荐 78份文档 百度图片明星相册 星光上线...
Agreeing to disagree Sea... 暂无评价 10页 免费 Public perception of pri...Some search sites claim to be Thai search engines, but their results are ...
erent access methods and interfaces. A data ...c search engines or meta-search engines (such ...The two agents can still disagree on the ...
erent access methods and interfaces. A data ...c search engines or meta-search engines (such ...The two agents can still disagree on the ...
热门文档