Table 1. Query List
Query Number
Q1 Q2 Q3 Q4 Q5
This paper is organized as follows: Section 2 describes the methodology employed to evaluate
Query Number
Query
network switch computer engineer fashion evolution baby sitter
Query
air conditioner Q6
nightmare Q7
atom Q8
reincarnation Q9
myopic Q10
After each run of the query or PwQ, the first 2
twenty documents retrieved were evaluated using search engines in terms of precision and normalized recall, Section 3 reports and discusses the experimental findings and the last section concludes the paper.
2. Methodology
Initially, three keyword-based search engines, namely, Google, Yahoo and Msn, and a semantic search engine Hakia were selected. Afterwards, ten queries that contain various topics and consist of one
or two terms1
were randomly determined as shown in Table 1. (Note that for some queries, Hakia displays categorized documents in its retrieval output before displaying web results. In order to have compatible retrieval outputs for search engine evaluations, during query selection process, particularly those queries were used that provide web results in the retrieval output, without the categorized documents, when run on Hakia).
Ten queries were run on, both, the selected keyword-based search engines as well as the semantic search engine. Additionally, four phrases, with different syntax but similar meanings, were used with each query and run, one by one, on the semantic search engine Hakia. The phrase-with-query (PwQ) forms were as follows: (1) “what is <query>”; (2) “information about <query>”; (3) “<query> definition”; and (4) “description of <query>”.
In order to have realistic results:
Only keywords were used on keyword-based
search engines since, in general, the users do not tend to use phrases (as observed in the most frequently used queries list of Wordtracker [11]).
Beside the keywords, phrases were used for
Hakia since one of the main features of Hakia, being a semantic search engine, is the use of phrases.
1
Jansen and Spink examined a transaction log of Dogpile in their
study [10] and they have revealed that 70% of users used queries that consist of two terms or less.
binary human relevance judgment and with this, every document was classified as “relevant” or “non-relevant”. Total 1600 documents were evaluated by the same author and in order to have stable performance measurement of search engines, all the searches and evaluations were performed in minimal non-distant time slots. While evaluating the retrieved documents following criteria were used: (1) Documents that contain any explanation about the searched query were considered “relevant”; (2) documents having same content but originating from different web addresses (i.e., mirror pages) were classified to be different [12]; (3) in case of duplicated documents, the first document that was retrieved was considered in the evaluation process, whereas its duplicates were classified to be “non-relevant” [12]; and (4) if, for some reason, a retrieved document became inaccessible, it was classified to be “non-relevant” [12].
Precision and normalized recall ratios of keyword-based search engines were calculated at various cut-off points (first 5, 10, 15 and 20 documents retrieved) for each pair of query and search engine. Furthermore, these calculations were performed for each pair of query and Hakia, and for each pair of PwQ and Hakia. However, for each query topic, the average precision and normalized recall ratios of the query and four PwQs was considered in evaluating the semantic search engine Hakia.
Precision is defined as the ratio of the number of relevant documents retrieved to the number of total documents retrieved [14]. As such, using precision at different cut-off points is helpful in estimating the distribution of relevant documents over their ranks [12].
The score-rank curve is strongly associated with the normalized recall, say Rnorm [15]. Rnorm is based on optimized expected search length [16].
2
In the study of Spink and Jansen [13], a data collection from
Dogpile was analyzed and one of the findings was that the ratios of the users that viewed only the first page and those that viewed only the first two pages of document search results were about 71.9% and 15.8%, respectively.
Ball and Brown_1968_An empirical evaluation of accounting income numbers_金融/投资_经管营销_专业资料。 今日推荐 160份文档 四级养生 中医养生与保健 中医...
An empirical evaluation of models of text ...Introduction Modeling the semantic similarity between...ts, improving the performance of search engines, ...
An Empirical Evaluation of Co-ordination Strategies for an AUV and UAV翻译 隐藏>> 对无人驾驶飞行机(无人机)和水下自航行器的一个基于经 验评估的协调...
Empirical analysis on a keyword-based semantic system arXiv:0801.4163v2 [physics.data-an] 9 May 2008 Zi-Ke Zhang a, Linyuan L¨ b , Jian-Guo ...
An empirical evaluation ... 暂无评价 27页 免费...performance of Bod & Kaplan's probability model ..."A Corpus-Based Approach to Semantic Interpretation...
An Empirical Evaluation ... 暂无评价 10页 免费...cation performance, feature set parsimony, and ... local and stochastic search wrappers (Guyon and ...
Web Service Discovery with UDDI Based on Semantic Similarity of Service_...services and will largely enhance search performance of recall and precision....
Empirical study on evaluation of regional agri-food logistics capability in e-commerce environme_专业资料。Dcmbr00Vlme,.2(eil。)eee 1,ou No1Sr 9 29aNo...
Empirical evaluation of confidence and prediction intervals for spatial models of forest structu_专业资料。Junlfoet eerh(0 2()1916oraoFrsyRsac21)22:5-6 ...
Evaluation and Improvement of a SVD-Based Empirical Atmospheric Model_专业资料。An empirical atmospheric model(EAM) based on the singular value decomposition...
热门文档