Table 2. Number of relevant documents retrieved Query
Google Yahoo Msn Hakia1 Hakia2 Hakia3 Hakia4 Hakia5
Number
5 9 6 7 9 1 6 3 Q1
3 12 4 4 4 7 7 9 Q2
2 2 1 2 6 0 10 9 Q3
5 6 3 7 12 3 15 4 Q4
9 10 2 7 6 4 6 1 Q5
5 5 5 7 6 3 9 2 Q6
2 1 2 7 8 1 11 4 Q7
2 9 3 11 9 2 9 2 Q8
10 13 9 16 15 7 12 13 Q9
6 7 9 7 6 4 12 5 Q10
Avg (%) 24.5 37 22 37.5 40.5 16 48.5 26
: The original query is used; : “what is <query>” is used; : “information about <query> ” is used;
45
: “<query> definition” is used; : “description of <query>” is used.
Hence, normalized recall considers 1 to be better than 2 if 1 provides fewer non-relevant documents; here 1 and 2 are two different retrieval outputs. In this study, Rnorm was calculated at four cut-off points (cut-off 5, cut-off 10, cut-off 15 and cut-off 20) in order to get values parallel to those for precision. Formula 1, proposed by Bollmann et al. [17], was used to calculate normalized recall values at various cut-off points.
1 R+ R
Rnorm(Δ)= 1+ (1) 2 Rmax
where R is the number of document pairs that have relevant documents ranked higher than non-relevant documents, R is the number of document pairs that have non-relevant documents ranked higher than
+
queries (“information about fashion”). Yahoo retrieved more relevant documents and Msn retrieved least number of relevant documents than other keyword-based search engines. Yahoo and Hakia1 retrieved approximately the same number of relevant documents in total. However, Hakia2 and Hakia4 retrieved more relevant documents than Yahoo, and Hakia5 retrieved more relevant documents than Google and Msn. Google, Yahoo and Msn retrieved approximately 75.5%, 63% and 78% non-relevant documents, respectively, and, for the original query and all PwQs, Hakia retrieved 62.5%, 59.5%, 84%, 51.5% and 74% non-relevant documents, respectively. Although Hakia3 retrieved least number of relevant documents in total, Hakia4 retrieved more relevant documents than all others.

relevant ones, and Rmaxgives the maximum number of R+ [12].
+
3. Experimental Results
Retrieval performance of search engines can be evaluated using the number of zero retrievals (i.e., no documents retrieved) or retrievals containing no relevant documents (i.e., the precision ratio is zero) [12]. The number of relevant documents retrieved by each search engine for the first twenty documents retrieved is shown in Table 2.
While the queries and PwQs ran on the search engines, the expectation was to retrieve documents that contain an explanation regarding the queries. Google, Yahoo and Msn retrieved at least one relevant document for all queries; however, Hakia3 failed to retrieve any relevant document for one of the
Figure 1. Mean precision ratios of search
engines
Mean precision ratios of keyword-based search engines and the semantic search engine at various cut-off points (for first 5, 10, 15, and 20 documents retrieved) are shown in Figure 1. Google’s precision ratio is the same with Hakia at cut-off point 5 (38%).
Ball and Brown_1968_An empirical evaluation of accounting income numbers_金融/投资_经管营销_专业资料。 今日推荐 160份文档 四级养生 中医养生与保健 中医...
An empirical evaluation of models of text ...Introduction Modeling the semantic similarity between...ts, improving the performance of search engines, ...
An Empirical Evaluation of Co-ordination Strategies for an AUV and UAV翻译 隐藏>> 对无人驾驶飞行机(无人机)和水下自航行器的一个基于经 验评估的协调...
Empirical analysis on a keyword-based semantic system arXiv:0801.4163v2 [physics.data-an] 9 May 2008 Zi-Ke Zhang a, Linyuan L¨ b , Jian-Guo ...
An empirical evaluation ... 暂无评价 27页 免费...performance of Bod & Kaplan's probability model ..."A Corpus-Based Approach to Semantic Interpretation...
An Empirical Evaluation ... 暂无评价 10页 免费...cation performance, feature set parsimony, and ... local and stochastic search wrappers (Guyon and ...
Web Service Discovery with UDDI Based on Semantic Similarity of Service_...services and will largely enhance search performance of recall and precision....
Empirical study on evaluation of regional agri-food logistics capability in e-commerce environme_专业资料。Dcmbr00Vlme,.2(eil。)eee 1,ou No1Sr 9 29aNo...
Empirical evaluation of confidence and prediction intervals for spatial models of forest structu_专业资料。Junlfoet eerh(0 2()1916oraoFrsyRsac21)22:5-6 ...
Evaluation and Improvement of a SVD-Based Empirical Atmospheric Model_专业资料。An empirical atmospheric model(EAM) based on the singular value decomposition...
热门文档