文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > IT/计算机 > 互联网 > An Empirical Evaluation on Semantic Search Performance of Keyword-Based and Semantic Search Engines
侵权投诉

An Empirical Evaluation on Semantic Search Performance of Keyword-Based and Semantic Search Engines

Table 2. Number of relevant documents retrieved Query

Google Yahoo Msn Hakia1 Hakia2 Hakia3 Hakia4 Hakia5

Number

5 9 6 7 9 1 6 3 Q1

3 12 4 4 4 7 7 9 Q2

2 2 1 2 6 0 10 9 Q3

5 6 3 7 12 3 15 4 Q4

9 10 2 7 6 4 6 1 Q5

5 5 5 7 6 3 9 2 Q6

2 1 2 7 8 1 11 4 Q7

2 9 3 11 9 2 9 2 Q8

10 13 9 16 15 7 12 13 Q9

6 7 9 7 6 4 12 5 Q10

Avg (%) 24.5 37 22 37.5 40.5 16 48.5 26

: The original query is used; : “what is <query>” is used; : “information about <query> ” is used;

45

: “<query> definition” is used; : “description of <query>” is used.

Hence, normalized recall considers 1 to be better than 2 if 1 provides fewer non-relevant documents; here 1 and 2 are two different retrieval outputs. In this study, Rnorm was calculated at four cut-off points (cut-off 5, cut-off 10, cut-off 15 and cut-off 20) in order to get values parallel to those for precision. Formula 1, proposed by Bollmann et al. [17], was used to calculate normalized recall values at various cut-off points.

1 R+ R

Rnorm(Δ)= 1+ (1) 2 Rmax

where R is the number of document pairs that have relevant documents ranked higher than non-relevant documents, R is the number of document pairs that have non-relevant documents ranked higher than

+

queries (“information about fashion”). Yahoo retrieved more relevant documents and Msn retrieved least number of relevant documents than other keyword-based search engines. Yahoo and Hakia1 retrieved approximately the same number of relevant documents in total. However, Hakia2 and Hakia4 retrieved more relevant documents than Yahoo, and Hakia5 retrieved more relevant documents than Google and Msn. Google, Yahoo and Msn retrieved approximately 75.5%, 63% and 78% non-relevant documents, respectively, and, for the original query and all PwQs, Hakia retrieved 62.5%, 59.5%, 84%, 51.5% and 74% non-relevant documents, respectively. Although Hakia3 retrieved least number of relevant documents in total, Hakia4 retrieved more relevant documents than all others.

An Empirical Evaluation on Semantic Search Performance of Keyword-Based and Semantic Search Engines

relevant ones, and Rmaxgives the maximum number of R+ [12].

+

3. Experimental Results

Retrieval performance of search engines can be evaluated using the number of zero retrievals (i.e., no documents retrieved) or retrievals containing no relevant documents (i.e., the precision ratio is zero) [12]. The number of relevant documents retrieved by each search engine for the first twenty documents retrieved is shown in Table 2.

While the queries and PwQs ran on the search engines, the expectation was to retrieve documents that contain an explanation regarding the queries. Google, Yahoo and Msn retrieved at least one relevant document for all queries; however, Hakia3 failed to retrieve any relevant document for one of the

Figure 1. Mean precision ratios of search

engines

Mean precision ratios of keyword-based search engines and the semantic search engine at various cut-off points (for first 5, 10, 15, and 20 documents retrieved) are shown in Figure 1. Google’s precision ratio is the same with Hakia at cut-off point 5 (38%).

上一页第3页

你可能喜欢

热门文档

相关文档

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email