文档下载 文档下载导航
设为首页 | 加入收藏
搜索 请输入内容:  
 导航当前位置: 文档下载 > 所有分类 > IT/计算机 > 互联网 > An Empirical Evaluation on Semantic Search Performance of Keyword-Based and Semantic Search Engines
侵权投诉

An Empirical Evaluation on Semantic Search Performance of Keyword-Based and Semantic Search Engines

Table 1. Query List

Query Number

Q1 Q2 Q3 Q4 Q5

This paper is organized as follows: Section 2 describes the methodology employed to evaluate

Query Number

Query

network switch computer engineer fashion evolution baby sitter

Query

air conditioner Q6

nightmare Q7

atom Q8

reincarnation Q9

myopic Q10

After each run of the query or PwQ, the first 2

twenty documents retrieved were evaluated using search engines in terms of precision and normalized recall, Section 3 reports and discusses the experimental findings and the last section concludes the paper.

2. Methodology

Initially, three keyword-based search engines, namely, Google, Yahoo and Msn, and a semantic search engine Hakia were selected. Afterwards, ten queries that contain various topics and consist of one

or two terms1

were randomly determined as shown in Table 1. (Note that for some queries, Hakia displays categorized documents in its retrieval output before displaying web results. In order to have compatible retrieval outputs for search engine evaluations, during query selection process, particularly those queries were used that provide web results in the retrieval output, without the categorized documents, when run on Hakia).

Ten queries were run on, both, the selected keyword-based search engines as well as the semantic search engine. Additionally, four phrases, with different syntax but similar meanings, were used with each query and run, one by one, on the semantic search engine Hakia. The phrase-with-query (PwQ) forms were as follows: (1) “what is <query>”; (2) “information about <query>”; (3) “<query> definition”; and (4) “description of <query>”.

In order to have realistic results:

Only keywords were used on keyword-based

search engines since, in general, the users do not tend to use phrases (as observed in the most frequently used queries list of Wordtracker [11]).

Beside the keywords, phrases were used for

Hakia since one of the main features of Hakia, being a semantic search engine, is the use of phrases.

1

Jansen and Spink examined a transaction log of Dogpile in their

study [10] and they have revealed that 70% of users used queries that consist of two terms or less.

binary human relevance judgment and with this, every document was classified as “relevant” or “non-relevant”. Total 1600 documents were evaluated by the same author and in order to have stable performance measurement of search engines, all the searches and evaluations were performed in minimal non-distant time slots. While evaluating the retrieved documents following criteria were used: (1) Documents that contain any explanation about the searched query were considered “relevant”; (2) documents having same content but originating from different web addresses (i.e., mirror pages) were classified to be different [12]; (3) in case of duplicated documents, the first document that was retrieved was considered in the evaluation process, whereas its duplicates were classified to be “non-relevant” [12]; and (4) if, for some reason, a retrieved document became inaccessible, it was classified to be “non-relevant” [12].

Precision and normalized recall ratios of keyword-based search engines were calculated at various cut-off points (first 5, 10, 15 and 20 documents retrieved) for each pair of query and search engine. Furthermore, these calculations were performed for each pair of query and Hakia, and for each pair of PwQ and Hakia. However, for each query topic, the average precision and normalized recall ratios of the query and four PwQs was considered in evaluating the semantic search engine Hakia.

Precision is defined as the ratio of the number of relevant documents retrieved to the number of total documents retrieved [14]. As such, using precision at different cut-off points is helpful in estimating the distribution of relevant documents over their ranks [12].

The score-rank curve is strongly associated with the normalized recall, say Rnorm [15]. Rnorm is based on optimized expected search length [16].

2

In the study of Spink and Jansen [13], a data collection from

Dogpile was analyzed and one of the findings was that the ratios of the users that viewed only the first page and those that viewed only the first two pages of document search results were about 71.9% and 15.8%, respectively.

上一页第2页

你可能喜欢

热门文档

相关文档

站点地图 | 文档上传 | 侵权投诉 | 手机版
本站所有资源均来自互联网,本站只负责收集和整理,均不承担任何法律责任,如有侵权等其它行为请联系我们.
文档下载 Copyright 2013 doc.guandang.net All Rights Reserved.  -  email