2009 Fourth International Conference on Internet Monitoring and Protection
An Empirical Evaluation on Semantic Search Performance of Keyword-Based
and Semantic Search Engines: Google, Yahoo, Msn and Hakia
Duygu Tümer1, Mohammad Ahmed Shah2, Y ltan Bitirim1
1
Department of Computer Engineering, Eastern Mediterranean University,
Famagusta / T.R.N.C.
027860@students.emu.edu.tr, yiltan.bitirim@emu.edu.tr
2
Department of Computer Engineering, Middle East Technical University
Northern Cyprus Campus, Güzelyurt / T.R.N.C.
shah@metu.edu.tr
Abstract
This paper investigates the semantic search performance of search engines. Initially, three keyword-based search engines (Google, Yahoo and Msn) and a semantic search engine (Hakia) were selected. Then, ten queries, from various topics, and four phrases, having different syntax but similar meanings, were determined. After each query was run on each search engine; and each phrase containing a query was run on the semantic search engine, the first twenty documents on each retrieval output was classified as being “relevant” or “non-relevant”. Afterwards, precision and normalized recall ratios were calculated at various cut-off points to evaluate keyword-based search engines and the semantic search engine. Overall, Yahoo showed the best performance in terms of precision ratio, whereas Google turned-out to be the best search engine in terms of normalized recall ratio. However, it was found that semantic search performance of search engines was low for both keyword-based search engines and the semantic search engine.
Keywords: Information retrieval, Semantic search performance, Semantic search engine, Keyword-based search engine, Evaluation.
1. Introduction
Web search engine is a computer program that allows users to search and retrieve web documents with queries for their information needs. The most popular search engines are Google [1], Yahoo [2] and Msn [3] with 71.9%, 17.7% and 4.2% volume of search ratios (based on US Internet usage), respectively [4]. Google, Yahoo and Msn are keyword-based search engines. However, semantic
search engines are an alternative to these search engines. The difference of semantic search engines from conventional search engines is that the semantic search engines are meaning-based. Hakia [5] is one of the emerging publicly available semantic search engines on the web [6].
Researchers have conducted several studies to evaluate retrieval effectiveness of keyword-based search engines and some have proposed search engines, methods and technologies for semantic search. Atsaros et al. compared the performance of keyword-based search engines (site-specific and general purpose) in terms of precision and relative recall [7]. Kasneci et al. proposed NAGA semantic search engine which uses knowledge base, introducing a graph-based query language and a novel scoring mechanism [8]. However, NAGA is not a publicly available search engine. Lee et al. proposed a semantic web services search system to provide better retrieval effectiveness using clustering and ontology approaches [9].
Although the research trend on semantic search engines is increasing, publicly available search engines on the web are, generally, keyword-based search engines. Keyword-based search engines, such as Google and Yahoo, have a large user-base [4]. Therefore, evaluation of semantic search performance of popular keyword-based and semantic search engines is a valuable work intended to motivate researchers and search engine providers to improve current systems further.
Perfect search engine model might be the one that always finds the precise document(s) on the web for the user. The result of a perfect search engine would, ideally, satisfy the expectations of its users, whenever a query is searched. The inspiration, for this study, is to motivate researchers and search engine providers towards reaching this perfect search engine model.
Ball and Brown_1968_An empirical evaluation of accounting income numbers_金融/投资_经管营销_专业资料。 今日推荐 160份文档 四级养生 中医养生与保健 中医...
An empirical evaluation of models of text ...Introduction Modeling the semantic similarity between...ts, improving the performance of search engines, ...
An Empirical Evaluation of Co-ordination Strategies for an AUV and UAV翻译 隐藏>> 对无人驾驶飞行机(无人机)和水下自航行器的一个基于经 验评估的协调...
Empirical analysis on a keyword-based semantic system arXiv:0801.4163v2 [physics.data-an] 9 May 2008 Zi-Ke Zhang a, Linyuan L¨ b , Jian-Guo ...
An empirical evaluation ... 暂无评价 27页 免费...performance of Bod & Kaplan's probability model ..."A Corpus-Based Approach to Semantic Interpretation...
An Empirical Evaluation ... 暂无评价 10页 免费...cation performance, feature set parsimony, and ... local and stochastic search wrappers (Guyon and ...
Web Service Discovery with UDDI Based on Semantic Similarity of Service_...services and will largely enhance search performance of recall and precision....
Empirical study on evaluation of regional agri-food logistics capability in e-commerce environme_专业资料。Dcmbr00Vlme,.2(eil。)eee 1,ou No1Sr 9 29aNo...
Empirical evaluation of confidence and prediction intervals for spatial models of forest structu_专业资料。Junlfoet eerh(0 2()1916oraoFrsyRsac21)22:5-6 ...
Evaluation and Improvement of a SVD-Based Empirical Atmospheric Model_专业资料。An empirical atmospheric model(EAM) based on the singular value decomposition...
热门文档