2012 Second International Conference on Cloud and Green Computing
A Content-based Image Retrieval System Based on Hadoop and Lucene
Chunhao GU
Suzhou Branch Office
Oracle (China) Software Systems Co., Ltd.
Suzhou, China guch2010@gmail.com
Abstract—This paper introduces a content-based image retrieval (CBIR) system based on Hadoop and Lucene. Hadoop is a kind of open source software with powerful parallelization and scalability and gradually becomes a popular technique for processing and storage of big data in recent years. We design and implement the system to overcome performance bottlenecks brought by computing complexity and big amount of data when constructing a CBIR system. We will present our ideas, designs and fruits of the system in this paper.
Keywords: CBIR; Hadoop; HBase; Lucene
Yang GAO
Department of Computer Science & Technology
Nanjing University Nanjing, China gaoy@nju.edu.cn
[4], Virage [1], MARS [15], TinEye [10], etc. For all CBIR systems, algorithms for image feature extraction, methods of similarity measure between images and methods of relevance feedback are most basic issues.
1) Feature Extraction Algorithms
Low-level feature extraction algorithms are very important for research of CBIR techniques. These algorithms extract color, texture, shape and space features from an image into feature vectors. Commonly used algorithms for color features are color histogram, color layout, color correlogram, etc [13] and texture features are Tamura texture, Gabor filter, etc [17]. 2) Similarity Measure
The similarity between two images is measured by the distance of feature vectors of the images. Commonly used methods are Euclidean distance, histograms’ cross, Mahalanobis distance, etc. 3) Reference Feedback
Relevance feedback technique is a kind of human-computer interaction method to improve performance of a system. It was introduced firstly in MARS [15]. Commonly used methods are naïve Bayes, SVM, and a method of dynamic adjustment of weights of feature vectors, etc. B. Hadoop
Hadoop consists of two components of HDFS and MapReduce, which are respectively the implement of Google’s GFS and MapReduce [18].
HDFS is a distributed file system for big data storage with outstanding scalability and fault tolerance [16].
MapReduce is a distributed framework for data processing, especially big data [3]. The MapReduce process consists of two steps, Map and Reduce. Splits of data are inputted into the Map process which will output intermediate key-value pairs. And then lists of these key-value pairs each of which has a common key are inputted into the Reduce process to output final key-value pairs. For Hadoop, a programmer needs to customize his MapReduce program and submit it to the Hadoop cluster as a job to run parallelly.
HBase is a kind of NoSQL database software based on Hadoop [5, 7]. A record of a HBase table consists of a sole row key and some column families. Every column family has one or more columns defined by qualifiers. A
I. INTRODUCTION
Recently, with the development of the Internet and multimedia techniques, global digital images are increasing at an unprecedented speed. Therefore, to retrieve and manage these distributed images has become an important research issue.
For the difference between text data and image data, e.g. an image’s texture is hard to describe in a text, traditional text-based image retrieval systems have not be able to satisfy all requests. A lot of companies and organizations have begun to develop content-based image retrieval systems, known as CBIR systems.
Hadoop is a kind of open source software under the Apache Foundation [6, 18]. For its powerful parallelization and scalability, Hadoop has gradually become a popular technique for processing and storage of big data. Yahoo!, eBay, Facebook and Baidu have applied Hadoop in many products, such as data mining, searching, recommendation, etc.
For a long time, high computation tasks caused by computing complexity and big amount of data during storing, indexing, etc have been bottlenecks for constructing a CBIR system. Therefore, we study, design and implement a system based on Hadoop to explore the solution to the problem.
This paper is organized as follows: we discuss previous related works in Section II; we introduce our design of the system’s architecture and modules in Section III; in Section IV, we show the demo of the system’s prototype.
II.
PREVIOUS RELATED WORKS
A. CBIR
CBIR technique has begun from 1990s [2]. After that, there appeared a lot of famous CBIR systems, such as QBIC
a Java library for content based image retrieval....system – is often relying on fast, global ...image mosaics (see Figure 3) based on LIRe is...
content-based image retrieval, and MapReduce, among...le system (DFS), being Hadoop [4, 5] the ...ciently evaluate analytic queries based on di?...
基于Lucene的面向主题搜索... 暂无评价 3页 2.00元 图像搜索引擎案例评价及检...CIRES: A System for Content-based Retrieval in Digital Image Libraries. ...
content-based and so on, than introduced system ...Apache Mahout起源于2008年, 是Apache Lucene的 子...(IR, Information Retrieval 的知识将文档拆分为词项...
lucene indexes fetch URL list pages (segments) ...Hadoop for SW Amazon S3 TIFF Image (4TB) PDF..., Information Retrieval, ???, ??? ? Hadoop...
2011#A Real-Time Search Engine with Lucene and ...S4 applied to Information Retrieval 2/5/2011 ...Let’s move it to Hadoop PDF Text Extractor ...
Hadoop集群性能优化技术... 3页 免费 Lucene_3.0...System’, Proceedings of the ACM-SIGMOD Conference...elimination of inconsistencies in data retrieval. ?...
Only few retrieval results are needed Not ideal ...A scale-out system, e.g., Nutch/Lucene, ...Hadoop based on SQL [Thusoo, Hive ApacheCon 2008...
Only few retrieval results are needed Not ideal ...A scale-out system, e.g., Nutch/Lucene, ...Hadoop based on SQL [Thusoo, Hive ApacheCon 2008...
热门文档