key of the record and the columns of the record without tokenizing. And then an IndexWriter will make inverted index for the Document.
Indexing is a process that is done by a MapReduce job. The job’s inputs should be key-value pairs each of which consists of a key that is the row key of a record of
FeatureBase and a value that is a list of column families of the record. The IdentityTableMap class of HBase library can convert records of a HBase table into this kind of key-pairs. So we use it directly and don’t customize Map process. And then, we implement a custom Reduce process, IndexTableReduce, and a custom output format, IndexOutputFormat, to accomplish the creation of inverted indexing as the previous paragraph said. The process is showed in Fig. 3.
The middle layer includes two servlets respectively
called SearchServlet which is responsible for searching and RelFeedbackServlet which is responsible for relevance feedback.
The bottom layer is the intermediate between the middle layer and IndexBase. It includes common classes that have implemented Euclidean distance for similarity measure when searching and a method of dynamic adjustment of weights of feature vectors for relevance feedback. This kind of relevance feedback is a re-searching based on the original searching results, users’ marks and a scoring rule, see [15].
IV. SYSTEM DEMO
For the demo, we established a Hadoop cluster with 4 nodes and ImageBase with over 1000 images. It took the system less than 10 minutes to update FeatureBase and IndexBase.
After the system was ready to serve for users, we landed onto the main page and used file selector to select a querying image about pandas.
After the Search button pressed, the system returned the results instantly as Fig 5.showed. The results were sorted by the descending similarities calculated by the system. But it seemed not so satisfied for us, so we marked all images about pandas and pressed the FeedBack button.
Figure 3. The process of the indexing.
For the parallelization of the indexing, we have adopted incremental indexing of Lucene. For all index files are stored in IndexBase on the HDFS, we have designed HDFSDirectory class which extends from Lucene
Directory and encapsulates HDFS. So it is easy for other modules to access IndexBase.
E. Searching
In Hadoop, MapReduce jobs must be firstly submitted in a waiting queue for running. If searching is done by a
MapReduce job, the instantaneity is preponderant when the number of users visiting the system at the same time is small but unsatisfied when the number is large. So we design the searching module uncoupled with Hadoop and use load balancing method which can make tasks allocated to different servers in the Hadoop cluster.
The design of the searching module is based on B/S architecture and JSP/Servlet technique. It consists of three layers. The top layer is the user interface implemented by HTML and JSP. The compact main page as Fig. 4 shows consists of a textbox, a file selector, a Search button and a logo.
Figure 5. The results of searching.
After the FeedBack button pressed, the system returned the results instantly as Fig. 6 showed. Obviously, all images about pandas ranked top.
Figure 6. The results of relevance feedbacking.
Figure 4. The design of the main page. More experiments showed the system could always rank first the image in ImageBase whose low-level features were
a Java library for content based image retrieval....system – is often relying on fast, global ...image mosaics (see Figure 3) based on LIRe is...
content-based image retrieval, and MapReduce, among...le system (DFS), being Hadoop [4, 5] the ...ciently evaluate analytic queries based on di?...
基于Lucene的面向主题搜索... 暂无评价 3页 2.00元 图像搜索引擎案例评价及检...CIRES: A System for Content-based Retrieval in Digital Image Libraries. ...
content-based and so on, than introduced system ...Apache Mahout起源于2008年, 是Apache Lucene的 子...(IR, Information Retrieval 的知识将文档拆分为词项...
lucene indexes fetch URL list pages (segments) ...Hadoop for SW Amazon S3 TIFF Image (4TB) PDF..., Information Retrieval, ???, ??? ? Hadoop...
2011#A Real-Time Search Engine with Lucene and ...S4 applied to Information Retrieval 2/5/2011 ...Let’s move it to Hadoop PDF Text Extractor ...
Hadoop集群性能优化技术... 3页 免费 Lucene_3.0...System’, Proceedings of the ACM-SIGMOD Conference...elimination of inconsistencies in data retrieval. ?...
Only few retrieval results are needed Not ideal ...A scale-out system, e.g., Nutch/Lucene, ...Hadoop based on SQL [Thusoo, Hive ApacheCon 2008...
Only few retrieval results are needed Not ideal ...A scale-out system, e.g., Nutch/Lucene, ...Hadoop based on SQL [Thusoo, Hive ApacheCon 2008...
热门文档