The Carnegie Mellon University Informedia group has enjoyed consistent success with TRECVID interactive search using traditional storyboard interfaces for shot-based retrieval. For TRECVID 2006 the output of automatic search was included for the first time
The topics are defined by NIST to reflect many of the sorts of queries real users pose, based on query logs against video corpora like the BBC Archives and other empirical data [6]. The topics include requests for specific items or people and general instances of locations and events, reflecting the Panofsky-Shatford mode/facet matrix of specific, generic, and abstract subjects of pictures. In video retrieval, a broadcast is commonly decomposed into numerous shots, with each shot represented by a keyframe: a single bitmap image extracted from that shot. The numerous keyframes can then be subjected to image retrieval strategies. This simplified approach to video retrieval is taken here, with the benefit that many lessons learned for such shot-based video retrieval will be applicable as well for digital still image retrieval. Two shortcomings left for separate investigations are determining how to best represent the contents of a shot with a keyframe or set of keyframes, and how to account for and leverage temporal information present in video, such as rate and direction of motion I
and camera pans and zooms. n the nformedia interfaces discussed here, the temporal progression of information in video is presented through the arrangement of shot thumbnail imagery into storyboards, where shots related in video time within the same video segment are displayed sequentially in time order. n each and every TRECVD prior to 2006, the nformedia storyboard-based interface [2, 3] has either produced the top score for interactive search, or produced high scores statistically clustered with the top-scoring MediaMill interface [8, 9]. It has been an interface with consistently proven effectiveness over the years, but was challenged in TRECVID 2005 with a different type of interface, an “extreme” video retrieval (XVR) interface based on rapid serial visual presentation of keyframes [5]. The XVR interface, the Informedia storyboard interface, and the MediaMill interface collectively produced significantly better interactive search runs than all other submissions in TRECVD 2005. MediaMill and Informedia shared 3 common means of querying the news corpus to produce shot sets: query-by-textual-keyword, query-by-image-example, and query-by-concept [9]. The XVR interface provided a fourth strategy: do no queries, but work only against the output of a fully automatic search result for the topic. For TRECVID 2006, we specifically examined the effects of different interfaces presenting the ranked shot output of automatic retrieval systems, to understand better the reason behind the good performance of extreme video retrieval interface. Is it because XVR is an inherently better interface than the dense storyboards used in the traditional Informedia interface, or is it due to the availability of good ranked shot output from automatic retrieval? In addition, we explored whether there would be a benefit toward including another shot set producing strategy into the Informedia storyboard interface: looking at the automatic search run output, in addition to query-by-text, query-by-image, and query-by-concept.
To facilitate easier comparison across TRECVID participants, we adopt the query access descriptions provided by Snoek and colleagues in their work [8, 9]: “query-by-textual-keyword”, “query-by-image-example”, and “query-by-concept.” To still distinguish the primary modality of the query but to save space, we discuss these access methods as “query-by-text”, “query-by-image”, and “query-by-concept”, respectively, with one addition: the available of the ranked shot output of the fully automatic search, which we label “query-by-best-of-topic” since this is a topic-specific shot list. The six TRECVID Informedia search runs
discussed in the remainder of this paper can be summarized as follows:xAutomatic Baseline: automatic search based only on transcript text
xAutomatic: automatic search encompassing all modalities text, visual, and aural
x
Informedia Full: storyboard interface with 4 means of access: query-by-text, query-by-image, query-by-concept, and query-by-best-of-topic (the latter “query-by-best-of-topic” looks through the ranked shot list produced by the Automatic run)
xInformedia Limited: the same interface as Informedia Full, except that only query-by-best-of-topic is available x
XVR-1: the same data as Informedia Limited, i.e., the output of the Automatic run just as XVR was used with success in TRECVD 2005, but an “extreme” interface manual browsing with resizable pages
x
XVR-2: the same data as Informedia Limited, but an “extreme” interface rapid serial visual presentation.
Given that all of the runs make use of automatic search, a discussion of the state of automatic search is given first, followed by the results from the different interactive search runs.
2.TRECVID AUTOMATIC SEARCH
TheAutomatic search run makes use of relevance-based probabilistic retrieval models to determine weights for combining a large set of semantic concepts when addressing a given topic [1, 4, 10]. This model translated the retrieval task into a supervised learning problem with the parameters learned discriminatively. Rather than treating retrieval as a classification problem, we used an algorithm called “ranking logistic regression” [11] by accounting for the order information between training data, so that the optimization is closely associated with the retrieval performance criteria. We also use a direct query expansion approach to automatically map the multimodal query to existing semantic concepts.
2.1Relevance-Based Retrieval Model
Although we considered a relevance-based probabilistic retrieval model for knowledge combination in our implementation, previous work showed that simply adopting a query-independent knowledge combination strategy is not flexible enough to handle variations in users’ information needs. We developed more advanced methods to incorporate the factor of query information into the probabilistic retrieval model, by making the following assumptions on the query space: 1.
The entire query space can be described by a finite number of mixtures, where the queries from each mixture have the similar characteristics and share the same combination function.
2.
Query descriptions can be used as indicators to which mixture the query (i.e., topic) belongs.
Based on these assumptions, the retrieval model can be represented as:
Merging Storyboard Strategies and Automatic Retrieval for Improving Interactive Video Search. In Proc. Image and Video Retrieval (CIVR) (July 2007). [2] ...
Merging Storyboard Strategies and Automatic Retrieval for Improving Interactive Video Search. In Proc. Image and Video Retrieval (CIVR) (July 2007). [2] ...
热门文档