Sample Queries

In this section, we present some example queries performed on a video data set consisting of 14 video sequences with 25.000 frames from TRECVID 2004 and 2008 data sets, including news, documentary, educational and archiving program videos. We obtained the MPEG-7 representations of the videos with our MPEG-7 feature extraction and annotation tool. The query result is a list of shots in ranked order, each shown with a representative keyframe in the following figures.

1. Spatial Query Examples
Query 1 : Golfer above golf cart (Figure 1.1, top; specified by keywords)
Query 2 : Clinton left Blair (Figure 1.1, bottom; specified by sketch)
Two spatial query examples are shown in Figure 1.1. The first query at the top searches for the video segments in which a golfer is above a golf cart. The query is formulated as "golfer above golf cart" in the Spatial Query Interface. The system successfully returns three relevant video segments that exactly match the query condition. The forth result contains a golfer but no golf cart and spatial condition is not satisfied. Therefore, its rank is lower than the first three. The second query is formulated by drawing two rectangles on the sketch-based Spatial Query Interface and labeling them as Clinton and Blair. The first two video segments satisfy the query condition exactly, while in the last two, the spatial condition is not satisfied but Clinton and Blair appear together. Due to our bottom-up fusion algorithm, as the number of satisfied query conditions for a video segment decreases its similarity also decreases, ranking lower in the query result. As a result, the ranking approach is effective and it produces query results that are close to our perception.

Figure 1.1: Two spatial query examples. Top: Golfer above golf cart (by keywords). Bottom: Clinton left Blair (by sketch).

Figure 1.2: Sketch-based spatial query example with 3 Moving Regions.
Hence, spatial queries containing any number of Still/Moving Regions are handled successfully.

2. Low Level Query Examples
Query 1 : Low Level Query by image (Figure 2.1)
Query 2 : Low Level Query by image region (Figure 2.2)
Query 2 : Low Level Query by video (Figure 2.3)
Figures 2.1, 2.2 and 2.3 show three low-level query examples. The first one is an image-based query, in which the query image is represented with Color Structure and Dominant Color descriptors. The region in the region-based query is represented with Color Structure and Region Shape descriptors. The video in the video-based query is represented with GoF/GoP descriptor. Both query results are satisfactory considering the types of descriptors used.

Figure 2.1: Low Level query by image example. Query image is represented with Color Structure and Dominant Color descriptors.

Figure 2.2: Low-level query by image region example. Query region is represented with Color Structure and Region Shape descriptors, and searched in Moving Regions.

Figure 2.3: Low-level query by video example. Query video is a portion of a shot consisting of 256 frames and is represented with GoF/GoP descriptor.

3. Composite Query Examples
The query shown in Figure 3.1 is a composite query, in which high-level semantics and low-level descriptors are used to describe the query inputs. Moreover, there are two different types of video segments in the query: a Keyframe and Moving Region. In Figure 3.2, he query is composed of Still Regions and Moving Regions which are represented with low-level descriptors or high-level semantic concepts. As the query results show, our system can handle such queries effectively. The number and type of video segments in the query, as well as the descriptors used to describe them are not limited. This makes the composite queries very flexible and powerful, enabling the user to formulate very complex queries easily. To our knowledge, our system is unique in supporting such complex queries.

Figure 3.1: Composite query example. Keyframe is represented with Dominant Color descriptor and semantic concept golf green.
Moving Region (foreground region) is represented with Color Structure, Region Shape descriptors and semantic concept golfer.

Figure 3.2: Composite query example. Two Still Regions at the top and at the bottom are represented with the Color Structure and Edge Histogram descriptors.
The Moving Region in the middle is represented with the semantic concepts airplane or boat or helicopter.

Figure 3.3: Composite query example. The Movýng Region on the left is represented with
the Color Structure descriptor and semantic concept horse; the Still Region on the right is
represented with the semantic concept green or grass.
The spatial relation between the two regions is also considered in the query.
The query processing and fusion architecture of BilVideo-7 provides a great flexibility
to formulate a wide range of queries with any number of video segments with their descriptors
and relations between the segments.

Here, we present query execution times for different types of queries.

Home - Contents
Last update: 05.05.2009
For questions and/or comments please contact Muhammet Baştan