Shot Change Detection via Local Keypoint Matching

    Shot change detection is an essential step in video content analysis. However, automatic shot change detection often suffers high false detection rates when there are camera or object movements. In this paper, we propose an approach to solve this problem based on local keypoint matching of the video frames. Experimental results show that the proposed algorithm is effective for all kinds of shot changes. This research was supported in part by NSC95-2422-H- 001-024, NSC 95-2422-H-001-007 and NSC 95-2752-E-002- 007-PAE from the National Science Council, Taiwan.

    A shot is a series of frames representing the same objects with continuous actions in time and space. The most intuitive approach of shot change detection is to recognize objects and scenes. If the same objects or scenes appear in consecutive frames, we may consider that there is no shot change (transition).

    The goal of feature matching is to match points on the same object in multiple images. Image correspondence obtained in this manner is useful in several fields of computer vision and image processing, such as object recognition, 3D structure reconstruction from images, image retrieval, building of panoramas, and augmented reality. Huang et al. proposed using the contrast_context_histogram (CCH), which is more efficient to compute, to find image correspondences as shown in the following figures.

    We produce lists of keypoints and their local descriptor values for each frame. Then, we perform keypoint matching for each pair of adjacent frames, which yields a 1D signal of numbers of the matched points. Since most objects that appear before the shot change should be replaced after the transition, we assume that a shot change takes place when there is a salient local minimum in the values of the 1D signal. Therefore, the problem is reduced to finding the minima of the 1D signal.

    The candidates found with local minima are only time instants, not intervals. However, since many shot changes are gradual transitions, it is necessary to find the intervals of such transitions. Our method for finding the intervals is also based on feature matching. Shot changes are likely to occur when the number of matched objects decreases; thus, there should not be any transitions when several objects in adjacent frames are matched. In our method, the local maxima to the left and right of the candidate transition are possible start and end frames of that transition. We add another condition: the video sequence before and after the shot change should also be “stable,” resulting in stable numbers of matched keypoints. Hence, the search for start and end points begins with the two maxima and continues until the number of matched keypoints is stable.

    Matching non-adjacent frames provides richer image correspondence information, but exhaustively matching a large number of pairs of frames within an interval is very time consuming. Since variations in a shot usually continue for a limited period of time, we match the frames before and after the intervals of candidate shot changes. If the number of matched CCH features between the first and last frames of an interval is relatively high, it indicates that the same objects remain visible; thus, the candidate transition detected initially is a false alarm and should be deleted.

[1] Chun-Rong Huang , Huai-Ping Lee and Chu-Song Chen, “Shot Change Detection via Local Keypoint Matching,” IEEE Transactions on Multimedia , vol. 10, no.6, pp. 1097-1108, 2008. (SCI)

[2] Chun-Rong Huang, Huai-Ping Lee and Chu-Song Chen, “A Local Keypoint Matching Technique for Transition Detection,”In the proceedings of the Tenth IAPR Conference on Machine Vision Applications, MVA'07, pp. 219-222, Tokyo, Japan, 2007.