CMU thesis proposal: Incorporating unsupervised image segmentation into object class detection and localization

Caroline Pantofaru, Robotics Institute, Carnegie Mellon University

28 Aug 2006

Abstract
As the performance of object recognition and localization systems improves, there is increasing demand for their application to problems which require an exact pixel-level object mask. Photograph post-processing and robot-object interaction are just two examples of applications which require knowledge of exactly which pixels in an image are part of a specific object, and which ones are not. Traditional object recognition systems which generate bounding boxes around the found objects are inappropriate for these applications. The point- and patch-based features that these systems use are also ill-suited to delineating an object mask for a highly deformable object. Thus we propose to explore a framework for using segmentation regions for object learning and recognition. Image segmentation regions have a data-driven shape, so they can adapt to object boundaries well. In fact, if the right set of regions is grouped together, the entire object can be defined. In this proposal we will examine the issues which accompany using segmentation regions for recognition, namely:

- describing segmentation regions in a reliable and discriminative manner,

- grouping over-segmented regions together for more robust recognition and complete object segmentation, and

- within the context of the above framework, generating multiple segmentations per image to overcome the inherent ambiguity in unsupervised segmentation.

Since obtaining training data with hand-segmented objects is extremely expensive, we propose to use semi-supervised training data for which only image-level object labels are known but the pixels themselves are not labeled. Upon completion of the items in this proposal, we will have a better understanding of the issues related to performing object recognition and localization for such demanding applications.

A copy of the thesis proposal document can be found at http://gs2051.sp.cs.cmu.edu/proposal.pdf.