21-03-2014, 03:07 PM
[u]Content-based Image Retrieval[/u]
[attachment=60382]
Executive summary
The aim of this report is to review the current state of the art in content-based image retrieval (CBIR), a technique for retrieving images on the basis of automatically-derived features such as colour, texture and shape. Our findings are based both on a review of the relevant literature and on discussions with researchers and practitioners in the field.
The need to find a desired image from a collection is shared by many professional groups, including journalists, design engineers and art historians. While the requirements of image users can vary considerably, it can be useful to characterize image queries into three levels of abstraction: primitive features such as colour or shape, logical features such as the identity of objects shown, and abstract attributes such as the significance of the scenes depicted. While CBIR systems currently operate effectively only at the lowest of these levels, most users demand higher levels of retrieval.
Users needing to retrieve images from a collection come from a variety of domains, including crime prevention, medicine, architecture, fashion and publishing. Remarkably little has yet been published on the way such users search for and use images, though attempts are being made to categorize users’ behaviour in the hope that this will enable their needs to be better met in the future.
Current indexing practice for images relies largely on text descriptors or classification codes, supported in some cases by text retrieval packages designed or adapted specially to handle images. Again, remarkably little evidence on the effectiveness of such systems has been published. User satisfaction with such systems appears to vary considerably.
CBIR operates on a totally different principle from keyword indexing. Primitive features characterizing image content, such as colour, texture, and shape, are computed for both stored and query images, and used to identify (say) the 20 stored images most closely matching the query. Semantic features such as the type of object present in the image are harder to extract, though this remains an active research topic. Video retrieval is a topic of increasing importance – here, CBIR techniques are also used to break up long videos into individual shots, extract still keyframes summarizing the content of each shot, and search for video clips containing specified types of movement.
Introduction
Interest in the potential of digital images has increased enormously over the last few years, fuelled at least in part by the rapid growth of imaging on the World-Wide Web (referred to in this report as ‘the Web’). Users in many professional fields are exploiting the opportunities offered by the ability to access and manipulate remotely-stored images in all kinds of new and exciting ways [Gudivada and Raghavan, 1995a]. However, they are also discovering that the process of locating a desired image in a large and varied collection can be a source of considerable frustration [Jain, 1995]. The problems of image retrieval are becoming widely recognized, and the search for solutions an increasingly active area for research and development. Some indication of the rate of increase can be gained from the number of journal articles appearing each year on the subject, growing from 4 in 1991 to 12 in 1994, and 45 in 19981.
Problems with traditional methods of image indexing [Enser, 1995] have led to the rise of interest in techniques for retrieving images on the basis of automatically-derived features such as colour, texture and shape – a technology now generally referred to as Content-Based Image Retrieval (CBIR). After a decade of intensive research, CBIR technology is now beginning to move out of the laboratory and into the marketplace, in the form of commercial products like QBIC [Flickner et al, 1995] and Virage [Gupta et al, 1996]. However, the technology still lacks maturity, and is not yet being used on a significant scale. In the absence of hard evidence on the effectiveness of CBIR techniques in practice, opinion is still sharply divided about their usefulness in handling real-life queries in large and diverse image collections. Nor is it yet obvious how and where CBIR techniques can most profitably be used [Sutcliffe et al, 1997].
Background
The growth of digital imaging
The use of images in human communication is hardly new – our cave-dwelling ancestors painted pictures on the walls of their caves, and the use of maps and building plans to convey information almost certainly dates back to pre-Roman times. But the twentieth century has witnessed unparalleled growth in the number, availability and importance of images in all walks of life. Images now play a crucial role in fields as diverse as medicine, journalism, advertising, design, education and entertainment.
Technology, in the form of inventions such as photography and television, has played a major role in facilitating the capture and communication of image data. But the real engine of the imaging revolution has been the computer, bringing with it a range of techniques for digital image capture, processing, storage and transmission which would surely have startled even pioneers like John Logie Baird. The involvement of computers in imaging can be dated back to 1965, with Ivan Sutherland’s Sketchpad project, which demonstrated the feasibility of computerised creation, manipulation and storage of images, though the high cost of hardware limited their use until the mid-1980s. Once computerised imaging became affordable (thanks largely to the development of a mass market for computer games), it soon penetrated into areas traditionally depending heavily on images for communication, such as engineering, architecture and medicine. Photograph libraries, art galleries and museums, too, began to see the advantages of making their collections available in electronic form. The creation of the World-Wide Web in the early 1990s, enabling users to access data in a variety of media from anywhere on the planet, has provided a further massive stimulus to the exploitation of digital images. The number of images available on the Web was recently estimated to be between 10 and 30 million [Sclaroff et al, 1997] – a figure which some observers consider to be a significant underestimate.
Video queries
Video sequences are an increasingly important form of image data for many users, and pose their own special challenge to those responsible for their storage and retrieval, both because of their additional complexity and their sheer volume. Video images contain a wider range of primitive data types (the most obvious being motion vectors), occupy far more storage, and can take hours to review, while the comparable process for still images takes seconds at most. Hence the process of organizing videos for retrieval is in some ways akin to that of abstracting and indexing long text documents. All but the shortest videos are made up of a number of distinct scenes, each of which can be further broken down into individual shots depicting a single view, conversation or action. A common way of organizing a video for retrieval is to prepare a storyboard of annotated still images (often known as keyframes) representing each scene. Another is to prepare a series of short video clips, each capturing the essential details of a single sequence – a process sometimes described as video skimming. For a detailed discussion of the issues involved in video data management, and a review of current and emerging techniques, see the reviews by Aigrain et al [1996] and Bolle et al [1998].
What is CBIR?
The earliest use of the term content-based image retrieval in the literature seems to have been by Kato [1992], to describe his experiments into automatic retrieval of images from a database by colour and shape feature. The term has since been widely used to describe the process of retrieving desired images from a large collection on the basis of features (such as colour, texture and shape) that can be automatically extracted from the images themselves. The features used for retrieval can be either primitive or semantic, but the extraction process must be predominantly automatic. Retrieval of images by manually-assigned keywords is definitely not CBIR as the term is generally understood – even if the keywords describe image content.
[attachment=60382]
Executive summary
The aim of this report is to review the current state of the art in content-based image retrieval (CBIR), a technique for retrieving images on the basis of automatically-derived features such as colour, texture and shape. Our findings are based both on a review of the relevant literature and on discussions with researchers and practitioners in the field.
The need to find a desired image from a collection is shared by many professional groups, including journalists, design engineers and art historians. While the requirements of image users can vary considerably, it can be useful to characterize image queries into three levels of abstraction: primitive features such as colour or shape, logical features such as the identity of objects shown, and abstract attributes such as the significance of the scenes depicted. While CBIR systems currently operate effectively only at the lowest of these levels, most users demand higher levels of retrieval.
Users needing to retrieve images from a collection come from a variety of domains, including crime prevention, medicine, architecture, fashion and publishing. Remarkably little has yet been published on the way such users search for and use images, though attempts are being made to categorize users’ behaviour in the hope that this will enable their needs to be better met in the future.
Current indexing practice for images relies largely on text descriptors or classification codes, supported in some cases by text retrieval packages designed or adapted specially to handle images. Again, remarkably little evidence on the effectiveness of such systems has been published. User satisfaction with such systems appears to vary considerably.
CBIR operates on a totally different principle from keyword indexing. Primitive features characterizing image content, such as colour, texture, and shape, are computed for both stored and query images, and used to identify (say) the 20 stored images most closely matching the query. Semantic features such as the type of object present in the image are harder to extract, though this remains an active research topic. Video retrieval is a topic of increasing importance – here, CBIR techniques are also used to break up long videos into individual shots, extract still keyframes summarizing the content of each shot, and search for video clips containing specified types of movement.
Introduction
Interest in the potential of digital images has increased enormously over the last few years, fuelled at least in part by the rapid growth of imaging on the World-Wide Web (referred to in this report as ‘the Web’). Users in many professional fields are exploiting the opportunities offered by the ability to access and manipulate remotely-stored images in all kinds of new and exciting ways [Gudivada and Raghavan, 1995a]. However, they are also discovering that the process of locating a desired image in a large and varied collection can be a source of considerable frustration [Jain, 1995]. The problems of image retrieval are becoming widely recognized, and the search for solutions an increasingly active area for research and development. Some indication of the rate of increase can be gained from the number of journal articles appearing each year on the subject, growing from 4 in 1991 to 12 in 1994, and 45 in 19981.
Problems with traditional methods of image indexing [Enser, 1995] have led to the rise of interest in techniques for retrieving images on the basis of automatically-derived features such as colour, texture and shape – a technology now generally referred to as Content-Based Image Retrieval (CBIR). After a decade of intensive research, CBIR technology is now beginning to move out of the laboratory and into the marketplace, in the form of commercial products like QBIC [Flickner et al, 1995] and Virage [Gupta et al, 1996]. However, the technology still lacks maturity, and is not yet being used on a significant scale. In the absence of hard evidence on the effectiveness of CBIR techniques in practice, opinion is still sharply divided about their usefulness in handling real-life queries in large and diverse image collections. Nor is it yet obvious how and where CBIR techniques can most profitably be used [Sutcliffe et al, 1997].
Background
The growth of digital imaging
The use of images in human communication is hardly new – our cave-dwelling ancestors painted pictures on the walls of their caves, and the use of maps and building plans to convey information almost certainly dates back to pre-Roman times. But the twentieth century has witnessed unparalleled growth in the number, availability and importance of images in all walks of life. Images now play a crucial role in fields as diverse as medicine, journalism, advertising, design, education and entertainment.
Technology, in the form of inventions such as photography and television, has played a major role in facilitating the capture and communication of image data. But the real engine of the imaging revolution has been the computer, bringing with it a range of techniques for digital image capture, processing, storage and transmission which would surely have startled even pioneers like John Logie Baird. The involvement of computers in imaging can be dated back to 1965, with Ivan Sutherland’s Sketchpad project, which demonstrated the feasibility of computerised creation, manipulation and storage of images, though the high cost of hardware limited their use until the mid-1980s. Once computerised imaging became affordable (thanks largely to the development of a mass market for computer games), it soon penetrated into areas traditionally depending heavily on images for communication, such as engineering, architecture and medicine. Photograph libraries, art galleries and museums, too, began to see the advantages of making their collections available in electronic form. The creation of the World-Wide Web in the early 1990s, enabling users to access data in a variety of media from anywhere on the planet, has provided a further massive stimulus to the exploitation of digital images. The number of images available on the Web was recently estimated to be between 10 and 30 million [Sclaroff et al, 1997] – a figure which some observers consider to be a significant underestimate.
Video queries
Video sequences are an increasingly important form of image data for many users, and pose their own special challenge to those responsible for their storage and retrieval, both because of their additional complexity and their sheer volume. Video images contain a wider range of primitive data types (the most obvious being motion vectors), occupy far more storage, and can take hours to review, while the comparable process for still images takes seconds at most. Hence the process of organizing videos for retrieval is in some ways akin to that of abstracting and indexing long text documents. All but the shortest videos are made up of a number of distinct scenes, each of which can be further broken down into individual shots depicting a single view, conversation or action. A common way of organizing a video for retrieval is to prepare a storyboard of annotated still images (often known as keyframes) representing each scene. Another is to prepare a series of short video clips, each capturing the essential details of a single sequence – a process sometimes described as video skimming. For a detailed discussion of the issues involved in video data management, and a review of current and emerging techniques, see the reviews by Aigrain et al [1996] and Bolle et al [1998].
What is CBIR?
The earliest use of the term content-based image retrieval in the literature seems to have been by Kato [1992], to describe his experiments into automatic retrieval of images from a database by colour and shape feature. The term has since been widely used to describe the process of retrieving desired images from a large collection on the basis of features (such as colour, texture and shape) that can be automatically extracted from the images themselves. The features used for retrieval can be either primitive or semantic, but the extraction process must be predominantly automatic. Retrieval of images by manually-assigned keywords is definitely not CBIR as the term is generally understood – even if the keywords describe image content.