The segmentation of images with inadequate contrast characteristics is an important challenge in Computer Vision: images are either oversegmented, with “objects” divided into parts, or images are incorrectly segmented, with two or more objects segmented as one object.This is a problem that occurs in all types of segmentation approaches, but is of particular importance in the field of region-growing algorithms, which are used in many medical applications. We present a new knowledge-based method, based on the inexact consistent labelling method, that enables the automated consistency checking of the results of region-growing segmentations and that is capable to automatically “fitting” erroneous segmentations, when they are oversegmented, given there exists a reliable domain model.
1. Introduction
A challenge in image analysis and computer vision is the segmentation of images with inadequate contrast characteristics: images are either oversegmented, with “objects” divided into parts, or images are incorrectly segmented, with two or more objects segmented as one object or even parts of different image objects considered to be a new object by the segmentation algorithm. This is a problem that occurs in all types of segmentation approaches, but is of particular importance in the field of region-growing algorithms, such as Watershed [1] or Mumford & Shah [2], which are used in many medical applications. The fact that such important step in image analysis is not stable has been a major fallback in former attempts to develop “fully automated” medical image analysis applications.
The method described here was specially developed to be used in the medical domain, more specifically, in the radiological domains of CT and MR images, where segmentation problems occur, but can be corrected by a posterior knowledge-based approach, since there is a well defined domain knowledge available.
Fig.1. MR image of a human´s brain and 2 different segmentations -performed intending to separate the brain tissue from the rest of the image done with the Mumfod & Shah algorithm using different parameter-settings.
Fig.1 shows two different segmentations performed on a MR slice of a human brain intending to separate the brain tissue from other structures represented in the image. Both segmentations resulted in a division of the brain tissue into different segments. Cases like this, where a segmentation either produces segments that “leak” into other structures or produces oversegmentated images, like the examples above, are one of the most important challenges in image understanding. For such an oversegmentation to be useful, it is necessary to identify which segments pertain to the targeted tructure and to “glue” them together, automatically generating a new segment as a result of the merging of the correct segments.
This work is part of the Cyclops Project [3], which is a German-Brazilian joint project aimed at the development of a framework for intelligent radiological image analysis and decision support.
2. Objectives
Given a) a medical CT or MR image segmented by any region growing technique and b) a model of expected “consistent” results for this image segmentation, this approach should match the model with the segmentation results and to classify these as correct or not, trying to fix them where possible. The model should be a description of anatomical structures expected to be found in the image, along with a list of parameters describing the appearance of these structures, such that an automated identification and matching between model and resulting segments is possible. This model should also allow the analysis of medical images where pathologies are searched for, but that are not necessarily present in the image, enabling the description of “obligatory” and “optional” structures. Obligatory are all anatomies that have to be well segmented in this particular image, optional are all pathological structures, that can be in the image, but should not necessarily be there.
The model should be a description of anatomical structures expected to be found in the image, along with a list of parameters describing the appearance of these structures, such that an automated identification and matching between model and resulting segments is possible. This model should also allow the analysis of medical images where pathologies are searched for, but that are not necessarily present in the image, enabling the description of “obligatory” and “optional” structures. Obligatory are all anatomies that have to be well segmented in this particular image, optional are all pathological structures, that can be in the image, but should not necessarily be there.
Since one technique to avoid that a segment contains areas that belong to different objects is to use segmentation parameters that are oversensitive and produce several segments for each object, but none that has parts of different objects, the approach should use the model to “classify” the segments and to merge those pertaining to the same image object. The approach should provide a means of performing this in a totally automatic way, without user intervention unless an acceptable corretion of the segmentation is not possible to be found.
3. Methods
Our approach was developed as an extension of the inexact consistent labelling method developed by Haralick and Shapiro [4]. This method was extended to enable the use of a) complex models of expected results where each unit has an a priori label and also to b) enable the use of “optional” labels representing pathologies. The representation of units is done as a pattern of segment parameters extracted by different algorithms, where some of them are general and other diagnosis-task specific. The a priori labelling is done by backpropagation neural networks based on prior training. Changing an a priori label will increase the error. Otherwise, if two segments are merged and the result is more similar to a prototype in the model, the global error decreases. The model can be generated from a static representation in a database or through a structured bayesian network using an approach based on Shastri´s model and parameters about the position of CT or MR slices.
3.1 Procedure
Based on the type of examination that is performed and on the position of the slice being examined, a graph structure representing the expected results of image processing is generated and passed to the Failure Detection Module. Each knot of this graph contains an object describing on anatomical or pathological structure by means of a set of parameters retrieved from the domain database. The parameters are described in Table 1. For each segment generated during segmentation, a list of parameters is calculated, which is used as a segment description in the consistent labelling that will be performed afterwards. For each segment during the labelling process, there are generated description parameters that differ partially from the domain database parameters: Bounding Box, Area, Grayvalue and Neighbours are computed the same way; Anatomy and Type are substituted by other parameters.
Table 1. Parameters used to represent views of anatomies in radiological images as stored in the domain knowledge database. The values aregenerated based on mean values obtained from a base of CT and MRimages of different patients.
Table 2 shows the extra parameters generated for each segment and used during the labelling. For the neural network preclassification, some extra grayvalue distribution parameters such as grayvalue variance are calculated. The task of the consistent labelling is to match the segmentation results with the domain model provided for this specific image. This matching allows an error that is bound to a threshold and is based on the differences between “ideal” parameters from the database and the calculated ones. If the error rises above the threshold, the segmentation is considered as failed.
Table 2. Parameters generated for each image segment resulting from an oversensitive segmentation that differ from those of the database prototypes.
The consistent labelling problem udes in our approach is defined as follows: A set of Units U, given by the set of resulting segments and their parameters. A set of Labels L, given by the list of anatomies and pathologies expected to be found in the image and their parameters. A Unit-Constraint-Relation T, given by all pairs of adjacent image segments, represented as a neighbourhood function. A Unit-Label-Constraint-Relation R, given by a set of functions that match parameter sets between segments and anatomies. An Error Function w, also given by a set of functions, that can handle future error calculations with possible segment mergings.
The Error Threshold e is a real value that gives the maximum allowed error in one search branch.
3.2 Main Extensions to Haralick and Shapiro´s Model
Error Estimation: Since not all labels are optional, the future error estimation cannot be performed over the unlabeled units. In our approach, it is based on the not already given obligatory labels and coded into a Future Error Table (FTAB).
A priori labelling: Each Unit has an a priori label with an associated error, both generated by a neural network preclassification. An a priori label can only be changed if it reduces the global error by a significant amount.
A same label can be given to more than one Unit: If the merging of two adjacent segments results in a parameter set for a new unit that reduces the global error, than they will be merged and a new unit will be generated. The merging is performed by a special Merging Function, that also calculates the new future error and that can be passed as a parameter to the labelling function.
4. Results
The approach was implemented and sucessfully tested on different segmentations of brain MR images. Now it is being used in an application for the 3D reconstruction and measurement of aortic aneurysms for the production of personalized endoluminal protheses. In this application domain, the aneurismatic tissue, which has to be detected, measured and rendered, has radiological densities very simillar to other anatomies around the abdominal aorta and images have to be oversegmented.
