Shape Similarity Measures, Properties, and Constructions Remco C. Veltkamp and Michiel Hagedoorn  Department of Computer Science, Utrecht University Padualaan 14, 3584 CH, Utrecht, The Netherlands fRemco.Veltkamp,mhg@cs.uu.nl Abstract In this paper we list a number of similarity measures, some of which are not well known (such as the Monge-Kantorovich metric), or newly introduced (reection metric). We formulate properties of similarity measures, and introduce new properties. We also give a set of constructions that have been used in the design of some similarity measures, including some new constructions. 1 Introduction Large image databases are used in many multimedia applications in elds such as enter- tainment, business, art, engineering, and science. Retrieving images by their content, as opposed to external features, has become an important operation. A fundamental ingredient for content-based image retrieval is the technique used for comparing images. There are two general methods for image comparison: intensity-based (color and texture) and geometry- based (shape). A recent user survey about cognition aspects of object retrieval shows that users are more interested in retrieval by shape than by color and texture [30]. However, re- trieval by shape is still considered one of the most diĆcult aspects of content-based search. Indeed, systems such as IBM's QBIC, Query By Image Content [25], perhaps one of the most advanced image retrieval systems to date, is relatively successful in retrieving by color and texture, but performs poorly when searching on shape. The Alta Vista photo nder [6] shows similar behavior. There is no universal de nition of what shape is. Impressions of shape can be conveyed by color or intensity patterns (texture), from which a geometrical representation can be derived. This is shown already in Plato's work Meno [24]. (This is one of the so-called Socratic dialogues, where two persons discuss aspects of virtue; to memorialize Socrates, one of the gures is called after him.) In this work, the word ` gure' is used for shape. Socrates' description \ gure is the only existing thing that is found always following color". does not satisfy Meno, after which Socrates gives a de nition in \terms employed in geomet- rical problems": \ gure is limit of solid".  supported by Philips Research Laboratories 1 In this paper too we consider shape as something geometrical. Shape similarity measures are an essential ingredient in shape matching. Matching deals with transforming a pattern, and measuring the resemblance with another pattern using some dissimilarity measure. The terms pattern matching and shape matching are commonly used interchangeably. The matching problem is studied in various forms. Given two patterns and a dissimilarity measure:  (computation problem) compute the dissimilarity between the two patterns,  (decision problem) for a given threshold, decide whether the dissimilarity between two patterns is smaller than the threshold,  (decision problem) for a given threshold, decide whether there exists a transformation such that the dissimilarity between the transformed pattern and the other pattern is smaller than the threshold,  (optimization problem) nd the transformation that minimizes the dissimilarity between the transformed pattern and the other pattern. Sometimes the time complexities to solve these problems are rather high, so that it makes sense to devise approximation algorithms that nd an approximation:  (approximate optimization problem) nd a transformation that gives a dissimilarity be- tween the two patterns that is within a speci ed factor from the minimum dissimilarity. 2 Properties In this section we list a number of possible properties of similarity measures. Whether or not speci c properties are desirable will depend on the particular application, sometimes a property will be useful, sometimes it will be undesirable. Some combinations of properties are contradictory, so that no distance function can be found satisfying them. A shape similarity measure, or distance function, on a collection of shapes S is a function d : S  S ! R. The following conditions apply to all the shapes A, B, or C in S. 1 (Nonnegativity) d(A; B)  0. 2 (Identity) d(A; A) = 0 for all shapes A. 3 (Uniqueness) d(A; B) = 0 implies A = B. 4 (Strong triangle inequality) d(A; B) + d(A; C)  d(B; C). Nonnegativity (1) is implied by (2) and (4). A distance function satisfying (2), (3), and (4) is called a metric. If a function satis es only (2) and (4), then it is called a semimetric. Symmetry (see below) follows from (4). A more common formulation of the triangle inequality is the following: 5 (Triangle inequality) d(A; B) + d(B; C)  d(A; C). Properties (2) and (5) do not imply symmetry. Similarity measures for partial matching, giving a small distance d(A; B) if a part of A matches a part of B, in general do not obey the triangle inequality. A counterexample is given in gure 1: the distance from the man to the centaur is small, the distance from the centaur to the horse is small, but the distance from the man to the horse is large, so d(man; centaur) + d(centaur; horse) > d(man; horse) does not hold. It therefore makes sense to formulate an even weaker form [14]: 2 Figure 1: Under partial matching, the triangle inequality does not hold. 6 (Relaxed triangle inequality) c(d(A; B) + d(B; C))  d(A; C), for some constant c  1. 7 (Symmetry) d(A; B) = d(B; A). Symmetry is not always wanted. Indeed, human perception does not always nd that shape A is equally similar to B, as B is to A. In particular, a variant A of prototype B is often found more similar to B than vice versa [32]. 8 (Invariance) d is invariant under a chosen group of transformations G if for all g 2 G, d(g(A); g(B)) = d(A; B). For object recognition, it is often desirable that the similarity measure is invariant under aĆne transformations, illustrated in gure 2. This however depends on the application, and sometimes a large invariance group is not wanted. For example, Sir d'Arcy Thompson [31] showed that the outlines of two hatchet shes of di erent genus, Argyropelecus olfersi and Sternoptyx diaphana, are shear invariant, so that they cannot be distinguished under shear transformations, see gure 3. The following four properties are about robustness, a form of continuity. Such properties are useful to be robust against the e ects of discretization, see gure 4. 9 (Perturbation robustness) For each  > 0, there is an open set F of deformations suĆciently close to the identity, such that d(f(A); A) <  for all f 2 F . 10 (Crack robustness) For each each  > 0, and each \crack" x in the boundary of A, an open neighborhood U of x exists such that for all B, A satisfying B B g(A) g(B) Figure 2: AĆne invariance: d(A; B) = d(g(A); g(B)). 3 Figure 3: Two shear invariant hatchet shes of di erent genus: Argyropelecus olfersi and Sternoptyx diaphana. After [31]. 12 (Noise robustness) For each x 2 R 2 distance between one pattern and another does not exceed the sum of distances between the one and two parts of the other: 13 (Distributivity) For all A and decomposable B [ C, d(A; B [ C)  d(A; B) + d(A; C). The following properties all describe variations of discriminative power. The rst one says that there is always a shape more dissimilar to A than some shape B. This is not possible if the collection of shapes is nite. 14 (Endlessness) For each A; B there is a C such that d(A; C) > d(A; B). The next property means that for a chosen transformation set G, the distance d is able to discern A as an exact subset of A [ B. No g(A) is closer to A [ B than A itself: 15 (Discernment) For a chosen transformation set G, d(A; A [ B)  d(g(A); A [B) for all g 2 G. The following says that changing patterns, which are already di erent, in a region where they are still equal, should increase the distance. 16 (Sensitivity) For all A; B with A\U = B \U , B The next property says that the change from A to A[B is smaller that the change to A[C if B is smaller than C: 17 (Proportionality) For all A \ B = ; and A \ C = ;, if B