The main goal of the contest is to provide a framework for the evaluation of different methods for symbol recognition in graphic documents. This framework is intended to be general and flexible so that it can be used to evaluate a wide range of symbol recognition methods.
The contest is based on a pre-defined set of symbols, whose description is available through this web site (symbol description). Using this set of symbols, different tests will be generated, consisting of several test images with increasing levels of degradation and distortion. Several tests are already available to the participants through this web site (sample tests). These tests can be used as a learning set to train the methods participating in the contest. For real contest execution, other tests containing different images will be generated, but they will be similar to these sample tests.
Constest will take place just before the GREC'11 worshop, on July, 4th. Each participant will be allowed to run his/her method on the selected set of tests he/she decides to take part in, depending on the characteristics of the method and the level of degradation and distortion of the test. It is not obliged for a participant to run his/her method on all evaluation tests. Files formats needed by the participants for the contest are available here.
As final result of the contest, we will provide several tables, one for each kind of test, showing the performance of the methods taking part in those tests. Performance will be measured in terms of recognition accuracy. Computation time is another optional performance measure which can be provided by participants. We will not provide a global measure of performance for every participating method as it is difficult to combine results in different kinds of tests.
Files formats needed by the participants for the contest are available.
Description Of Symbols
All symbols used in the contest are symbols which can be found in drawings belonging to two different domains: architecture and electronics. All of them are only composed of two types of primitives: lines and arcs. They do not contain filled regions, text or any other kind of primitive.
A description of the ideal shape of each symbol is available in this web site (symbol description). We provide two different representations for every symbol: a binary image in TIFF format corresponding to a printing of the ideal shape of the symbol, and a vectorial description in VEC format. VEC format has been used in previous contests held in past editions of GREC workshop.
Description Of Evaluation Tests
Several tests will be generated to evaluate the performance of the methods participating in the contest. Some of these tests are available in this site (sample tests), prior to the contest, to let the participants train their methods. Real evaluation tests will only be available during the execution of the contest. They will be similar to training sets.
Each evaluation test will consist of a set of model images and a set of test images. Each one will contain different types of images according to the main following criteria:
- Domain of application: electronics and architecture.
- Image degradation: several types of degradation will be proposed, resulting from the combination of affine transformations, binary noise and shape distorsions.
- Segmentation: test data will consist of pre-segmented images as well as real complete drawings with non-segmented symbols.
- Scalability: several datasets including an increasing number of symbols will be defined in order to evaluate scalability.
Images for every test will be provided in two formats: as binary images in TIFF format and as vectorial images in VEC format. Vectorial images will be generated by application of our own vectorization method. Each participant can choose to work on binary images or on vectorial images, depending on his/her method.
Each test will be described by a file (test format) which will specify the label of all symbols involved in the test, the path where the descrition of these symbols can be found, and the name and the path of all images involved in the test. Participants must read the file describing the test and run their method on every image of the test, yielding a final result file in the format specified in the section "formats" (result format).
The contest will be held on July, the 4th, some weeks before the beginning of the ICDAR'11 conference. Then, real evaluation tests will be made in public during the GREC'11 Workshop, and each participant will decide in which categories he/she will take part in.
Participants are encouraged to come with their own laptop to run the tests on it. In case some participant could not take his/her own laptop, he/she should contact the organization to look for an alternative way of executing the contest.
Each participant must prepare an application able to read a file describing a test, run his/her method on all images described in the test and generate a result file in the specified format.