Colortable (LUT)

Color segmentation/classification is an essential step in the vision system. We implement this using a Look-Up Table (LUT). The LUT is a mapping between the YUV pixel values to a color label. (The term COLORTABLE is synonymous with the LUT).

LUT Format

We use a special index created from the YUV color values to map the trained colors

The raw YUV values are three 8-bit numbers. The first thing we do is discard the 2 least significant bits from each of the channels leaving three 6-bit numbers. This reduction is done to reduce the size of possible color combinations and as a result decreasing the required memory and increasing the processing time. We make the assumption that least significant bits do not provide enough information to help in the classification and ignoring them will not greatly affect our ability to distinguish colors.

The YUV index is then just the concatenation of these 6 bit values, Y then U then V. For example:

// original YUV values
y = 0b11000011;
u = 0b00110001;
v = 0b00001110;
// discard the 2 least significant bits
y = y >> 2; // 0b110000;
u = u >> 2; // 0b001100;
v = v >> 2; // 0b000011;
// concatenate bits to create the index
index = (y << 12) | (u << 6) | v; // 0b110000001100000011;

The LUT is a array, with 2¹⁸ entries, where the value stored at each index is the color label for that index.

Color Classification

Now it is just a matter of learning the mapping between the YUV color values to the color labels we are interested in. For RoboCup the colors of interest are:

• Orange - Ball
• Cyan - Goal 1
• Yellow - Goal 2
• Green - Field
• White - Lines/Robots
• Blue - Robot Uniform 1
• Pink - Robot Uniform 1

The actual colors/names do not actually matter, just the mapping between the YUV values and whatever you define as each label.

There are many methods and theories about what the best way to learn the classification is. We provide a MATLAB program, described below, that uses supervised learning with a Gaussian Mixture Model approach to learn the classification.

MATLAB Colortable Utility

We provide a MATLAB utility in Lib/Util/Matlab/colortable to aid in this process. The utility we provide uses a Gaussian Mixture Model to train the classifier.

Log Files

The first step in training the classifier is to log images from the robot/platform you will be using. The utility takes in an WxHxN array of the YUYV image data. If you use the MATLAB logger we provide you can use the convert_logs utility to extract the images

>> % The function takes in the string path that contains the saved log files.
>> %   For each 'log_*.mat' file in the directory it will extract the 
>> %   yuyv image data and save the image array as 'yuyv_*.mat'
>> % The 'yuyv_*.mat' file is what you will load into the colortable utility
>> convert_logs('path/to/log/files');

Training the Colortable

1. Start the colortable gui:

>> colortable
initializing global COLORTABLE struct...done

The following GUI should open:

Colortable GUI

1. Click the __Load Montage__ button in the top left and select the yuyv montage you want to load:

Load Montage

1. For each image in the montage click on the pixels that correspond to each color. * Select the color from the radio button list. * Click on pixels in the image that you want to be classified as that color.

* There are a couple different types of clicks you can do: * __\__ - Select the pixels as positive examples for that color. * __\ + \__ - Select the pixels as negative examples for that color. * __\ + \__ - De-select the pixels (both positive and negative). * Press the left and right arrow buttons to change images. 1. After you have gone through all the images and indicated the color labels, close the GUI.

All of the data collected during the training is stored in the global struct COLORTABLE. You can use the Save Colors button to save this struct. You can load the data to use later, e.g. if you collect more logs and wish to augment your current training data.

If you want to use a saved colortable just load the struct before starting the colortable GUI:

>> load('my_colortable.mat');
>> colortable

The utility will then use that colortable data instead of starting a new one.

Create the LUT

Now that you have the training data you can create the LUT to use for the color classification.

1. Use the Gaussian smear function to create the mixture model:

>> % the function accesses the global COLORTABLE struct directly 
>> %   so no arguments are needed
>> colortable_smear();

1. Create the actual LUT array from the mixture model:

>> % the function accesses the global COLORTABLE struct directly 
>> lut = colortable_lut();

1. Finally, save the LUT as a binary file so that the Lua code can load and use it:

>> write_lut_file(lut, 'my_luar_lut.raw');

Verify the LUT Quality

It is a good idea to verify the quality of the LUT that you created. A poor LUT will result in poor performance of the vision algorithms. There is a trade off between the LUT's ability to generalize and discrimination between the color classes. It is somewhat dependent on the target application so there is no quantitative way to compare classifications and that determination will be up to the user.

We provide another utility to help visualize the LUT classification you produced by plotting the 3-D YUV classification. This is accomplished by transforming the index based LUT into the 3-D YUV colorspace representation. We then take slices of the resulting cube at set Y (luminescence) values. This produces a series of images showing the U-V plane for that Y value.

To generate this display use lut_montage.m:

>> lut_montage(lut);

LUT Montage

This visualization is helpful in determining the LUT's ability to discriminate between the color classes. The display shows the color classification in the YUV colorspace, neighboring pixels in the image are similar to each other in color. The closer together that the classification regions are then the more similar those color classes are. If the classes get too close, or are touching/adjacent to each other, it means that the class label for pixels in that region are ambiguous. Ideally you want to see some separation between all of the color classes.

NOTE: Depending on the color classes you are using and the environment it may be impossible to get a perfect separation of the color classes as shown in the example LUT montage. The color classifier will have a limit in the information it can provide and will only get you so far in detecting objects.

Colortable Recommendations (2019 Onward)

With the addition of sunlight to the field in 2018 as well as other prior color changes, the vision module has moved away from handcrafted models and towards deep learning. Thus, the need for the Gaussian Mixture Model (GMM) to classify has diminished greatly and now is only used to distinguish between green and white (2019), mostly to detect field lines. However, it is still important to have high quality colortables, so below are some tips on how to make a high quality colortables.

Due to the GMM being a semi-supervised method, it has close to zero-ability to reject False Positives As a result, you should avoid at all costs mislabeling more than a handful of pixels for the entire training process. It is FAR better to not label, than to include a mislabel so don't worry about missing a bunch of pixels in an image. If you can't tell, just don't label. For example, a single grey/white pixel in the field of grass is often a reason not to label that pixel; it could be reflected light that hit the camera in just the right way.

Make note of sunlight and camera configurations. Having high gain and/or exposure time limits the ability of the robot to see bright regions. The camera should be configured to make the image as bright as possible, but not so bright as to have washed out regions that max out the pixel values (100% white pixels, no grey). Lowering these settings so that there are very few maxed pixels is extremely beneficial. Sometimes this will mean having a dark image, which is totally fine. The GMM will be able to handle this. Also make sure your NNs for other parts of the vision pipeline will be able to handle this.

Play with color temperature and include several in your training datasets. Color temperature shifts the image between blue and yellow to compensate for lighting conditions. During play, you should set this to have a perfect white balance. However, during colortable training, including variations in this setting make you colortables more robust to different lighting conditions. For example, in 2019 the field was located next to a blue tinted window and clouds from outside would greatly shift the color balance between the blue outdoor light and warm interior light.

Don't be afraid to have multiple colortables, but at the same time, you probably don't need one for every field. If you practice the above recommendations, you will have very robust colortables that will work for all but the most extreme cases. You will probably make a colorable for interior lighting conditions and one for outdoor/sunlight lighting. If you run into a case like I had with tinted-windows, maybe one for that too. With good enough colortables, your main priority should be the camera configurations.