COSARAD is an applied research project for anomaly detection in point clouds.
COSARAD's primary goal is to detect geometric defects such as cracks, holes, and bulges in industrially manufactured parts. It has been designed at the Mechatronics, Automation and Control Systems lab MACS at the University of Washington. COSARAD is under active development, and the repository structure may change. However, we are doing our best to maintain the documentation up to date.
COSARAD compares a test shape, which may or may not contain defects, to a set of defect-free reference shapes. All object shapes are represented by their point clouds (PCs). It is a patch-based detector: we register reference and test shapes to the common coordinate frame, and then uniformly sample small patches from these shapes. Patches from the test shapes are then compared to the patches from the reference shapes. Significant deviations in the geometry of patches are considered a sign of an anomaly.
Of course, explicitly comparing the geometries of patches is unrealistic. Patches are unordered sets of 3D points that cannot be compared directly. Instead, we compare the representation vectors for these patches. Representations are extracted by the feature extractor network—a deep learning network that ingests a point cloud of the patch and outputs a d-dimensional vector that characterizes the patch geometry.
The key features of the COSARAD are:
- Contrastive feature extraction. We formulate anomaly detection as a semi-supervised contrastive learning problem and develop a deep feature extractor highly optimized for detecting geometric anomalies. Our extractor minimizes the distance between representations of geometrically similar patches and maximizes the distance between anomalous representations.
- Spatially aware patch comparison. We design an anomaly detection algorithm that maintains a precise alignment of a test object with the shapes of reference objects. We store representations of reference objects in a multitude of small local memory banks, each bank containing features extracted from a specific location of an object. We compare a patch originating from a “test” object only against the contents of the corresponding local memory bank. In other words, we preserve an exact spatial (i.e. location) alignment of patches during the comparison. And, by comparing a test object to a number of different reference shapes we make COSARAD robust to the representations’ noise.
We train and evaluate COSARAD on two public datasets for anomaly detection in 3D: Real3D-AD and Anomaly-ShapeNet v.2. We use the Area under the ROC curve (AUROC) as a measure of the object-wise detection accuracy (O-ROC), i.e. how accurately COSARAD detects anomalous objects; as well as for the point-wise anomaly localization (P-ROC), i.e. how accurately we localize the anomalous points within a PC.
To the best of our knowledge, COSARAD is more accurate on these data than any other method for 3D anomaly detection. The table below compares our results on the Real3D-AD dataset with other popular anomaly detection methods. Results are reported as O-ROC / P-ROC. The best result is in bold, second best in italics. COSARAD achieved highest performance on 8 out of 12 objects.
| Object | Reg3D-AD | PO3AD | Simple3D | MC3D-AD | PointCore | COSARAD |
|---|---|---|---|---|---|---|
| Airplane | 71.6/63.1 | 80.4/- | 76.5/88.1 | 85.0/62.8 | 66.0/60.8 | 91.2/96.4 |
| Candybar | 82.7/72.2 | 78.5/- | 85.1/96.2 | 77.8/73.6 | 97.6/76.0 | 99.9/98.2 |
| Car | 69.7/71.8 | 65.4/- | 98.1/99.2 | 74.9/81.9 | 86.6/70.6 | 96.5/98.2 |
| Chicken | 85.2/67.6 | 68.6/- | 82.6/86.1 | 71.5/64.0 | 84.1/78.0 | 85.4/87.4 |
| Diamond | 90.0/83.5 | 80.1/- | 100/99.0 | 95.5/94.2 | 96.3/81.0 | 100/97.5 |
| Duck | 58.4/50.3 | 82.0/- | 78.7/96.6 | 83.1/82.2 | 68.4/71.2 | 94.8/97.0 |
| Fish | 91.9/85.2 | 85.9/- | 91.2/99.6 | 91.2/90.6 | 99.2/78.2 | 99.4/96.2 |
| Gemstone | 41.7/54.5 | 69.3/- | 70.4/97.3 | 56.0/45.8 | 53.4/51.5 | 97.1/97.4 |
| Seahorse | 76.2/81.7 | 75.6/- | 93.0/94.2 | 90.1/95.0 | 97.3/84.1 | 80.5/96.5 |
| Shell | 35.8/81.1 | 80.0/- | 85.1/97.6 | 51.5/47.1 | 86.1/78.1 | 79.1/91.2 |
| Starfish | 50.6/71.7 | 75.8/- | 69.5/85.8 | 76.6/69.0 | 65.2/73.6 | 84.4/95.0 |
| Toffees | 68.5/75.9 | 77.1/- | 88.9/96.8 | 78.3/93.4 | 92.9/74.5 | 88.7/92.5 |
| Mean | 70.4/70.5 | 76.7/83.6 | 80.4/92.3 | 78.2/76.8 | 82.9/73.1 | 91.6/95.3 |
Anomaly-ShapeNet contains as many as 60 different objects, so we instead provide averaged results as a distribution of O-ROC and P-ROC values for each different object type.
COSARAD has two distinct modes of operation: training and inference. The inputs to the training stage are known object classes, their point clouds (PCs), and the anomaly masks—a set of all the anomalous points in a given PC.
During the training, we train a feature extractor that minimizes the similarity between the representations of "good" (i.e. anomaly-free) and "bad" (anomalous) patches, and maximizes similarity between representations of anomaly-free patches that are extracted from exactly the same location of a point cloud.
We start by registering all the PCs into the same coordinate frame. Then, we extract patches from the input PCs. This includes patches that contain an anomaly (i.e., "bad" ones), as well as anomaly-free, "good" patches. Next, we arrange patches into contrastive tuples—a small set of patches that contain both "good" and "bad" patches; such a tuple structure is required for contrastive training. Then, we train a feature extractor network using a combination of contrastive losses. The network itself is based on the RIConv++ rotation-invariant convolutional backbone for 3D point clouds, although PointNet++ can be used as well.
During inference, COSARAD first loads the set of anomaly-free reference shapes and registers them into the common coordinate frame. It extracts patches from the reference shapes, ensuring the uniform coverage of an object by patches. It feeds patches to the trained feature extractor, and stores the extracted features into memory banks. For each patch location, we have a separate local memory bank that contains features extracted from all the reference patches at that location.
Similarly, COSARAD loads the test object and registers it into the same coordinate frame as the reference objects. Next, it selects patches (from exactly the same locations as for the test objects) and retrieves patches' representations. Finally, an anomaly detection algorithm compares patches in a spatially-aware manner to localize anomalies. The result of the comparison is the object anomaly score (the higher the score, the more likely the object contains a defect) and anomaly scores for individual points. In this manner, we detect and localize anomalies for the possible corrective actions.
Below we provide documentation to facilitate re-use of our research. We also provide brief instructions how to reproduce our results. Please contact us if you encounter issues with our code or results.
The training and evaluation pipeline is documented in doc/training_validation_pipeline.md. It covers the full workflow (cross-validation training, inference, statistical comparison against a baseline), preset tiers, run IDs, stage design, and common workflows.
We maintain Docker images for training and evaluation of COSARAD on different GPU architectures, ranging from Ada to Blackwell. A "bare metal" setup can be done through Conda / Mamba configuration files, however this setup will require some manual steps. We provide detailed instructions for setting up the environment.
COSARAD has been trained on data extracted from two publicly available datasets:
- Real3d-AD
- Anomaly-ShapeNet v.2 We sincerely thank the authors and maintainers of these datasets for their contribution to the anomaly detection community.
Practical demand drives research interest in defect detection, and new datasets appear regularly. To make COSARAD easily extensible with new data, we maintain our own intermediate dataset format. We also provide a description of a pipeline that generates the dataset for training the contrastive representation encoder.
COSARAD requires a GPU with at least 32 Gb of VRAM for training and at least 16 Gb for evaluation. See Training instructions.
We provide instructions to evaluate our work on objects extracted from Real3D-AD and Anomaly-ShapeNet datasets: evaluation instructions.
At the moment, we are refactoring the code to enable stand-alone inference, so users can pass COSARAD a set of reference shapes as well as test shapes and obtain anomaly scores for each object. Alongside the object-wide anomaly score, COSARAD will return the per-point anomaly score, which enables precise localization of anomalies. We plan to release a ROS2 node wrapper that will enable integrating COSARAD into a live robotic system for industrial defect detection.