Static Segmentation by Tracking: A Frustratingly Label-Efficient Approach to Fine-Grained Segmentation

Imageomics Institute

Zhenyang Feng, Zihe Wang, Saul Ibaven Bueno, Tomasz Frelek, Advikaa Ramesh, Jingyan Bai, Lemeng Wang, Zanming Huang, Jianyang Gu, Jinsu Yoo, Tai-Yu Pan, Arpita Chowdhury, Michelle Ramirez, Elizabeth G Campolongo, Matthew J Thompson, Christopher G. Lawrence, Sydne Record, Neil Rosser, Anuj Karpatne, Daniel Rubenstein, Hilmar Lapp, Charles V. Stewart, Tanya Berger-Wolf, Yu Su, Wei-Lun Chao

[arXiv] [Dataset] [BibTeX]

🗓️ TODO

• Release inference code
• Release beetle part segmentation dataset
• Release online demo
• Release one-shot fine-tuning (OC-CCL) code
• Release trait retrieval code
• Release butterfly trait segmentation dataset

🛠️ Installation

To use SST, the following setup must be ran on a GPU enabled machine. The code requires torch>=2.5.0, and python=3.10.14 is recommended. (Note: Make sure your system is using GXX and GCC compilers)

Example Conda Environment Setup:

# Clone repo
git clone https://github.com/Imageomics/SST.git
cd SST
# Create conda environment
conda create --name sst python=3.10.14
conda activate sst
# Download the version of PyTorch that matches the GPU's CUDA version, see https://pytorch.org/get-started/locally/
pip3 install torch torchvision torchaudio --index-url ...
# Download and setup GroundingDINO
(git clone https://github.com/IDEA-Research/GroundingDINO.git && cd GroundingDINO/ && pip install -e .)
# Install SAM 2
(cd segment-anything-2 && pip install -e .)
# Download required python packages, ignore the conflicts message
pip install -r requirements.txt --no-dependencies
# Download model checkpoints
(cd checkpoints && ./download_ckpts.sh)
(cd checkpoints && wget -q https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth)

🧑‍💻 Usage

Segmentation

For one-shot trait/part segmentation, please run the following demo code:

python code/segment.py --support_image /path/to/sample/image.png \
  --support_mask /path/to/greyscale_mask.png \ 
  --query_images /path/to/query/images/folder \
  --output /path/to/output/folder \
  --output_format "png" # png or gif, optional

Trait-Based Retrieval

For trait-based retrieval, please refer to the demo code below:

python code/trait_retrieval.py --support_image /path/to/sample/image.png \
  --support_mask /path/to/greyscale_mask.png \ 
  --trait_id 1 \ # target trait to retrieve, denote by the value in support mask  \
  --query_images /path/to/query/images/folder \
  --output /path/to/output/folder \
  --output_format "png" \ # png or gif, optional
  --top_k 5 # n top retrievals to save as results

📊 Dataset

Beetle part segmentation dataset is out! Available here. Butterfly trait segmentation dataset can be accessed here!

❤️ Acknowledgements

This project makes use of the SAM2 and GroundingDINO codebases. We are grateful to the developers and maintainers of these projects for their contributions to the open-source community. We thank LoRA for their great work.

📝 Citation

If you find our work helpful for your research, please consider citing using the following BibTeX entry:

@misc{feng2025staticsegmentationtrackingfrustratingly,
      title={Static Segmentation by Tracking: A Frustratingly Label-Efficient Approach to Fine-Grained Segmentation}, 
      author={Zhenyang Feng and Zihe Wang and Saul Ibaven Bueno and Tomasz Frelek and Advikaa Ramesh and Jingyan Bai and Lemeng Wang and Zanming Huang and Jianyang Gu and Jinsu Yoo and Tai-Yu Pan and Arpita Chowdhury and Michelle Ramirez and Elizabeth G. Campolongo and Matthew J. Thompson and Christopher G. Lawrence and Sydne Record and Neil Rosser and Anuj Karpatne and Daniel Rubenstein and Hilmar Lapp and Charles V. Stewart and Tanya Berger-Wolf and Yu Su and Wei-Lun Chao},
      year={2025},
      eprint={2501.06749},
      archivePrefix={arXiv},
      primaryClass={cs.CV},
      url={https://arxiv.org/abs/2501.06749}, 
}