Client-side image segmentation with SAM2

This is a Next.js application that performs image segmentation using Meta's Segment Anything Model V2 (SAM2) and onnxruntime-web. All the processing is done on the client side.

Demo at sam2-seven.vercel.app

small_next-sam.mov

Features

• Utilizes Meta's SAM2 model for segmentation
• onnxruntime-web for model inference
• webgpu-accelerated if GPU available and supported by browser, cpu if not
• Model storage using OPFS (not working in Safari)
• Image upload or load from URL
• Mask decoding based on point prompt (positive and negative points)
• Cropping
• Tested on macOS with Edge (webgpu, cpu), Chrome (webgpu, cpu), Firefox (cpu only), Safari (cpu only)
• Fails on iOS (17, iPhone SE), not sure why

Installation

Clone the repository:

git clone https://github.com/geronimi73/next-sam
cd next-sam
npm install
npm run dev

Open your browser and visit http://localhost:3000

Usage

1. Upload an image or load from URL.
2. Click the "Encode image" button to start encoding the image.
3. Once the encoding is complete, click on the image to decode masks.
4. Left click to include area ("positive click"), right click to exclude area ("negative click").
5. Click the "Crop" button to crop the image using the decoded mask.

Acknowledgements

• Meta's Segment Anything Model 2
• @raedle for adding positive/negative clicks
• onnxruntime
• Shadcn/ui components
• transformer.js
• https://github.com/ibaiGorordo/ONNX-SAM2-Segment-Anything
• https://github.com/lucasgelfond/webgpu-sam2
• https://github.com/microsoft/onnxruntime-inference-examples