OpenSSE: Open Sketch Search Engine


Hello, everybody!

I'm very interested in Mathias Eitz's works on SIGGRAPH 2012, so I developed OpenSSE, Just for fun.

This is an open source sketch search engine for 3D object retrieval based on sketch image as input. In fact, it can retrieve more, such as images, videos, point cloud models and volume models, you just need get their line drawing images than I know you can.

In order to clearly show how to work, I used single thread and ascii encoding for file storage.

Zhang Dongdong

Demo - SketchSearchDemo

This demo deals with mesh models. See more search results, click demo videos.

Demo - SketchRecognize

Try via web

See more search results, click demo videos.

How to compile

OpenSSE uses Qt 5.1+, OpenCV 2.4.7+ and boost 1.55.0, it's a cross-platform library.

You need config INCLUDEPATH and LIBS in ../opensse/opensse.pri file related your path.

Now compile it, so easy!

How to use

Use OpenSSE library

You just need include one line in your .pro file.

include(../../opensse.pri)

Use tools

There are a series of tools for you under opensse/tools/. When compiled, those tools are under build-xxx-Release/bin.

Use gui

This is gui demo for you under opensse/gui.

How to train data

Database

Database in my demo comes from SHREC 2012, download 3D target dataset(~112MB). We use models under directory Watertight_dataset/Extended.

$ mv ~/Download/Watertight_dataset/Extended ~/Database/SHREC12/

Get line drawing views

Using my another project, you will get 102 views for each model.

Generate vocabulary

• Step 1: Generate line draing images filelist

$ ./generate_filelist -d ~/Database/SHREC12/ -f "*.jpg" -o ~/Database/SHREC12/2012_filelist

• Step 2: Extract descriptors

$ ./extract_descriptors -d ~/Database/SHREC12/ -f ~/Database/SHREC12/2012_filelist -o ~/Database/SHREC12/2012_

you will gain two files: 2012_features and 2012_keypoints.

• Step 3: Generate vocabulary (optional)

$ ./generate_vocabulary -f ~/Database/SHREC12/2012_features -n 1000 -o ~/Database/SHREC12/vocabulary

It will take you some times, but only need run once. or use my vocabulary under ../opensse/data/.

Create inverted index file

• Step 4: Quantize feature

$ ./quantize -v ~/Database/SHREC12/vocabulary -f ~/Database/SHREC12/2012_features -o ~/Database/SHREC12/2012_samples

When dealing with large scale database, we can directly generate samples using extract_quantize tool to reduce disk usage without feature files.

$ ./extract_and_quantize -d ~/Database/SHREC12/ -f ~/Database/SHREC12/2012_filelist -v ~/Database/SHREC12/2012_filelist -v ~/Database/SHREC12/vocabulary -o ~/Database/SHREC12/2012_samples

• Step 5: Create inverted index file

$ ./create_index -s ~/Database/SHREC12/2012_samples -o ~/Database/SHREC12/2012_index_file

Now all data are ready.

Test OpenSSE

Command line

You can test opensse through command line

$ ./sketch_search -i ~/Database/SHREC12/2012_index_file -v ~/Database/SHREC12/vocabulary -d ~/Database/SHREC12/ -f ~/Database/SHREC12/2012_filelist -n 10 -o output

You will get a interactive interface, like:

>> open sketch search :
>> input absolute path, like "/Users/zdd/zddhub.png"
>> input q exit
>> good luck!
>> /Users/zdd/Database/SHREC12/Extended/D00057view/1.jpg
0.985509 /Users/zdd/Database/SHREC12/Extended/D00057view/1.jpg
0.953196 /Users/zdd/Database/SHREC12/Extended/D00050view/1.jpg
0.938705 /Users/zdd/Database/SHREC12/Extended/D00041view/1.jpg
0.856004 /Users/zdd/Database/SHREC12/Extended/D00050view/2.jpg
0.852683 /Users/zdd/Database/SHREC12/Extended/D00057view/2.jpg
0.849087 /Users/zdd/Database/SHREC12/Extended/D00041view/2.jpg
0.823276 /Users/zdd/Database/SHREC12/Extended/D00041view/0.jpg
0.821133 /Users/zdd/Database/SHREC12/Extended/D00050view/10.jpg
0.811917 /Users/zdd/Database/SHREC12/Extended/D00057view/10.jpg
0.810096 /Users/zdd/Database/SHREC12/Extended/D00050view/0.jpg
>> q

Gui Demo - SketchSearchDemo

You must config params.json file using your data path, like:

{
    "searcher":
    {
        "indexfile": "/Users/zdd/Database/SHREC12/2012_index_file",
        "filelist": "/Users/zdd/Database/SHREC12/2012_filelist",
        "vocabulary": "/Users/zdd/Database/SHREC12/vocabulary",
        "rootdir": "/Users/zdd/Database/SHREC12",
        "results_num": "12",
        "views_num": "102"
    }
}

You'd better to use the absolute path. And then, set params.json in ../opensse/gui/SketchSearchDemo/mainwindow.cpp:

boost::property_tree::read_json("/Users/zdd/Database/SHREC12/params.json", params); 

Compiled and You will get my demo. Good luck!

Notice: To make sure line drawing images correspond to 3d models, please copy ../opensse/data/view/ to you execute file path.

Evaluation

You can use dataset and evaluation methodology to evaluate opensse on SHREC 2012, SHREC 2013, SHREC 2014.

You can generate data use test_search tool in tests/test_search.

Usage:
test_search -p paramsfile -i searchfilelist -r searchfileroot -o resultdir

To do list

• A sketch collect, share and evaluate platform
• A better search framework
• A new rank algorithm

Thanks

• Mathias Eitz
• Szymon Rusinkiewicz
• SHREC 2012 - 2014
• Tilke Judd

License

Code is under the Apache License, Version 2.0.

Donate

I accept tips through Alipay. Scan below qr code to donate:

.

Contact me

If you have any question or idea, please email to me. Or search 'zddhub' to find me in social networking platform.