Merge to Mondego/SourcererCC master, fixed a bug which causes tokenizer not processing files, added 'c' 'h' file extensions to the ini file#3

Open

rayrzh wants to merge 16 commits intoMondego:masterfrom

Commits on Apr 28, 2016

Added tokenizer for tar files, create mirror from token files, and minor modifications to the original tokenizer
rayrzh
committed
Added config files
rayrzh
committed

Commits on May 16, 2016

Added missing config file for tokenizer.py
rayrzh
committed

Commits on May 24, 2016

Commits on Jun 4, 2016

Merge to Mondego's master
rayrzh
committed

Commits on Jun 10, 2016

Commits on Jun 15, 2016

Move sub-scripts to src folder. tokenizer.py and ini files remain in the root level
rayrzh
committed