Skip to content

Create multiCleaner.py#180

Open
garanews wants to merge 4 commits into
spender-sandbox:masterfrom
garanews:master
Open

Create multiCleaner.py#180
garanews wants to merge 4 commits into
spender-sandbox:masterfrom
garanews:master

Conversation

@garanews

Copy link
Copy Markdown
Contributor

This script will delete samples (present into mongodb and on disk) older than X seconds (find in delta variable), with a malware score less than Y value (find in MAX_MALSCORE) and with no virustotal evidences (0 is considered a report).
Before run it, some index into monngodb is needed:
mongo cuckoo then
db.calls.createIndex({"pid": 1})
db.analysis.createIndex({"virustotal_summary": 1, "info.ended": -1})
db.analysis.createIndex({"malscore": 1, "info.ended": -1})
db.analysis.createIndex({"virustotal_summary": -1, "info.ended": -1})
db.analysis.createIndex({"malscore": -1, "info.ended": -1})

It is working, need to fix the duplicates id output from mongo:
db.analysis.find( {$and : [{"malscore": {$gt : -1}}, {"malscore": {$lt : 4}}], "info.ended": {$lt : "2016-06-23 10:00:42"}, "virustotal_summary": {$exists: false } } , {"info.id": 1} )

{ "_id" : ObjectId("576b9ae39992744853f44f2e"), "info" : { "id" : 11 } }
{ "_id" : ObjectId("576b9a3599927449392608ef"), "info" : { "id" : 11 } }
{ "_id" : ObjectId("576ba3949992744937261c39"), "info" : { "id" : 26 } }
{ "_id" : ObjectId("576ba3bf9992744853f4803b"), "info" : { "id" : 26 } }

in order to avoid the double try removal, if "debug = true" will see:

python multiCleaner.py
2016-06-23 15:32:06 starting
found 66 samples
/home/analyst/cuckoo-advanced/storage/analyses/11
deleted: 11
/home/analyst/cuckoo-advanced/storage/analyses/11
[Errno 2] No such file or directory: '/home/analyst/cuckoo-advanced/storage/analyses/11'
/home/analyst/cuckoo-advanced/storage/analyses/26
deleted: 26
/home/analyst/cuckoo-advanced/storage/analyses/26
[Errno 2] No such file or directory: '/home/analyst/cuckoo-advanced/storage/analyses/26'

This script will delete samples (present into mongodb and on disk) older than X seconds (find in delta variable), with a malware score less than Y value (find in MAX_MALSCORE) and with no virustotal evidences (0 is considered a report).
Before run it, some index into monngodb is needed:
mongo cuckoo then
db.calls.createIndex({"pid": 1})
db.analysis.createIndex({"virustotal_summary": 1, "info.ended": -1})
db.analysis.createIndex({"malscore": 1, "info.ended": -1})
db.analysis.createIndex({"virustotal_summary": -1, "info.ended": -1})
db.analysis.createIndex({"malscore": -1, "info.ended": -1})

It is working, need to fix the duplicates id output from mongo:
db.analysis.find( {$and : [{"malscore": {$gt : -1}}, {"malscore": {$lt : 4}}], "info.ended": {$lt : "2016-06-23 10:00:42"}, "virustotal_summary": {$exists: false } } , {"info.id": 1} )

{ "_id" : ObjectId("576b9ae39992744853f44f2e"), "info" : { "id" : 11 } }
{ "_id" : ObjectId("576b9a3599927449392608ef"), "info" : { "id" : 11 } }
{ "_id" : ObjectId("576ba3949992744937261c39"), "info" : { "id" : 26 } }
{ "_id" : ObjectId("576ba3bf9992744853f4803b"), "info" : { "id" : 26 } }

 in order to avoid the double try removal, if "debug = true" will see:

python multiCleaner.py
2016-06-23 15:32:06 starting
found 66 samples
/home/analyst/cuckoo-advanced/storage/analyses/11
deleted: 11
/home/analyst/cuckoo-advanced/storage/analyses/11
[Errno 2] No such file or directory: '/home/analyst/cuckoo-advanced/storage/analyses/11'
/home/analyst/cuckoo-advanced/storage/analyses/26
deleted: 26
/home/analyst/cuckoo-advanced/storage/analyses/26
[Errno 2] No such file or directory: '/home/analyst/cuckoo-advanced/storage/analyses/26'
@doomedraven

Copy link
Copy Markdown
Contributor

@garanews

Copy link
Copy Markdown
Contributor Author

I have seen, the reason I created new one is to keep samples with a malscore greater than X value and/or positive to VT for comparison (malheur) etc
In that script you can set only days...

garanews added 2 commits May 12, 2017 09:04
Whitelisting  adobe.com in misp.conf will whitelist both domains and urls containing adobe.com domain:
 
skipped domains from misp submit: adobe.com , acroipm2.adobe.com 
skipped urls from misp submit: http://acroipm2.adobe.com/15/rdr/ITA/win/nooem/none/consumer/279_15_20_20039.zip
@doomedraven

Copy link
Copy Markdown
Contributor

check this one https://github.com/doomedraven/cuckoo-modified/blob/master/utils/remove_older.py

there also your misp improvements merged

support non utf8 chars in filename
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants