How can i make `deko` a client for `fcatalog_server` ?

[hackmoonlight]

this is deko :
https://github.com/hackmoonlight/deko

[xorpd]

Hi hackmoonlight, thank you for the interest in using fcatalog.
I think that all that you need is at the db_endpoint.py file, here:
db_endpoint.py
This code is the layer used by the IDA client, and it does not rely on anything IDA specific. Just note that it is python2. If your code is python3, you might need to port some stuff.

Take a look at the tests for live server:
live_server.py

You can test stuff against the public server testfcatalog.xorpd.net:1337

Please tell me if you bump into any specific problems.

[hackmoonlight]

thank you @xorpd , my code is python2

[hackmoonlight]

@xorpd , fcatalog use machine learning to find similarities between different binary blobs?

[xorpd]

At its core it uses the idea of Jaccard similarity coefficient. I probably wouldn't call it machine learning.
You can see the full explanation here: http://www.xorpd.net/pages/fcatalog.html

[xorpd]

Hi hackmoonlight, I got some notifications about your messages here but when I got here they were gone. Are things working for you or do you still having problems with running the server?

[hackmoonlight]

I installed the dockerized version of the server

[xorpd]

Did you manage to run the tests against your server, to make sure that it works properly?
To make sure that it works properly, you can run:

c:\python27\python.exe -m fcatalog_client.tests.live_server <host> <port>

And see that the test passes. Replace and with the relevant host and port number for your server.

[hackmoonlight]

I run this command and it works , now i am trying to modify the code to give the server the functions stored in my database for analysis :

➜  tests git:(master) python2 live_server.py 127.0.0.1 1337
----------------------------------------------------------------------
Ran 1 test in 2.111s
OK

[hackmoonlight]

I have a database and i want to give it as a dataset to fcatalog , the informations stored in DB are function data , function address and function name , i tried to create a loop to pass the information from my db to the Db of fcatalog but it did not works :

     try:
            c = lite.connect('/home/younes/Bureau/deko.db')
            cur = c.cursor()
            cur.execute("SELECT * FROM disas")
            dataa = cur.fetchall()
            i=0
            for d in dataa :
                dbe.add_function(d[0],d[1],d[3]) 
# d[0]: functions name , d[1]: function address , d[3]: functions data
                dbe.request_similars(d[3],i)
                # Check if the amount of returned functions is reasonable:
                similars = dbe.response_similars()
                self.assertEqual(len(similars),i)
                self.assertEqual(similars[0].name,d[0])
                self.assertEqual(similars[0].comment,d[1])
                self.assertEqual(similars[0].sim_grade,NUM_HASHES)
                i+=1
            c.commit()
     except lite.Error, e:
            if c:
                c.rollback()
                print "Error %s:" % e.args[0]
                sys.exit(1)
      finally:
            if c:
                cur.close()
                c.close()
        dbe.close()

[xorpd]

Hi hackmoonlight,
Could you point what didn't work for you? For example, was an exception raised, or is it just that the assertions didn't pass correctly?

I had another idea. Given that you want to build an independent system (With you own database), relying on a running fcatalog server might be too much of a hassle for you. You can just use the core algorithm "catalog1" used in fcatalog server. You can find it here:

https://github.com/xorpd/fcatalog_server/tree/master/catalog1

This algorithm compares two functions and returns a grade of similarity between those two functions. You can see the python implementation here:

https://github.com/xorpd/fcatalog_server/blob/master/fcatalog/fcatalog/catalog1.py

It is not used at the fcatalog server because this implementation is slow.

[hackmoonlight]

I just changed the type of d[0] , d[1] and d[3] and It works , and this is the modified code :

try:
            c = lite.connect('/home/younes/Bureau/deko.db')
            cur = c.cursor()
            cur.execute("SELECT * FROM disas")
            dataa = cur.fetchall()
            i=0
            for d in dataa :
                # print type(str(d[3]))
                dbe.add_function(str(d[0]),str(d[1]),str(d[3]))
                # dbe.add_function(d[0],d[1],d[3])
                dbe.request_similars(str(d[3]),2)
                # Check if the amount of returned functions is reasonable:
                similars = dbe.response_similars()
                print d[0]
                print similars
                # quit()
                # print similars
                i+=1
            c.commit()
        except lite.Error, e:
            if c:
                c.rollback()
                print "Error %s:" % e.args[0]
                sys.exit(1)
        finally:
            if c:
                cur.close()
                c.close()
        dbe.close()

the execution of the script gives us this output :

 tests git:(master) ✗ python2 live_server.py 127.0.0.1 1337
sym._init
[FSimilar(name='sym._init', comment='0x80482ac', sim_grade=16)]
sym.imp.printf
[FSimilar(name='sym.imp.printf', comment='0x80482e0', sim_grade=16), FSimilar(name='sym._init', comment='0x80482ac', sim_grade=1)]
sym.imp.puts
[FSimilar(name='sym.imp.puts', comment='0x80482f0', sim_grade=16), FSimilar(name='sym.imp.printf', comment='0x80482e0', sim_grade=7)]
sym.imp.__libc_start_main
[FSimilar(name='sym.imp.__libc_start_main', comment='0x8048300', sim_grade=16), FSimilar(name='sym.imp.puts', comment='0x80482f0', sim_grade=6)]
sub.__gmon_start___72_310
[FSimilar(name='sub.__gmon_start___72_310', comment='0x8048310', sim_grade=16), FSimilar(name='sym.imp.printf', comment='0x80482e0', sim_grade=4)]
entry0
[FSimilar(name='entry0', comment='0x8048320', sim_grade=16), FSimilar(name='sym.imp.printf', comment='0x80482e0', sim_grade=2)]
sym.__x86.get_pc_thunk.bx
[FSimilar(name='sym.__x86.get_pc_thunk.bx', comment='0x8048350', sim_grade=16)]

[hackmoonlight]

@xorpd
hello, you mentioned that the test database will be deleted every once in a while , i want to save this database , how to do it ?

[xorpd]

Sorry for the late response, @hackmoonlight.
The dockerized version of the server does not delete the database every day. This happens only on my server testfcatalog.xorpd.net, to not run out of space.

If you want to know how to back up your databases, see here :https://github.com/xorpd/fcatalog_docker#backups