Future erlog implementation

[rvirding]

There has been some discussion on how to develop erlog. Two issues have arisen:

• The basic API and whether to have a server interface or not, and if so what should it look like.
• The interface to the database.

My views on these are:

Erlog should only contain a basic local functional interface and no server interface. Users can write their own servers when necessary and choose which interface they want/need. The erlog interface should work at the erlang data level; users can build other interfaces on top of this, for example using the erlog parser to build a string interface.

The database interface should be abstracted out of the interpreter. My current idea is that there is a predefined function interface to the database and when you start the interpreter you give the name of the callback module to access the database. This would work in much the same way as OTP behaviours. This would allow using dict (the default) or ETS directly, or in a server, or a more complex system where different types of procedures are stored in different ways.

My reason for keeping both interfaces very basic is to make it easier for users to plug in specific code. There is no way that erlog can cater to all the different ways users may want to interface it so I feel it is better to keep it basic and provide support to extend it. Hopefully by keeping it basic we can make something everyone can use and specialise for their own needs.

(Should we have the discussion in the mailing list instead?)

[comtihon]

Eh, I removed server interface, as I thought it is useless, but zkessin convinced me to return it back)
I agreed with db interface, as I have already done this:

Proplist = [{database, mysql_storage_impl_module}],
erlog:start_link(Proplist).

[zkessin]

Right now I have a bunch of quickcheck properties that use the server interface, those can be converted over to use the function interface if needed (and I probably will)

In addition I am working on an external project https://github.com/zkessin/erlog-server which will let you take a prolog file and build a custom gen_server around it. That way you can take a prolog file and define what it exposes to erlang and it will be compiled into a beam file that will work like any other erlang gen_server but will be running prolog internally.

It is still in early phases at this point

[rvirding]

I will seriously start working on it after the the OSCON conference for which I am preparing a talk + demo. It won't contain erlog, though it could, but ny Lua implementation instead. Then we can discuss details.

[comtihon]

I can't understand. You will leave functional interface, but remove server interface?

[zkessin]

I think we should view the function interface as a low level building block, which may be best used to create other more specific ways of interacting with erlog.

On top of that we can then create a basic gen_server, specific servers that map to prolog code, event handlers or whatever we want.

[comtihon]

Function interface is awful for developing, debugging and support. If you decide so - I have to make my own branch, as I don't take this function code to our production systems.

[zkessin]

It is a clunky interface, but it has the advantage of being able to be easily embeded in something else. I don't plan to use it directly but to use it as a building block of a much higher level system.

[zkessin]

I think the idea is that we will create several high level interfaces that can serve different uses. I would not want to use the function interface directly at the REPL, but if I am embeding erlog in webmachine it would probably be idea

[rvirding]

OK, here are my thoughts on the "new" erlog interface as they stand so far. Valery when you mention the "functional interface" if you mean the interface through funs then it is going. I never liked it much myself and I only added it as I had two interfaces in the same module and this was the easiest way to keep them apart without having too long names. The new interface is NOT backwards compatible with the existing one.

The call API is very basic and only contains what is really necessary. The idea is that users can create their own more specific interfaces on top of this when necessary. There is also no predefined server interface. The rationale for this is that it is very difficult to reach some form of accord how these types of interfaces should look. We are not in agreement. :-) So I will provide tools rather than solutions.

The interface is in two parts, the main calls into the erlog engine and the database interface. The module erlog contains the calls to the engine:

erlog:new(DbModule, DbInitArgs) -> {ok,ErlogState}.

erlog:prove(ErlogState, Goal) -> {succeed,VarBindings} | fail {error,Error}.

erlog:next_solution(ErlogState) -> {succeed,VarBindings} | fail {error,Error}.

erlog:consult(ErlogState, FileName) -> ok | {error,Error}.

erlog:reconsult(ErlogState, FileName) -> ok | {error,Error}.

erlog:load(ErlogState, ModuleName) -> ok | {error,Error}.

This loads in an erlang module containing erlog procedures, both interpreted and compiled through a defined interface into the database.

erlog:get_db(ErlogState) -> {ok,DbRef}.

erlog:set_db(ErlogState [, DbModule], DbRef) -> ok.

This allows changing both the DbRef and the DbModule name on the fly. It cannot be done while stepping over solutions, the call context is lost.

I am not certain about the consult/reconsult calls but I see no better way to load in files written in prolog. The DbRef is what is handled by the DbModule. These are the main functions, there will be some useful service functions as well. Should the ErlogState argument be first or last?

The database is accessed through the DbModule module which is specified through the erlog:new call and managed in the erlog engine.

DbModule:new(DbInitArgs) -> DbRef.

Called when the erlog state is initialised.

DbModule:add_built_in(Functor, DbRef) -> DbRef.

DbModule:add_compiled_code(Functor, Mod, Func, DbRef) -> DbRef.

DbModule:asserta_clause(Clause, DbRef) -> DbRef.
DbModule:assertz_clause(Clause, DbRef) -> DbRef.

DbModule:retract_clause(Functor, ClauseTag, DbRef) -> DbRef.

DbModule:abolish_clause(Functor, DbRef) -> DbRef.

DbModule:get_procedure(Functor, DbRef) ->
    built_in | {compiled,Mod,Func} | {clauses,Clauses} | undefined.

DbModule:get_procedure_type(Functor, DbRef) ->
    built_in | compiled | interpreted | undefined.`

DbModule:get_intepreted_functors(DbRef) -> [Functor].

The DbRef is some reference to the database, it could be the dictionary itself or an ETS table or a pid to a database server or even a combination of different storage means. This should allow users to easily define their own storage method. These functions should generate errors so there are no tagged return values. Should the DbRef argument be first or last?

Anyway these are my ideas as they stand now.

[comtihon]

Please see my erlog interfase in develop branch, as I came to a conclusion, that it is optimal.
Instead erlog:new/2 I use simple start_link (as it is otp gen_server). It can be without params (all will be default) or with one param - proplist, which contains all implementations, handlers and pids of customisations. It return {ok, Pid}.
Second - I use execute/2 instead prove_goal/2, but they are the same.
And last interface is select/2. It is analog of next_solution/2.
I renamed them, as I think now they named as they do.
consulting and reconsulting can be done through prolog, I added bifs consult/1 and reconsult/1. And can be done in erlang through [finenames_list].
I do not understand, why should we have setdb and getdb. Can you give me an example? I think if you need other database - just start new erlog worker.
And I can't understand usage of load. Are they for loading libraries, such as erlog_bips, erlog_dcg and erlog_lists? But they are loaded, when server is starting. Or you plan to load them dynamically, for not storing them in memory, when libs quantity will be much more? If so - prolog term for loading libs can be more useful.
As for database - I just added findall/2 for findall, bagof, setof and it is important.

For your question - DbRef is a Pid of dbserver and should be the first argument, as it is maid in otp.

[zkessin]

I use get_db and set_db in the erlog-server code I am working on. I create an erlog instance load some stuff into it and then export the DB and actually put it in a beam file to be re-hydrated later. So that you can start erlog in a known state. In short it lets me include a prolog file into a beam file

[zkessin]

I like @rvirding 's comments above.
I can see arguments where ErlogState could be a PID or an Immutable data structure.

The question is do we want to make it a fun that returns a new fun or maybe we want to make it a tuple module so it will return {erlog, State}

If its a PID we should have some standard protocol which a process can implement to work with it I think.

[rvirding]

Seeing we both want a server based interface and a purely local interface I think it is better that the base system provide the local interface, and at a basic level. With that it is not difficult to build a server based interface when necessary, or have a different interface.

get_db and set_db are easy to implement and useful if you want to work with the whole database at once.

The load interface is for loading libraries written in erlang like erlog_lists. While some standard libraries are loaded at startup time this would allow you to write library modules in erlang and load them into erlog. They can be a combination of compiled code and interpreted code as is done erlog_lists. I will define the interface properly as I go now.

There is a findall/3 in the new version in develop, but no bagof/3 or setof/3 yet.

The ErlogState is an opaque datastructure and should never be accessed directly. It is solely used by the interpreter. It contains the database module and reference plus everything else the interpreter needs to continue its computation, for example the current choice points and variable bindings. These you should never directly fiddle with.

Note that the database reference is opaque to the interpreter, it uses the API in the database module to access it.

[comtihon]

If you want to fill database and than move them to another system - just don't use ets or dict. I made behaviour callbacks for this purpose. Try dets or sqlite/h2.
I understand - what load is for. I'll make it.
But we should think of - how it will be loaded from prolog code. Maybe special include world? Or automatically?
As now I am quickly develop erlog I advice you to view my code first (for new features), and then write your own, as it can turned out we make same job 2-3 times.

[zkessin]

Why not use dict? it would seem an ideal pure functional data structure if you want to export it, esp if you want to embed that data in a larger file.

[rvirding]

Using dict will be the default if you just call erlog:new() without specifying DbModule and DbInitArgs.

[rvirding]

Should the database interface given in DbModule actually generate errors using the calls in erlog_int, or should it just return an error value, error, and let the interpreter generate the proper error? I am leaning towards the latter as it will standardise things.

[zkessin]

I like the return error plan.

[rvirding]

So do I so I will probably go with that. It will also make the database implementation modules easier to write. Though they will still have to follow the rules for what is illegal, for example if a functor had been declared a built-in you can't add clauses to it or make it compiled.

[rvirding]

OK I have now pushed a first version of the new erlog interface in the branch new-interface. It is not too far from my suggestion earlier. Some comments:

• The database has been abstracted out pretty much as I described earlier. There is a default version using dict, erlog_db_dict, and another using ETS, erlog_db_ets, these can also be used as examples of writing your own database.
• A first version of the documentation of the erlog interface and the erlog database interface is in doc/erlog.txt.
• There is no defined erlog server, if you want one then you define it yourself and get the interface you want.
• I have included a simple erlog shell both as a useful interface and as an example of writing an erlog server. See doc/erlog_shell.txt for a description of the trivial API.
• I intend to improve the i/o to give a better API and eventually to be able to add operators in the prolog way.

Note that this is still experimental but it is very much the way I want to go.

[rvirding]

Forgot to mention one thing that definitely will be coming and that is more properties of procedures, for example of whether they are dynamic or not.

[zkessin]

I am going to merge this into my testing branch to make everything pass (today or tomorrow if I have time)

I was planning to add some IO predicates to match SWI Prolog's as much as possible

[rvirding]

OK but take it slowly as things may change as we test it. I am going to try
and implement Valery's database to test that it works. After that we can
feel a bit more comfortable.

I was planning to follow the i/o procedures in standard prolog but they
should be close. Also to look at the standard prolog module system which a
little different. After all that we are almost big time. We shall see.

Take it slowly until this reaches the develop branch.

From my Nexus
On Jul 30, 2014 5:14 AM, "Zachary Kessin" [email protected] wrote:

I am going to merge this into my testing branch to make everything pass
(today or tomorrow if I have time)

I was planning to add some IO predicates to match SWI Prolog's as much as
possible

—
Reply to this email directly or view it on GitHub
#18 (comment).

[zkessin]

Fixing the tests was pretty easy, it was mostly just refactor to the New API, none of them look at the internals

[rvirding]

All that I see is left to do is to add a property list to procedures in the DB and modify code to test for them. For example 'dynamic'.