dlss

DLSS is a distributed key-value storage for Erlang/OTP applications.

Features

• Keys and values are arbitrary Erlang terms.
• Data is stored sorted by key.
• Each storage is segmented. Segments can be distributed among Erlang nodes conforming a cluster.
• Full support of ACID transactions (many thanks to mnesia)
• Support next types of storage:
  • ram. In-memory storage. Fast but not persistent (erlang ets is under the hood).
  • ramdisc. All the data is kept in memory (erlang ets), but the storage also provides persistence. Read/Scan operations are as fast as for 'ram' type. Write operations are a bit slower because of maintaining a sequential log.
  • disc. Persistent type storage. Read/Write/Scan operations are slower than for 'ram' and 'ramdisc' types. But the storage resides on the disc (except for buffer and cache) and therefore does not require a lot of RAM. This type of storage uses leveldb under the hood. Many thanks to developers of leveldb, eleveldb, and mnesia_eleveldb projects. You work is great!

Build

$ rebar3 compile

Usage

Start with adding a new storage (e.g. storage1) with type of disc

dlss:add_storage(storage1,disc),

Check for type of storage

dlss:get_storage_type(storage1), %% disc

Check for created the new storage (in our case only storage1)

dlss:get_storages(), %% [storage1]

When adding a new storage, there will be created automatically a new Root Segment (table), which name is generated by "dlss_" + Storage + "_id" (in our case: "dlss_storage1_1"). To check for all segments , it is used as follows

dlss:get_segments(), %% [dlss_storage1_1]

To list all segments for related storage

dlss:get_segments(storage1), %% [dlss_storage1_1]

To get segment (table) info

SInfo = dlss:get_segment_info(dlss_storage1_1), %% #{type:= disc,local:=false,nodes:=[[email protected]]}

Let's write some data to the storage with dirty operation (e.g key as {x,} and value as {y,})

dlss:dirty_write(storage1,{x,30},{y,30}),
dlss:dirty_write(storage1,{x,20},{y,20}),
dlss:dirty_write(storage1,{x,10},{y,10}),

Let's write some data to storage with transaction

dlss:transaction(fun()-> dlss:write(storage1,{x,40},{y,40}) end ),
dlss:transaction(fun()-> dlss:write(storage1,{x,50},{y,50}) end ),
dlss:transaction(fun()-> dlss:write(storage1,{x,60},{y,60}) end ),

Let's read some data from storage with dirty operation and transaction

V1 = dlss:transaction(fun()-> dlss:read(storage1,{x,20}) end ), %%{ok, {y, 20}}
V2 = dlss:dirty_read(storage1, {x,40}), %% {y, 40}

Iteration Find the first key (the least)

First = dlss:dirty_first(storage1), %% {x,10}
First1 = dlss:transaction(fun()-> dlss:first(storage1) end ), %% {ok, {x,10}}

Find the last key (the greatest)

Last = dlss:dirty_last(storage1), %% {x,60}
Last1 = dlss:transaction(fun()-> dlss:last(storage1) end ), %% {ok, {x,60}}

Find the next key

Next = dlss:dirty_next(storage1,{x,50}), %% {x,60}
Next1 = dlss:transaction(fun()-> dlss:next(storage1,{x,50}) end), %%{ok, {x,60}}
Next2 = dlss:dirty_next(storage1,{x,60}), %% '$end_of_table'
Next3 = dlss:transaction(fun()-> dlss:next(storage1,{x,60}) end ), %% '$end_of_table'

Find the prev key

Prev = dlss:dirty_prev(storage1,{x,20}), %% {x,10}
Prev1 = dlss:transaction(fun()-> dlss:prev(storage1,{x,20}) end), %%{ok, {x,10}}
Prev2 = dlss:dirty_prev(storage1,{x,10}), %% '$end_of_table'
Prev3 = dlss:transaction(fun()-> dlss:prev(storage1,{x,10}) end ), %% '$end_of_table'

Let's add a new node and to copy the segment (dlss_storage1_1) to it

dlss:add_node('[email protected]'),
dlss:add_segment_copy(dlss_storage1_1,'[email protected]'),

Then it can be removed as follows

 dlss:remove_segment_copy(dlss_storage1_1,'[email protected]'),

To clean up

dlss:remove_storage(storage1),
dlss:get_storages(),    %% [] empty
dlss:get_segments().    %% [] empty

Run Tests

./rebar3 ct --spec=./test/module/test.spec
./rebar3 ct --spec=./test/rebalance/test.spec
./rebar3 ct --spec=./test/load/test.spec

API

dlss