ebils

EBILS - Erlang Binary Lightweight Search

A single binary search for large and huge binary into pure erlang, this uses small chunk of binaries to search in them using single processes and delivering message to the main process.

With single tests of a large binary this method can be 17x faster that single binary:match and allows to you seek results in the process.

A single example:

1> {ok, File} = file:read_file("test.data").
{ok,<<"AAAAAAAA|J|89\nBBBBBBBB|J|89\nCCCCCCC|J|89\nDDDDDDDD|J|89\nEEEEEEE|J|89\nFFFFFFFF|J|89\n"...>>}
2> byte_size(File).
234179947
3> ebils:load(File, <<"\n">>).
true
4> timer:tc(ebils, search, [<<"ZbZbZbZbZb">>]).
{4031,{{ok,{2341677,13},<0.161.0>}}
5> {Found, Pid} = ebils:search(<<"ZbZbZbZbZb">>).
{ok,{2341677,13},<0.160.0>}
6> gen_server:call(Pid, {get, Found, 4}).
{ok,<<"ZbZbZbZbZb|J|89">>}

In the example:

• Line 1: Load a huge file
• Line 2: Check the byte size ( it's very large :-D )
• Line 3: Parse file into chunks and load into memory
• Line 4: Perform a single search, the result is the position of the match
• Line 5: Perform a search and store result in a var to use after
• Line 6: Extract a binary part of the chunk where found the match