BRDF - Bitcoin chain data represented as RDF

This example demonstrates the use of BP library to import Bitcoin chain data into an RDF triple store, specifically Franz Inc.'s AllegroGraph, which is written in Common Lisp and provides a high-performance direct Common Lisp interface.

Note that direct Common Lisp AllegroGraph interface only works on AllegroCL, so the same limitation applies to this project.

This is an extension of the example here that was originally written in Python.

Setup

First we need to install AllegroGraph. The instructions on how to do this can be found here. Once AllegroGraph is installed, we can start the server by running

<path/to/AG>/bin/agraph-control --config <path/to/AG>/lib/agraph.cfg start

and stop it when necessary by running

<path/to/AG>/bin/agraph-control --config <path/to/AG>/lib/agraph.cfg stop

Next, we need to get the direct Common Lisp client for AllegroGraph. It is distributed as a .fasl file and can be downloaded from the same place as the AllegroGraph server.

Supported graph types

The first argument to the function brdf.etl:start is graph type. This tool will eventually support several types of Bitcoin data graphs, but currently only one is available:

• transactions - a Tx/Output graph, which is intended as the main way of using this tool;
• chain - a complete legacy representation of Bitcoin time chain (very redundant).

Running the loader

In order to start the loader, run

(brdf.etl:start :transactions
                "http://user:password@127.0.0.1:8332"
                "http://user:password@127.0.0.1:10035/repositories/brdf"
                :workers 4)

You should see a loader log that looks similar to this:

[2021-09-21T20:44:40] Blocks: 0/701589, txs: 0/672153493, progress: 0.00000
[2021-09-21T20:44:50] Blocks: 1494/701589, txs: 1520/672153493, progress: 0.00000
[2021-09-21T20:45:00] Blocks: 3068/701589, txs: 3118/672153493, progress: 0.00000
...

In order to stop the loader, run

(brdf.etl:stop)

Once the function call above returns, all the workers have been stopped and no data will be added to the repository.

In order to continue the load, run the same form but make sure the value of cleanp argument is nil:

(brdf.etl:start :transactions
                "http://user:password@127.0.0.1:8332"
                "http://user:password@127.0.0.1:10035/repositories/brdf"
                :workers 4)

By default, the status of the load will be checked every 10 seconds. As the load continues, it might make sense to change the status check period to a larger value (e.g. 1 hour) by evaluating

(setf brdf.etl:*status-check-period* 3600)

since it is expected to take anywhere from several days to several weeks on an average home machine.

Running the explorer

With some data loaded into a triple store, you start the transaction graph explorer UI by running

(brdf.explorer:start "http://user:password@127.0.0.1:10035/repositories/brdf")

This starts an HTTP server on a default port 6102. Open http://localhost:6102/ in a browser to use the explorer.

An optional :port keyword argument overrides the default port:

(brdf.explorer:start "http://user:password@127.0.0.1:10035/repositories/brdf" :port 16102)

To stop the server, run

(brdf.explorer:stop)