Probo-DB

A database service that consumes event data from an eventbus stream and stores it in PostgreSQL.

The database is designed to be able to process and reprocess data from a system such as Apache Kafka. Therefore each step of the data processing and storage pipeline needs to be able to see the same piece of data multiple times without issue.

The database is abstracted by knex and relies on postgres. Review the appropriate documentation for details on these projects.

The REST API is built using Restify.

Configuration

By default, the knexfile.js is used to run migrations and as the configuration to the proboDb server. You can override this is a few ways, first, you can specify a path to a knexfile anywhere on the filesystem by passing the -k flag when starting the server:

./bin/probo-db -k /path/to/knex/file.js

Configuration is also inherited from yaml config that can be passed in via the -c flag:

./bin/probo-db -c /path/to/config/file.yaml

If both options are used, the knex connection data from the yaml file will be used over that passed in via the knexfile.js.

Since knex and migrations assume a knexfile.js, it is recommended to provide this rather than specifying configuration in the yaml file.

Eventbus / Kafka

The service connects to a Kafka event stream to consume build events. The eventbus is configured in the YAML config under buildsEventStream:

buildsEventStream:
  plugin: Kafka
  config:
    group: 'proboDb'
    topic: 'build_events'
    version: 1

• plugin — The eventbus plugin to use (e.g. Kafka).
• group — The Kafka consumer group name.
• topic — The Kafka topic to consume build events from.

REST API

The service exposes the following endpoints:

Method	Endpoint	Description
GET	/	Returns version info
HEAD	/	Returns version info
GET	/build	List all build IDs
GET	/build/:build	Get a build by UUID
GET	/build/:build/disk-usage	Get disk usage for a build
GET	/project	List all project IDs
GET	/project/:project/disk-usage	Get total disk usage for a project (excludes reaped builds)

Build and project IDs are validated as UUIDs before processing.

Plugins

The project supports plugins. Plugins should be a separate project and should be referenced by path in the configuration. The plugin project should expose the plugins as properties on the exported object. Database plugins should be called dbPlugins and should be in the order they should be run. Api plugins will add routes to the REST API server. They must be added as an array to a property called apiPlugins. Order is not important as these will be routes to access data out of band.

const myPlugin = require('./myDbPlugin');
const myApiPlugin = require('./myApiPlugin');

module.exports = {
  dbPlugins: [
    myPlugin
  ],
  apiPlugins: [
    myApiPlugin
  ]
};

Each database plugin must export a class that must implements a process method. The process method will be given a build object which it can then act on. Additionally, the constructor for this class will be handed options that include a logger and a database connection via knex. A valid database plugin might look like the following:

class myDbPlugin {

  constructor(options) {
    this.knex = options.knex;
    this.logger = options.logger;
  }

  process(build) {
    let record = this.prepare(build);

    return this.knex('some_table')
      .insert(record);
  }

  prepare(build) {
	  // extract data and return object.
    self.logger.info('Return some feedback');
  }
}

module.exports = myDbPlugin;

An API plugin will also receive an object that includes a knex connection and a logger. The API plugin must implement an addRoutes method that will be handed a restify server object and will append new routes to it. A valid api plugin might look like the following:

class myApiPlugin {

  constructor(options) {
    this.knex = options.knex;
    this.logger = options.logger;
  }

  addRoutes(server) {
    let self = this;
    server.get('/some/route', function(req, res, next) {
      self.knex
        .select('*')
        .from('mytable')
        .then(function(result) {
          self.logger.info('Return some feedback');
          res.send(result);
          next()
        })
        .catch(function(error) {
          self.logger.error('Bad stuff happened...');
          next(error);
        });
    });
  }

}

module.exports = myApiPlugin;

Migrations

Migrations are stored in the baseMigrations directory. Create these using knex.

Plugins can also have migrations in order to add their own tables. Migrations are combined into a single folder and run together using ./bin/migrate. To have your plugin's migrations run, add them to a directory called migrations in the plugin's folder. See the knex migration documentation for more information on creating migrations.

Docker

Build and push the Docker image using:

./build.sh <repository_name> <tag>

For example, to build and push to DockerHub:

./build.sh mbagnall dev

Or to a private registry:

./build.sh docker.example.com/probo dev

The container exposes port 8442 and expects configuration files at /etc/probo/db.yaml and /etc/probo/knexfile.js. On first start, it runs migrations automatically before starting the server.