Kuiper - JSON transform language

This library defines a JSON to JSON transform and templating language. The language itself is inspired by JavaScript. Expressions always terminate, as the language has no form of recursion. This means that while there are loops, they only operate on input arrays. So it is possible to iterate over an array, and even pairs of arrays, but it is not possible to implement recursion.

Features

• Operators, +, -, *, /, ==, !=, >=, <=, >, <, &&, || with precendence taken from the C++ standard.
• Arrays, [1, 2, "test", 123.123, [123, 2]]
• Objects, { "test": "123", concat("test", "test"): 321 }
• Built in functions, like map, float, concat, etc. Either pow(base, exp) or base.pow(exp)
• Functors, map is a functor, meaning it accepts a lambda: map(arr, field => ...) or arr.map(field => ...)
• Selector expressions, [1, 2, 3][1] == 2, input.field.value["dynamic"], etc.
• Macros, #my_macro := (a, b) => a + b; my_macro(1, 2)

Usage

There are several ways of using Kuiper. It can be used both as a standalone program, and as a library in other programs.

Using the CLI

The easiest way to start using Kuiper is to use the standalone CLI. See the kuiper CLI README for more info on the CLI.

As a rust library

Kuiper can also be used as a Rust library.

use kuiper_lang::compile_expression;
use serde_json::json;

let transform = compile_expression("input.value + 5", &["input"]).unwrap();

let input = [json!({ "value": 2 })];
let result = transform.run(input.iter()).unwrap();

assert_eq!(result.as_u64().unwrap(), 7);

Bindings to other languages

Bindings to several other languages exists, for more info on those see their specific package:

• Python
• .NET
• C
• Javascript (WebAssembly)
• Java

Design

This library contains part of a compiler:

• lexer contains a lexer built on Logos, which converts the input to a sequence of spanned tokens. The first stage of the compilation outputs an iterator of Result<Token, LexerError>
• parse contains glue code for a parser generated by lalrpop. The grammar file itself is jsontf.lalrpop. Here you will find the AST for the language, which is the output of the second stage of compilation.
• expressions contains a large set of executable expressions. The executable tree is first build by compiler/exec_tree.rs.
• compiler/optimizer.rs contains the fourth and final stage of the compiler, which attempts to execute each part of the program recursively. If this fails with a source missing error, we traverse its children recursively. If it fails with any other error, this is a compile-time error. If it succeeds, the expression is replaced by a constant. This stage also replaces the variable identifiers with indexes in the input array.

So in summary, an expression is produced from raw input -> Token stream -> AST -> Expression tree -> Optimized tree

In this repository

This repository contains a number of language bindings and auxillary packages, the following is an overview:

• kuiper_lang is the rust crate containing the language itself.
• kuiper_cli is a CLI tool for kuiper, letting you use it like you would a tool like jq. It also contains a REPL.
• kuiper_documentation contains utilities for generating markdown documentation as well as a few source code files from a list of built-in functions in functions.yaml.
• kuiper_interop is a set of C bindings for kuiper_lang, these are currently used for KuiperNet only.
• kuiper_java is java bindings for kuiper, using JNI. These are not currently published anywhere.
• kuiper_js is a set of WASM bindings for kuiper, published to NPM.
• kuiper_lezer is a lezer library for kuiper. Lezer is a JS library to create parsers. It is used for the frontend code editor.
• kuiper_frontend_test is a very simple react app using kuiper_js and kuiper_lezer to provide a live editor.
• kuiper_lang_macros is an auxillary macro library used for kuiper_lang.
• kuiper_python contains python bindings for kuiper, using pyo3. These are published to PyPI.
• KuiperNet contains .NET bindings for kuiper. These are published to nuget, with native binaries for Windows and Linux.
• KuiperNet.Test is a test project for KuiperNet.
• fuzz is a set of fuzz tests for kuiper.