Protobuf implementation in Jai

Implementation of Google Protocol Buffers "Protobuf" serialization format in pure Jai.

Protobuf uses a language-neutral message description format which can be used to interchange data between different programming languages, implemented by generating source code for each language for De/Serialization. The official C++ repo is here.

Example

The protocol buffer text format:

message Person {
    string name = 1 [default = "?"];
    int32 age = 2;
    float height_cm = 3;
    repeated sint32 favourite_numbers = 4;
}

Produces the following Jai structure and function definitions:

Person :: struct {
    name: string = "?";
    age: s32;
    height_cm: float32;
    favourite_numbers: [..]s32;
}

Serialize : (message: Person) -> []u8;
Deserialize : (message: *Person, bytes: []u8) -> ok: bool, used: s64 = 0;
FreeContents : (message: *Person);

The code can be generated by calling:

jai_code : string = generate_jai("person.proto");
// Write to file, ...
// Or inserted directly at compile time:
#insert #run generate_jai("person.proto");

Quick start

In your projects directory, add this repo as a module. Jai automatically looks in ./modules/ for modules, so that's a good place to put it. If you use git, you can submodule it, or otherwise copy it in there.

git init
git submodule add https://github.com/segcore/jai-protobuf.git  modules/Protobuf

// file: main.jai
#import "Basic";
#import "Protobuf";

#insert #run generate_jai("modules/Protobuf/examples/protos/some_things.proto");

main :: () {
    group: Group;
    group.group_name = "Players";
    array_add(*group.people, .{name = "Bob", age = 42, height_cm = 169.4});
    print("group: %   bytes: %\n", group, Serialize(group));
}

API

The main parsing and code generation is in generator.jai, which has:

generate_jai :: (input_protos: ..string, search_dirs: []string = .[""], protobuf_import := "")
           -> jai: string, failed_imports: []string { ... }

// called like
generate_jai("a.proto", "b.proto", "c.proto", search_dirs=["", "/usr/include"]);

Primitive encode and decode functions are available in respective files. This can be used without using the code generation (which requires running the compiler to be useful). This could be used to create a tool which encodes and decodes any protobuf at runtime, for example.

Why use Protocol Buffers?

1. If you need to interoperate with another application which uses this serialization format.
2. To serialize/deserialize data for your own projects, if it suits your purposes. Protobuf has forwards and backwards compatibility, with missing values taking the default.

Details

Structures, enumerations, and serialization-related functions are generated according to protobuf text descriptions. See the complete example's protobuf description and its generated code. The generated code is very simple -- normal structures with (public) fields just as you would write them normally. No added indirection. Repeated fields use [..]type and can be used directly. Use the language's context features to setup custom allocators.

The parser is very relaxed. It ignores anything it doesn't understand and just keeps soldiering on. This can be good in that it doesn't need every possible feature to be implemented here to be able to parse realistic external files. It should work for correct protobuf files, but presently gives poor messaging for failures/unknown features.

The parser/generator also does not follow all Protobuf rules, because it does not need to in order to work. For example, you can use an Enum or Message from another protobuf text file without importing it. The parser and code generation don't need it. If the other file is eventually #loaded at build time, it will work. Otherwise, it will be a compile time error. This simplifies the process too -- leveraging the compiler to do all of the linking work. Afterall, it is going to have to do it anyway, so why double up?

Feature set

• Encode primitive types
• Decode primitive types
• Parse protobuf description files
  • Basic
  • Nested messages/enums
  • comments (ignored)
  • import
  • package
  • reserved fields (ignored)
  • options
    • deprecated, packed, default
  • Any and other well-known types
• Generate Jai enums
• Generate Jai structures
• Generate Jai serialize functions
• Generate Jai deserialize functions
• Repeated fields
  • Unpacked repeated fields
  • Packed repeated fields - Encode
  • Packed repeated fields - Decode
• Map types
• One-of types
• Recursive types (e.g. google/protobuf/struct.proto)

Not intended to be implemented:

• Field presence information
• Message "extensions"