Where do library customization data extension YAML files go, and how can we actually develop and use them?

[gsutherland-trailofbits]

The documentation for library customisation / data extensions describes the structure of the YAML files, but I cannot see any documentation on where these files should be placed, or how they can be invoked (or debugged) using CodeQL CLI or the vscode extension.

Some of the documentation implies that these YAML files are intended to be built into query packs and are loaded that way, but I didn't see any examples on how this is actually done in practice, nor any that explained how to develop and test the extensions before packing them.

Could someone provide some insight into this, and potentially update the documentation to include an example of a practical development flow using this feature? I know this feature is in beta, so the docs aren't 100% there yet, but a practical example would go a long way towards getting folks hands-on with it. Thanks.

[mbg]

Hi @gsutherland-trailofbits 👋🏻

Thanks for the question! You're correct in identifying that the YAML files with the models are intended to be included in a query pack.

The most user friendly way of creating a data extension is using the model editor in the CodeQL VSCode extension (after opening a database, select "CodeQL: Open CodeQL Model Editor (Beta)" from the command palette), which will analyse an existing database to identify parts that aren't yet modelled and suggest them to you (and help you model them), but it will also create a query pack with the right structure (e.g. for Java):

# codeql-pack.yml
name: pack/your-pack-name
version: 0.0.0
library: true
extensionTargets:
  codeql/java-all: '*'
dataExtensions:
  - models/**/*.yml

And then your custom models would go in YAML files relative to codeql-pack.yml which match the dataExtensions pattern(s). Those can either be generated using the Model Editor in the VSCode extension or by hand following the format described in the docs. We'd recommend the Model Editor for languages that are supported by it, rather than doing it by hand.