Is SQL the best exchange format for the HDB?

[sarahec]

I was thinking about testing (and especially CI for the new release) and I keep stumbling over the HDB. It's hard to build hermetic tests when you have a DB with previous data. It's also hard to be clear about what's in the database you're using.

It was perfectly appropriate 20 years ago, but is it the right thing today?

I'd like to propose a new way to manage these dependencies: using a manifest file (in YAML or similar) that points to its needed resources by URL. Resources would be stored in files (source, Bitcode, metadata) collected into a folder.

The easiest migration path would be a command-line tool that reads the manifest, pulls down the files it needs (possibly with a compile step), then builds the local SQL database.

This would make it easier to submit a complete project, put projects under source control, etc.

Your thoughts?

• S

[pjaaskel]

I agree the SQLite is cumbersome for HDB. We have discussed internally it several times, and the best option could be just ascii-based metadata files (JSON, YAML, whatever is fashionable these days) next to the VHDL files or such, a directory structure based database like you mention, no need to build the SQL. However, this is non-trivial amount of engineering work which is difficult to get done in an University environment due to the publication pressures. We are happy to review any pull requests though!

[sarahec]

Thank you. I will take a look at the required work.

[nrother]

The simplest starting point might be to just store the HDB content in the form of SQL statements, and them have a build step that actually creates an SQLite-DB out of it. This would not need any changes in the code, but would still allow to have a textual representation of the DB content. Instead of SQL statements JSON might also be an alternative, than it should even be possible to convert the code later to directly use that text file.

[sarahec]

Yes. I just need a few quiet days to prototype a command-line tool to export/import a DB.

[pjaaskel]

@sarahec you can just redirect the SQL in ascii to the sqlite3 cmdline tool.

[sarahec]

I'd rather store the objects in a human-friendly serialized format and build on top of the HDB manager, for two reasons:

1. Once we start exporting pieces this way (in ASCII/UTF-8), people will want to look at the files, check them into source control, make edits, etc. These files are another kind of human interface, so they should be user-friendly.
2. Having legacy SQL in files can interfere with making changes to the database schema.

[pjaaskel]

Yep, I agree that's the best end point, but @nrother's suggestion is a good (but not required) starting point (the SQLite binary blobs are annoying in source control).

[sarahec]

I could see dumping the raw SQL as an interim fix that's more friendly to source control. My longer-term vision is that a project could check in their code with a collection of files defining the parts of their machine, all in human-editable form.

I suspect a pair of shell scripts or a small python program could handle the SQL import/export for now. I'll do that first.