Import/Export for Common Formats

[lgessler]

Introduction

Preliminary interactions with users indicate they expect to be able to import and export their data in commonly used formats, such as the export formats of ELAN and FLEx, ELAN Annotation Format (EAF) and FieldWorks XML (FWXML). Both of these formats are commonly used in day-to-day workflows, and are also the deposit format used by some language archives.

Background

These formats are the two most popular that are in use. It's not clear to me that we need to immediately support other formats, but we should look into it more by talking to users. Some other ones that come to mind

• Toolbox, if it has its own format.
• Plaintext: I think users will expect a formatted whitespace representation of at least interlinear data.
• LaTeX interlinear package formats (gb4e, linguex): a nice convenience.

Currently, we do not support any import, and we only support export of the raw JS representation of a document.

Problem

We should add import and/or export support for these formats, prioritizing according to demand. Some decisions that need to be made:

1. Where and how in the code should this functionality be implemented? It could take place on the backend, or purely in the frontend.
2. For each import/export function, what is the appropriate level of granularity (e.g. project, document, or perhaps even sentence)?
3. For particular format requirements, would it be sufficient to support a core set of formats and expect users to write their own code, separate from our system, to accomplish the formatting need they have?
4. If the answer to the former is "no", is it worth investing in some kind of bring-your-own-code (e.g. in JavaScript) solution for users who want to produce their own export/import functionality?

Proposal

FLEx and ELAN at the very least should be covered. As for the rest, TBD.

Child Issues

(List child issues here as they are created. This issue should not be closed until all child issues are closed.)

[lgessler]

Some points from a discussion with Bhargav:

• On the backend, having an endpoint which accepts a JSON that is much like the export JSON seems like the way to go
• A few common formats (ELAN, FLEx) should have first-class support where the conversion happens with JS in the frontend and the backend receives aforementioned JSON
• On the view that errors will be rare during actual use, document creation should fail if there is an error, and some number of errors should be sent back in the HTTP response, with any information that the user ought to report to us in order to reproduce the error.
• Someone, either an admin or a user, maybe both, needs to construct a mapping between Glam layers and external data structures. Would be instructive to refer to how this is handled for ELAN and FLEx: https://ordinaryworkinggrammarian.blogspot.com/2019/12/elan-to-flex-to-elan.html https://www.mpi.nl/tools/elan/docs/manual/Sec_Importing_a_document_from_Fieldworks_Language_Explorer_FLEx.html

[lgessler]

We're punting on this for now—needs a lot more thinking than we thought.