Feature request: "Append" button on FOXDEN web portal

[jpk96]

@vkuznet, you recently implemented "Amend" button that allows users to modify or update existing metadata records (#9). Thank you! While this feature is useful, it can be error-prone, especially when users intend only to add new metadata rather than change existing information.

To address this, an "Append" button could be introduced. I am imagining that this feature would enable users to add new metadata to existing records without altering the original data. For example, users could submit a JSON object containing additional metadata, which would then be appended to the existing record. This approach preserves the integrity of the original data while allowing for more robust and safer updates.

Tagging @keara-soloway @wernersun @ksg52 for their thoughts.

[vkuznet]

Peter, I'm afraid it will backfire. The issue with append is that it goes against existing schema, and so far all our records satisfy specific schema. This consistently is crucial in data lookup as it allows to index data and guarantee that all records will have specific keys. Moreover, if we open this Pandora box it will allow users to add arbitrary data which are not compatible with the existing schema and neither will be compatible with other records which will break consistency. Do you really want this feature?

If your intention is to support extension of existing records with new metadara attributes then what we need is to modify schema files with new matadata keys which should be optional.

Please do not be discouraged by my reply but rather think about consequences and provide concrete use cases when such support is desired.

[jpk96]

@vkuznet Thank you for raising an excellent point -- I appreciate your insight. You are absolutely right that my goal is to support updating existing records by adding new attributes defined in the updated schema (see PR #12). In this revised schema, all new keys are optional, which means appending additional metadata to existing records should not introduce issues or conflicts. I think this approach allows for integration of new data while preserving the integrity of the original records.

A relevant use case is at ID3B, where we've been submitting very basic records. We now want to include additional metadata related to experimental parameters that are important to track. This kind of enhancement aligns well with the append functionality, allowing us to enrich existing records without modifying their original content.

[vkuznet]

In this case I don't see any difference between append and amend once we will update the schema. Both options will update records with new attributes allowed by the schema. Am not missing something?

I'll be back to work next week and will review and the new schema changes.

[jpk96]

@vkuznet The Append function helps ensure that existing records remain unchanged, minimizing the risk of unintentionally modifying original data. In contrast, using the Amend function may inadvertently alter existing records, especially when the intent is only to add new information.

I’m envisioning the Append function as a mechanism where users can submit a JSON object containing additional metadata. This data would then be appended to the existing record, ensuring that the original information remains intact and reducing the risk of unintended edits.

The Amend function remains useful when the intent is to modify existing records -- for example, correcting errors or updating outdated values.

Both the Amend and Append functions serve distinct purposes and can complement each other depending on the user's goals.

[vkuznet]

Peter, to simplify discussion let's use concrete example with very simple schema:

# schema
{"foo": string}

Ok, we have records with foo key as string values. Now, let's say that we extend this schema to include new optional key bla, e.g.

# new schema
{ "foo": string, "bla": int}

and we restart FOXDEN services.

Now, we user supply new JSON {'bla': 1} either option Amend or Append will add this key to a record without modifing its original content which was {"foo": some_stirng}. Therefore, to embrace what you're asking the FOXDEN role is to provide new schema and user to provide ONLY new attributes. Why do we want to protect records from user who supply valid JSON with valid keys if we don't know user intention? Does user want to ONLY update record with new attributes or does user want to update record and replace some of its values. My point is that if user will use Append button AND supply foo value nothing will stop it from FOXDEN point of view since the new schema (when we update it) will allow both foo and bla. Therefore, I see that Append is redundant from server point of view as it does not really protect the update of foo key (because schema is allowed it). To achieve the behavior you are asking for we MUST compare existing record and what user provide and discard anything in provided user's record which belong to a previous schema, but as I pointed out above we (server) don't know what is original user intention. And, because of that I can guarantee that someone will use Append in a sense of Ammend and vice versa. In my view if schema allows new metadata key both action are identical through the UI unless we know user intention (and trust me because both actions will be allowed users will confuse them and eventually abuse them). To avoid this it should be no ambiguities neither on user or server side which is not achievable in your request.

Once, again I'm not discouraging you from pushing forward with this request but rather arguing about consequences which you may not foresee and fully evaluate. Please think about it and let's discuss it further (in person if you want next week).