Spark Connect: keep proto/python in sync with the Java Model. Question on how we handle proto field number evolution

[Chandanydv1234]

Branch: Spark Connect

The wire schema in zingg_command.proto is a mirror of the java model, and ZinggProtoConverters hand-maps each proto field to its java setter. proto->python is generated automatically, but java model-> proto is manual.

Problem 1 (Sync) : If a field is added/renamed/removed in the java model, nothing forces the proto, the python stub, or the converter to update.
Automate the changes in the python stub, so that it is in sync with the java model.

Question: Each proto filed has a permanent tag number that governs wire compatibility. As the schema evolves, new fields, removed fields, type changes, how are these numbers managed or how do we want to manage those numbers so old and new clients stay compatible? - flagging this for a decision.

[sonalgoyal]

There are few other things to think about

• how does this run in notebook env like Fabric or Databricks?
• We can layer in a gui now for the labeller. Where does that run in the notebook? Or can we have an app?

[Chandanydv1234]

checked each proto message against the java classes it mirrors. most of it matches(matchtype, fieldDefinition, Pipe and arguments are in complete sync).
The one gap is ClientOptions, two flags(--remote, --preprocess) exist in java but not in proto. So fully autogenerating 1:1 seems tricky and fragile.
And also there nothing exists as of today that checks this, so there wouldn't be any warning if they drift apart.
A better idea would be a small test that fails the build whenever the java and proto don't match.

About the behaviour of the field number, we never change or reuse a field number, when removed it should be marked as reserved to avoid being used by mistake.