Feedback on where to host the operator documentation generator

[xuang7]

Hi everyone, as we are preparing the first version of the operator documentation, we would like to get feedback on where to host the scripts that generate it.

Background
The generator is fairly tightly coupled with the main repo's operator code. It relies on JVM reflection over operator classes, reads Jackson and schema-related annotations, uses runtime-generated JSON schema metadata such as propertyOrder, and directly depends on compiled operator classes on the classpath.

We are currently considering two options:

Option 1: Main repo (e.g. under /docs)

• Pros
  • Shares the same sbt build as the operator classes, so reflection and schema generation work out of the box.
  • Easier for developers to regenerate docs. When adding or updating an operator, they can run the generator locally and inspect the resulting markdown immediately.
• Cons
  • The generator and the published docs would live in different repos, so the generated markdown would still need to be synced to the site repo.

Option 2: Texera site repo

• Pros
  • Keeps the generator close to the Hugo config, theme, and layouts.
  • Keeps the main repo more focused on product code.
• Cons
  • It would still need access to the compiled operator module, either through published artifacts or some way of repo linkage.
  • There is a higher risk of docs drifting out of sync with code.

From a quick look at projects such as Flink, Spark, and Airflow, the general pattern seems to be that documentation is built close to the source code it depends on, while the site repo mainly acts as a publishing target. That seems more aligned with Option 1.

Any feedback or alternative suggestions would be very welcome. Thanks.

[chenlica]

If the generator code is not heavy, I support option 1 so that it's synchronous with the main codebase.

[Yicong-Huang]

I also support option 1. It is self contained.

[chenlica]

@Ma77Ball @aglinxinyuan @bobbai00 @kunwp1 @Xiao-zhen-Liu @mengw15 @aicam and others: your thoughts?

[aglinxinyuan]

I like option 1.

[mengw15]

I also support option 1

[Ma77Ball]

Option 1 is ok given the build dependencies. Worth noting, though, Flink and Spark don’t just keep docs in their main repo; they keep the entire website there (themes, config, layouts) and sync the built output to a separate site repo. It seems a big reason for this is versioning: keeping docs alongside code means each release branch has the correct docs, which is harder to manage if the generator lives in the site repo. They still deal with the same cross-repo sync overhead listed as a con here.

[kunwp1]

I support option 1 as well because it's hard to sync if we go with option 2.

[Xiao-zhen-Liu]

I support option 1.

[chenlica]

@xuang7 and all: we have enough support for option 1. So let's go with this option.