Request for feedback on tooling change

[cam72cam]

Hello, I'm one of the maintainers of https://github.com/opentofu/opentofu and am in the middle of re-building our CLI. As part of this, we are considering switching away from https://github.com/posener/complete/ (out of date and likely unmaintained) and using the the builtin completion of https://github.com/urfave/cli or https://github.com/spf13/cobra.

While investigating this issue, I found that this library ships a default completion for tofu that uses the complete -C method. While that is useful, it makes migrating to a new solution a bit trickier, especially since the two mentioned above use custom completion functions (complete -F).

As of writing this, I have proposed that we add in backward compatibility for $COMP_LINE and respect the existing installations. The only other alternative here is to suggest users run tofu -install-autocomplete and re-source their ~/.bashrc, though I don't much like that approach. Luckily, my initial prototyping has shown that the maintenance burden of keeping rudimentary support for $COMP_LINE is pretty minimal.

My primary question is if the approach of supporting the both paths is typical and if there is a way to change the shipped bash-completion default for tofu to something along the lines of source <(tofu completion bash).

Additionally, is it preferred for tooling to use use -C or -F when building in auto-completion?

My last question is if user installation into $HOME/.bash_completion is the recommended option if targeting older systems? I tried using the $BASH_COMPLETION_USER_DIR and $XDG_DATA_HOME startup options, but ran into it being a relatively (2yr old) feature that debian testing still does not yet ship.

If you are curious and are looking for more details, we are discussing this on our end in an RFC at opentofu/opentofu#4445.

Thank you again for your time in considering my questions and for the ongoing work that goes into maintaining this project!

[akinomyoga]

My primary question is if the approach of supporting the both paths is typical and if there is a way to change the shipped bash-completion default for tofu to something along the lines of source <(tofu completion bash).

The short answer is, please install <prefix>/share/bash-completion/completions/tofu.bash or <userdir>/completions/tofu.bash (which loads the proper completion by eval -- "$(tofu completion bash)") at the upstream side.

First of all, we provide the loader for the tofu completion as a third-party fallback completion loader. The "fallback" intends to be a fallback when the upstream didn't install a completion setting at /usr/share/bash-completion/completions/tofu.bash (or whatever path that is appropriate for the user's environment, see our README.md for details). In particular, if the upstream cares about it, we request the upstream to install its own completion setting at <prefix>/share/bash-completion/completions/tofu.bash (with the contents like e.g. eval -- "$(tofu completion bash)"). The reason is exactly because of the compatibility issue you reported here. The only solution for the version-dependent completion is letting the upstream install the completion corresponding to the installed upstream version.

Since our completions-fallback/tofu.bash (or completions/_tofu in older versions of bash-completion) intends to be a fallback for the old versions of the upstream that didn't install the proper (i.e., version-dependent) completion at <prefix>/share/bash-completion/completions/tofu.bash, the fallback should basically be kept to be a loader for the last version of the upstream that didn't install the proper completion file.

Additionally, is it preferred for tooling to use use -C or -F when building in auto-completion?

There is no particular recommendations from the bash-completion side. You can use either one that you consider is more useful for implementing completion for your command. Only when you want to use the utilities provided by bash-completion in implementing completion, you obviously need to implement the completion through a shell function (-F func), because otherwise the bash-completion utilities implemented as shell functions cannot be called from a separate process (-C cmd).

My last question is if user installation into $HOME/.bash_completion is the recommended option if targeting older systems?

No, $HOME/.bash_completion is not meant for completions for specific commands, at least since bash-completion-2.0 released in 2012, 14 years ago. If you want to install a completion file in the user's home directory, it is recommended to be put in $BASH_COMPLETION_USER_DIR/completions or $XDG_DATA_HOME/bash-completion/completions.

I tried using the $BASH_COMPLETION_USER_DIR and $XDG_DATA_HOME startup options,

Though, $BASH_COMPLETION_USER_DIR and $XDG_DATA_HOME are not startup options. Completion settings are dynamically loaded only after an actual completion for the corresponding command is attempted by the user.

but ran into it being a relatively (2yr old) feature that debian testing still does not yet ship.

Which version of bash-completion does the Debian testing use? $BASH_COMPLETION_USER_DIR and $XDG_DATA_HOME are introduced in 1d25d72 (11 years ago) and released in bash-completion-2.2 (10 years ago).

[scop]

A couple of quick additional comments from my side:

FWIW, IME, spf13/cobra is much better regarding shell completions than urfave/cli. (Cobra does have its warts otherwise.) I'm not sure if/how much completions are a priority for urfave/cli whereas they certainly continue to be for spf13/cobra. An example from lefthook whose completions regressed significantly when they switched, with breadcrumbs to further info:

• feat: more rudimentary shell completions evilmartians/lefthook#1230
• refactor!: use another cli framework evilmartians/lefthook#1155 (comment)
• More powerful shell completion urfave/cli#1905

Even if what we ship for tofu is indeed a in a dir labeled for fallbacks, in my experience and bubble, single binary installs from e.g. GitHub releases using tools like mise or aqua are becoming more and more of a norm. For those kinds of installs, there's no good possibility to "install" completions by upstreams, and that's what we try to help with. But for packages/setups that do have that possibility to install, the above is good advice.

[cam72cam]

@akinomyoga we'll recommend package maintainers ship their own tofu.bash completion file instead of relying on the fallback presented here.

As @scop notes, our users install tofu in a wide variety of methods. Many of which don't have the opportunity to install completions to reasonable locations.

I was looking at the "startup completion file" portion of the README.md and realized that debian testing is on 2.16.0 and does not yet support that feature. I was testing binaries outside $PATH, which AFAIK does not invoke the custom completions with a matching command name. The more I think about it, that is mostly a problem for my development environment and not a realistic end user scenario. For the users that want to use a -install-autocomplete like option, I will try to use your <userdir>/completion/tofu recommendation.

We started out using cobra, but realized that their flag parsing requires POSIX compatible flags and that feature can't be turned off without some serious overrides that make it not worth using. Thanks for the links describing other people having issues with urfave/cli's completion limitations, we will consider that thoroughly.

At this point, we'll have to take a hard look at our options and what path makes the most sense. I'm leaning toward urfave/cli, with a custom completion hook or posener/complete integration. The good news is that the abstractions I've put in place make it relatively easy to swap out the frontend without touching any of the actually command definition and implementation logic.

@scop I saw you posted a PR to posener/complete, it would be nice to have a well supported completion library for golang. I'll give you a ping if we end up forking that project or creating something similar.

Thank you again for your time and input!

[scop]

Out of interest, did you evaluate or have experience with other Go CLI libraries with completion support besides the mentioned ones? Not sure if there are other "big" ones around, but one list (not limited to ones having completion support) is at https://awesome-go.com/standard-cli/

it would be nice to have a well supported completion library for golang. I'll give you a ping if we end up forking that project or creating something similar.

It sure would, and thanks! I'd likely be willing to contribute something to such a thing.

[cam72cam]

The only one that might fit our needs is https://github.com/integrii/flaggy. It's a less used library, but it seems to support many of the features that we are looking for. I'll give it a try today. I'm curious what you think about their approach of just dumping all possible entries into COMPREPLY?

I'm also going to play around with forking posener/complete to add a few missing features as a potential fallback.

[scop]

I think including ~everything possible in COMPREPLY and letting the shell work it out is pretty much the standard practice. Not the fastest surely, but for it to be an issue requires quite a few candidate entries. Caveat: I haven't studied how any of the mentioned libs actually work with respect to this, e.g. if they do any automatic pre-filtering or the like. In bash-completion we tend to only "partition" based on if we're completing a flag or something else. (And we do not handle everything after -- being positional well or practically at all or at least consistently, but I digress.)