Contemplate redesign or decommission in favor of AWS CLI

[sopel]

The initial goals of this project have been as follows:

1. Exposing boto via scripts for CLI usage to avoid the need to use the various legacy AWS API tools per service
2. Easing cross region AWS operations in general and for AWS CloudFormation in particular (i.e. the latter has been the driver of the former)
3. Amending cross cutting concerns like e.g. backup management

The meanwhile available new AWS Command Line Interface seems to address or at least enable these goals one way or another as well:

• Ad 1) This mostly is (and certainly will be) addressed in an all encompassing fashion, see Is this project going to supersede the existing CLI tools per AWS service? aws/aws-cli#31 for a respective clarification.
• Ad 2) This is not included, but respective shell based usage is trivial, see e.g. the respective StackFormation snippets (albeit lacking resiliency eventually, see also Add graceful handling of region connection issues. #32).
  • Also it might be possible to reuse/amend the AWS CLI driver from cross region wrapper code, be it via botocross or a respective upstream addition, see also Refactor/Unify cross region looping #42.
  • Either way it is obvious that the one tool with sub commands vs. multiple scripts design scales much better for the use case at hand, thus suggesting a respective refactoring (implying a migration from boto to botocore, which is desired anyway).
• Ad 3) This is not included and doesn't seem to be a good fit either at first sight (even though boto currently features such cross cutting functionality in fact, see boto.manage and e.g. Volume.trim_snapshots in particular).

Accordingly it seems to be appropriate to either redesign botocross by reusing/amending the new AWS CLI and/or the underlying new botocore or even decommision it alltogether, if the remaining functionality could be incorporated upstream or migrated elsewhere (for example, the motivating CloudFormation related features could be provided by the dedicated StackFormation project instead).

[sopel]

Two upstream design aspects are worth noting regarding the issue at hand:

1. botocore is build upon generated code, see Botocore Development - the Model Customizations might allow to easily amend missing functionality, be it via an upstream patch (of course this would likely need to be something broadly usable to be accepted) or by a customized version incorporated here somehow.
2. The AWS CLI is apparently going to be extendable via plugins going forward, see Initial plugin support aws/aws-cli#98 - this might also provide an option to submit or maintain additional functionality (at first sight it is not entirely clear what kind of extendability this plugin support is supposed to cover, i.e. what events/hooks will be available, also the design seems to be a moving target still).

[garnaat]

I was not aware of this project. Thanks for bringing it to our attention.

As you say, the plugin mechanism is still being actively designed and developed. If you have specific requirements or suggestions, now is the time to bring them up. We have some specific goals with the plugins but we would like the general mechanism to be as flexible as possible.

[mrdavidlaing]

It seems to me that the ecosystem is in too much flux to make it easy to
decide which direction to take now.

I'd advise delaying the refactoring until it becomes clearer how the AWS
cli plugin architecure will work.

On 21 April 2013 13:26, Steffen Opel notifications@github.com wrote:

The initial goals of this project have been as follows:

1. Exposing boto via scripts for CLI usage to avoid the need to use
   the various legacy AWS API tools per service
2. Easing cross region AWS operations in general and for AWS
   CloudFormation in particular (i.e. the latter has been the driver of the
   former)
3. Amending cross cutting concerns like e.g. backup management

The meanwhile available new AWS Command Line Interfacehttp://aws.amazon.com/cli/seems to address or at least enable these goals one way or another as well:

• Ad 1) This mostly is (and certainly will be) addressed in an all
  encompassing fashion, see Is this project going to supersede the existing CLI tools per AWS service? aws/aws-cli#31https://github.com/aws/aws-cli/issues/31for a respective clarification.
• Ad 2) This is not included, but respective shell based usage is
  trivial, see e.g. the respective StackFormationhttps://github.com/sopel/stackformation/wiki/Usage#via-the-aws-command-line-interface-clisnippets (albeit lacking resiliency eventually, see also
  Add graceful handling of region connection issues. #32 Add graceful handling of region connection issues. #32).
  • Also it might be possible to reuse/amend the AWS CLI driverhttps://github.com/aws/aws-cli/tree/develop/awsclifrom cross region wrapper code, be it via botocross or a respective
    upstream addition, see also Refactor/Unify cross region looping #42Refactor/Unify cross region looping #42
    .
  • Either way it is obvious that the one tool with sub commands vs.
    multiple scripts design scales much better for the use case at hand, thus
    suggestion a respective refactoring (implying a migration from botohttps://github.com/boto/bototo
    botocore https://github.com/boto/botocore, which is desired
    anyway).
    • Ad 3) This is not included and doesn't seem to be a good fit either
      at first sight (even though boto currently features such cross cutting
      functionality in fact, see boto.managehttp://docs.pythonboto.org/en/latest/ref/manage.html#module-boto.manageand e.g.
      Volume.trim_snapshotshttp://docs.pythonboto.org/en/latest/ref/manage.html#boto.manage.volume.Volume.trim_snapshotsin particular).

Accordingly it seems to be appropriate to either redesign botocross by
reusing/amending the new AWS CLI and/or the underlying new botocorehttps://github.com/boto/botocoreor even decommision it alltogether, if the remaining functionality could be
incorporated upstream or migrated elsewhere (for example, the motivating
CloudFormation related features could be provided by the dedicated
StackFormation https://github.com/sopel/stackformation project instead).

—
Reply to this email directly or view it on GitHubhttps://github.com//issues/44
.

David Laing
Open source @ City Index - github.com/cityindex
http://davidlaing.com
Twitter: @davidlaing

[sopel]

@mrdavidlaing - No worries, I haven't used the word 'Contemplate' for nothing; this mainly aims to collect my thoughts, track the upstream developments and enable a respective discussion, insofar there is quite some technical debt on file already (partially requiring a significant refactoring anyway), which would obviously benefit from removing code in favor of upstream solutions, if possible.

Please note though that this isn't just about the AWS CLI plugin architecture, rather the overall direction/design of the AWS CLI and the underlying botocore package, and while both are young and in development, these are obviously high profile projects here to stay and they should mature fast accordingly (that's why I have cross referenced my clarifying question aws/aws-cli#31 above).

[sopel]

@garnaat - Thanks for commenting; this project had been extracted from an internal one due to being applicable in various more generic contexts, but I've been laying low a bit due to lack of time to stabilize the package and therefore not published it to PyPI yet (see #6), thus foregoing the most public Python PR channel.

I'll closely follow the AWS CLI development and submit suggestions or inquiries concerning specific requirements via separate issues over there, thanks much for your offer!