pl-s3retrieve

Table of Contents

• Abstract
• Description
• Usage
  • Arguments
  • Run
• Development

Abstract

A ChRIS ds app to retrieve data of interest from Amazon S3 service

Description

s3retrieve is a ChRIS-based application to retrieve file/folders from Amazon S3 service.

Usage

python s3retrieve.py
    [-h] [--help]
    [--json]
    [--man]
    [--meta]
    [--savejson <DIR>]
    [-v <level>] [--verbosity <level>]
    [--version]
    <inputDir>
    <outputDir>
    --bucket <BUCKET>
    --prefix <PREFIX>
    --awskeyid <KEYID>
    --awssecretkey <SECRETKEY>

Arguments

[-h] [--help]
If specified, show help message and exit.

[--json]
If specified, show json representation of app and exit.

[--man]
If specified, print (this) man page and exit.

[--meta]
If specified, print plugin meta data and exit.

[--savejson <DIR>]
If specified, save json representation file to DIR and exit.

[-v <level>] [--verbosity <level>]
Verbosity level for app. Not used currently.

[--version]
If specified, print version number and exit.

<inputDir>
Input directory.

<outputDir>
Output directory.

--bucket <BUCKET>
Name of the Amazon S3 bucket.

[--prefix <PREFIX>]
If specified, retrieve directory/file prefix path in s3.

--awskeyid <KEYID>
AWS access key id.

--awssecretkey <SECRETKEY>
AWS secret access key.

Getting inline help is:

docker run --rm fnndsc/pl-s3retrieve s3retrieve --man

Run

You need you need to specify input and output directories using the -v flag to docker run.

docker run --rm                                                         \
    -v /tmp/input:/incoming                                             \
    -v /tmp/output:/outgoing                                            \
    fnndsc/pl-s3retrieve                                                \
    s3retrieve --awskeyid KEYID --awssecretkey ACCESSKEY --bucket bch-fnndsc  \
    --prefix test \
    /incoming /outgoing \

The above will retrieve a copy of each file/folder inside the test "folder" in Amazon S3 storage into the local /outgoing directory. Some metadata files should have previously been read from /incoming directory.

Development

Build the Docker container:

docker build -t local/pl-s3retrieve .

Run unit tests:

docker run --rm local/pl-s3retrieve nosetests