How to use AWS S3 CLI to dump files to stdout in BASH

Question

I'm starting a bash script which will take a path in S3 (as specified to the ls command) and dump the contents of all of the file objects to stdout. Essentially I'd like to replicate cat /path/to/files/* except for S3, e.g. s3cat '/bucket/path/to/files/*'. My first inclination looking at the options is to use the cp command to a temporary file and then cat that.

Has anyone tried this or similar or is there already a command I'm not finding which does it?

Answer 1

All file objects must have their contents written to stdout.

To get this, use the aws s3 cp command and provide "-" as the destination. $ aws s3 cp s3:/mybucket/stream.txt is an example of a command.

Is this what you're after? ::

#!/bin/bash

BUCKET=your-bucket-name

for key in `aws s3api list-objects --bucket $BUCKET --prefix bucket/path/to/files/ | jq -r '.Contents[].Key'`

do

  echo $key

  aws s3 cp s3://$BUCKET/$key - | md5sum

Done