Parallel uploads to the same s3 bucket directory with s3cmd

0 votes

I try to run the following code but index.js turns out to be corrupted.

Any idea why?

gzip dist/production/index.js
mv dist/production/index.js.gz dist/production/index.js

s3cmd --access_key="$S3_ACCESS_KEY" --secret_key="$S3_SECRET_KEY" \
      --acl-public --no-mime-magic --progress --recursive         \
      --exclude "dist/production/index.js" \
      put dist/production/ 
      "s3://${BUCKET}/something/${BUILD_IDENTIFIER}/production/" &

s3cmd --access_key="$S3_ACCESS_KEY" --secret_key="$S3_SECRET_KEY" \
      --acl-public --no-mime-magic --progress --recursive         \
      --add-header="Content-Encoding:gzip" \
      put dist/production/index.js 
      "s3://${BUCKET}/something/${BUILD_IDENTIFIER}/production/" &

wait

Notice the & in the end of the two commands that makes two uploads to the same location in parallel.



Sep 5, 2018 in AWS by bug_seeker
• 15,350 points
49 views

1 answer to this question.

0 votes

Is your problem is not with the line?

cp dist/production/index.js.gz dist/production/index.js

You are copying gzipped file, not the plain index.js text file.

Hope it helps.

EDIT1:

If you are doing it on purpose why not maintain the gz extension. Extensions does lot of things when you handle with browser.

cp dist/production/index.js.gz dist/production/index.js.gz

If you use plain s3 to download and verify the hash, they should be the same file. I did verify it.

answered Sep 5, 2018 by Priyaj
• 56,520 points

Related Questions In AWS

0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,090 points
3,167 views
0 votes
1 answer

AWS S3 uploading hidden files by default

versioning is enabled in your bucket. docs.aws.amazon.com/AmazonS3/latest/user-guide/….... the ...READ MORE

answered Oct 4, 2018 in AWS by Priyaj
• 56,520 points
200 views
0 votes
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 56,520 points
367 views
0 votes
1 answer

Import my AWS credentials using python script

Using AWS Cli  Configure your IAM user then ...READ MORE

answered Nov 16, 2018 in AWS by Jino
• 5,560 points
253 views
0 votes
2 answers
+1 vote
3 answers

How to get ARN for s3 Bucket using aws cli .

An ARN is a non-opaque, constructible identifier, ...READ MORE

answered Aug 16, 2018 in AWS by Priyaj
• 56,520 points
1,122 views