Throttling S3 commands with AWS CLI

0 votes

I'm running a backup script using AWS CLI to perform an S3 sync command on my MediaTemple server.It was working properly for sometime , but I updated my Plesk installation and now when the backup script runs, MediaTemple disables my server due to excessive usage.

There is a limit on networking snapshots at the time they take the server offline which includes many open connections to Amazon IP addresses

Is there anything I can do to throttle the connections to AWS? Preferably I'm looking for an option within the AWS API , but other than that , is there something I can do on my end to manage the connections at the network level?

Aug 2, 2018 in AWS by datageek
• 2,440 points
50 views

1 answer to this question.

0 votes

The AWS CLI S3 transfer commands (which includes sync) have the following relevant configurations :

  • max_concurrent_requests -

    • Default: 10

    • The maximum number of concurrent requests.

  • multipart_threshold -

    • Default: 8MB

    • The size threshold the CLI uses for multipart transfers of individual files.

  • multipart_chunksize -

    • Default: 8MB

    • When using multipart transfers, this is the chunk size that the CLI uses for multipart transfers of individual files.

This isn't as efficient  as throttling packets per second, but it might help if you set concurrent request value and lower both multipart threshold and chunksize.

answered Aug 4, 2018 by Archana
• 4,090 points

Related Questions In AWS

+1 vote
3 answers

How to get ARN for s3 Bucket using aws cli .

An ARN is a non-opaque, constructible identifier, ...READ MORE

answered Aug 16, 2018 in AWS by Priyaj
• 56,520 points
1,123 views
0 votes
1 answer

AWS:Using S3 with EC2 instances

Yes, it can be used for instances ...READ MORE

answered Jun 26, 2018 in AWS by hemant
• 5,750 points
113 views
0 votes
1 answer
0 votes
1 answer

Invalid Bucket name when creating s3 bucket with AWS SDK

AWS provides naming standards when naming an ...READ MORE

answered Aug 30, 2018 in AWS by Priyaj
• 56,520 points
268 views
0 votes
1 answer

How to use AWS CLI with Elastic Beanstalk?

You have to create a source bundle ...READ MORE

answered Sep 28, 2018 in AWS by Archana
• 4,090 points
139 views
0 votes
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 56,520 points
367 views
0 votes
1 answer

AWS S3 CLI : error while trying to copy files locally using terminal

For the first error you should add ...READ MORE

answered Aug 3, 2018 in AWS by Archana
• 4,090 points
1,129 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,090 points
3,170 views