Throttling S3 commands with AWS CLI

Question

I'm running a backup script using AWS CLI to perform an S3 sync command on my MediaTemple server.It was working properly for sometime , but I updated my Plesk installation and now when the backup script runs, MediaTemple disables my server due to excessive usage.

There is a limit on networking snapshots at the time they take the server offline which includes many open connections to Amazon IP addresses

Is there anything I can do to throttle the connections to AWS? Preferably I'm looking for an option within the AWS API , but other than that , is there something I can do on my end to manage the connections at the network level?

Answer 1

The AWS CLI S3 transfer commands (which includes sync) have the following relevant configurations :

• max_concurrent_requests -

  • Default: 10

  • The maximum number of concurrent requests.

• multipart_threshold -

  • Default: 8MB

  • The size threshold the CLI uses for multipart transfers of individual files.

• multipart_chunksize -

  • Default: 8MB

  • When using multipart transfers, this is the chunk size that the CLI uses for multipart transfers of individual files.

This isn't as efficient  as throttling packets per second, but it might help if you set concurrent request value and lower both multipart threshold and chunksize.