How to combine multiple S3 objects in the target S3 object w/o leaving S3

+1 vote

I understand that the minimum part size for uploading to an S3 bucket is 5MB Is there any way to have this changed on a per-bucket basis?

The reason I'm asking is there is a list of raw objects in S3 which we want to combine in the single object in S3.

Using PUT part/copy we are able to "glue" objects in the single one providing that all objects except last one are >= 5MB. However sometimes our raw objects are not big enough and in this case when we try to complete multipart uploading we're getting famous error "Your proposed upload is smaller than the minimum allowed size" from AWS S3.

Any other idea how we could combine S3 objects without downloading them first?

Aug 28, 2018 in AWS by bug_seeker
• 14,970 points
972 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

"However sometimes our raw objects are not big enough... "

You can have a 5MB garbage object sitting on S3 and do concatenation with it where part 1 = 5MB garbage object, part 2 = your file that you want to concatenate. Keep repeating this for each fragment and finally use the range copy to strip out the 5MB garbage

answered Aug 28, 2018 by Priyaj
• 56,120 points

Related Questions In AWS

0 votes
1 answer

How to list all the objects in Amazon S3?

As stated already, Amazon S3 indeed requires ...READ MORE

answered Oct 5, 2018 in AWS by Archana
• 3,770 points
95 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 3,770 points
1,523 views
0 votes
1 answer

How to upload an object into Amazon S3 in Lambda?

I suspect you are calling the context.done() function before s3.upload() has ...READ MORE

answered Sep 27, 2018 in AWS by Archana
• 3,770 points
22 views
0 votes
1 answer

How to make multiple files in Amazon S3 public?

I had to change several hundred thousand ...READ MORE

answered Oct 17, 2018 in AWS by Archana
• 3,770 points
38 views
+13 votes
2 answers
+1 vote
2 answers

AWS CloudWatch Logs in Docker

The awslogs works without using ECS. you need to configure ...READ MORE

answered Sep 6, 2018 in AWS by bug_seeker
• 14,970 points
229 views
0 votes
1 answer

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.