Boto3 - python script to view all directories and files

0 votes

I tried to follow the Boto3 examples, but can literally only manage to get the very basic listing of all my S3 buckets via the example they give:

import boto3
s3 = boto3.resource('s3')
for bucket in s3.buckets.all():

I cannot find documentation that explains how I would be able to traverse or change into folders and then access individual files.

I'm trying to get to my SNS delivery reports, which are stored in a folder for each day of the month - so it is a pain to manually have to download each file for the month and then to concatenate the contents of each file in order to get the count of all SMS messages sent for a month.

Does anyone have an example of a script that can help me with this, or pointers to really basic documentation/examples of helping me to do this?

I have 3 S3 buckets, and all the files are located in sub folders in one of them:


etc etc Underneath the "Day" folder is a single text file called 001.txt SO I am trying to concatenate all the 001.txt files for each day of a month and then find the rowcount of the concatenated text file - which would give me the count of all SMS sent - successful and failed.

Any help, much appreciated.

Sep 14, 2018 in AWS by bug_seeker
• 15,520 points

1 answer to this question.

0 votes

There are no folders, only S3 object keys.

Using the Bucket Resource interface, you can filter the list of objects in a bucket using the objects collection filter() method (see example).

You can also use the Client interface to call list_objects() with a suitable prefix and delimiter to retrieve subsets of objects.

See Listing Keys Hierarchically for a high-level description.

Check this out for an end to end walkthrough of AWS

Hope this helps!!

If you need to learn more about Python, It's recommended to join Python Training today.


answered Sep 14, 2018 by Priyaj
• 58,090 points

Related Questions In AWS

0 votes
1 answer

How to download S3 Objects With Python and Boto3?

Hi@akhtar, You can download one file from a ...READ MORE

answered Oct 8, 2020 in AWS by MD
• 95,440 points
0 votes
1 answer

Wants to transfer files using python

First install pyformulas pip install pyformulas==0.2.7 Then run the ...READ MORE

answered Jun 2, 2018 in AWS by Flying geek
• 3,280 points
+5 votes
14 answers

Python AWS Boto3: How do i read files from S3 Bucket?

You can use the following code, import boto3 s3 ...READ MORE

answered Dec 7, 2018 in AWS by Nitesh
0 votes
1 answer

Boto3 - python script to view all directories and files

There are no folders, only S3 object ...READ MORE

answered Sep 17, 2018 in AWS by Priyaj
• 58,090 points
0 votes
1 answer

what php API can be used to upload and download files to Amazon S3

Amazon have a PHPSDK , check the sample code // ...READ MORE

answered Oct 17, 2018 in AWS by Priyaj
• 58,090 points
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP