Read file content from S3 bucket with boto3

+1 vote

I read the filenames in my S3 bucket by doing

objs = boto3.client.list_objects(Bucket='my_bucket') while 'Contents' in objs.keys(): objs_contents = objs['Contents'] for i in range(len(objs_contents)): filename = objs_contents[i]['Key']

Now, I need to get the actual content of the file, similarly to a open(filename).readlines(). What is the best way?

Oct 23, 2018 in AWS by datageek
• 2,490 points
22,503 views

2 answers to this question.

+1 vote

boto3 offers a resource model that makes tasks like iterating through objects easier. Unfortunately, StreamingBody doesn't provide readline or readlines.

s3 = boto3.resource('s3')
bucket = s3.Bucket('test-bucket')
# Iterates through all the objects, doing the pagination for you. Each obj
# is an ObjectSummary, so it doesn't contain the body. You'll need to call
# get to get the whole body.
for obj in bucket.objects.all():
    key = obj.key
    body = obj.get()['Body'].read()
answered Oct 23, 2018 by anonymous
where we have to pass the access key and endpoint URL

You could declare it when you create the client

client = boto3.client( 's3', aws_access_key_id="***", aws_secret_access_key="****" )
+1 vote
s3_client=boto3.resource('s3')
bucket = s3_client.Bucket('test')
for obj in bucket.objects.all():
contents=obj.get()['Body'].read().decode(encoding="utf-8",errors="ignore")
for line in contents.splitlines():
print(line)
answered Jul 3, 2019 by reddy

edited Jul 4, 2019 by Kalgi
I tried iterating through a bucket using above code,but I am getting below error:

AttributeError: 'str' object has no attribute 'objects'

Kindly check and advise.

Hi Vishal,

Check if you are passing your bucket in the following line. 

bucket = s3_client.Bucket('test')

You might be getting an error for this line as far I know. s3_client.Bucket should have a bucket argument passed.

Related Questions In AWS

0 votes
0 answers

I want to get file name from key in S3 bucket wanted to read single file from list of file present in bucket

1 <class 'boto.s3.key.Key'> <Key: numbers-email, staging/Procured_Numbers_Status/procured_numbers_status_2019-05-15:06:09:04.csv> I ...READ MORE

May 15, 2019 in AWS by anonymous
1,218 views
+1 vote
2 answers

Want my AWS s3 Bucket to read Name from CloudWatch Event

CloudTrail events for S3 bucket level operations ...READ MORE

answered May 28, 2018 in AWS by Cloud gunner
• 4,650 points
670 views
0 votes
1 answer
+4 votes
13 answers

Python AWS Boto3: How do i read files from S3 Bucket?

You can download the file from S3 ...READ MORE

answered Dec 7, 2018 in AWS by Jino
162,171 views
0 votes
1 answer

How to copy .csv file from Amazon S3 bucket?

Boto3 is the library to use for ...READ MORE

answered Jul 6, 2018 in AWS by Priyaj
• 57,640 points
845 views