Read file content from S3 bucket with boto3

Question

I read the filenames in my S3 bucket by doing

objs = boto3.client.list_objects(Bucket='my_bucket') while 'Contents' in objs.keys(): objs_contents = objs['Contents'] for i in range(len(objs_contents)): filename = objs_contents[i]['Key']

Now, I need to get the actual content of the file, similarly to a open(filename).readlines(). What is the best way?

Answer 1

boto3 offers a resource model that makes tasks like iterating through objects easier. Unfortunately, StreamingBody doesn't provide readline or readlines.

s3 = boto3.resource('s3')
bucket = s3.Bucket('test-bucket')
# Iterates through all the objects, doing the pagination for you. Each obj
# is an ObjectSummary, so it doesn't contain the body. You'll need to call
# get to get the whole body.
for obj in bucket.objects.all():
    key = obj.key
    body = obj.get()['Body'].read()

For a detailed explanation on S3, check this out!

https://www.youtube.com/watch?v=XjPUyGKRjZs

Hope this helps!

Comments

where we have to pass the access key and endpoint URL

---

You could declare it when you create the client

client = boto3.client( 's3', aws_access_key_id="***", aws_secret_access_key="****" )

Answer 2

s3_client=boto3.resource('s3')
bucket = s3_client.Bucket('test')
for obj in bucket.objects.all():
contents=obj.get()['Body'].read().decode(encoding="utf-8",errors="ignore")
for line in contents.splitlines():
print(line)

Comments

I tried iterating through a bucket using above code,but I am getting below error:

AttributeError: 'str' object has no attribute 'objects'

Kindly check and advise.

---

Hi Vishal,

Check if you are passing your bucket in the following line. 

bucket = s3_client.Bucket('test')

You might be getting an error for this line as far I know. s3_client.Bucket should have a bucket argument passed.