How to read a csv file stored in Amazon S3 using csv DictReader

Question

I have code that fetches an AWS S3 object. How do I read this StreamingBody with Python's csv.DictReader?

import boto3, csv

session = boto3.session.Session(aws_access_key_id=<>, aws_secret_access_key=<>, region_name=<>)
s3_resource = session.resource('s3')
s3_object = s3_resource.Object(<bucket>, <key>)
streaming_body = s3_object.get()['Body']

Answer 1

The code would be something like this:

import boto3
import csv

# get a handle on s3
s3 = boto3.resource(u's3')

# get a handle on the bucket that holds your file
bucket = s3.Bucket(u'bucket-name')

# get a handle on the object you want (i.e. your file)
obj = bucket.Object(key=u'test.csv')

# get the object
response = obj.get()

# read the contents of the file and split it into a list of lines

lines = response[u'Body'].read().split()

# now iterate over those lines
for row in csv.DictReader(lines):

    # here you get a sequence of dicts
    # do whatever you want with each line here
    print(row)

You can compact this a bit in actual code, but I tried to keep it step-by-step to show the object hierarchy with boto3.

For detailed know-how on AWS check this out! https://www.youtube.com/watch?v=XjPUyGKRjZs

Comments

how to skip the header while reading

---

Hey, which header are you referring to? what's your query?

---

next(read, None)

---

Can you explain your query?

Answer 2

Hi,

You can use .get() on an s3.Object() will retrieve the object information and metadata from S3. One of the keys in that dict is Body as shown below.

import boto3

s3 = boto3.resource('s3')
# .get()['Body'] returns a generator!
fileobj = s3.Object('mybucket', 'hello.txt').get()['Body']
fileobj.read(1)  # h
fileobj.read(1)  # e
fileobj.read(1)  # l
fileobj.read(1)  # l
fileobj.read(1)  # o