How to read a csv file stored in Amazon S3 using csv DictReader

+1 vote

I have code that fetches an AWS S3 object. How do I read this StreamingBody with Python's csv.DictReader?

import boto3, csv

session = boto3.session.Session(aws_access_key_id=<>, aws_secret_access_key=<>, region_name=<>)
s3_resource = session.resource('s3')
s3_object = s3_resource.Object(<bucket>, <key>)
streaming_body = s3_object.get()['Body']
Oct 25, 2018 in AWS by datageek
• 3,090 points
54,450 views

2 answers to this question.

+2 votes

The code would be something like this:

import boto3
import csv

# get a handle on s3
s3 = boto3.resource(u's3')

# get a handle on the bucket that holds your file
bucket = s3.Bucket(u'bucket-name')

# get a handle on the object you want (i.e. your file)
obj = bucket.Object(key=u'test.csv')

# get the object
response = obj.get()

# read the contents of the file and split it into a list of lines

lines = response[u'Body'].read().split()

# now iterate over those lines
for row in csv.DictReader(lines):

    # here you get a sequence of dicts
    # do whatever you want with each line here
    print(row)

You can compact this a bit in actual code, but I tried to keep it step-by-step to show the object hierarchy with boto3.

For detailed know-how on AWS check this out! https://www.youtube.com/watch?v=XjPUyGKRjZs

answered Oct 25, 2018 by Archana
• 5,640 points
how to skip the header while reading
Hey, which header are you referring to? what's your query?
next(read, None)
Can you explain your query?
0 votes

Hi,

You can use .get() on an s3.Object() will retrieve the object information and metadata from S3. One of the keys in that dict is Body as shown below.

import boto3

s3 = boto3.resource('s3')
# .get()['Body'] returns a generator!
fileobj = s3.Object('mybucket', 'hello.txt').get()['Body']
fileobj.read(1)  # h
fileobj.read(1)  # e
fileobj.read(1)  # l
fileobj.read(1)  # l
fileobj.read(1)  # o
answered Dec 16, 2020 by MD
• 95,460 points

Related Questions In AWS

0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14, 2019 in AWS by athenauserz
11,870 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,170 points
19,878 views
+4 votes
5 answers

Usage of Amazon Cloudfront or S3

When to use S3? S3 is like many ...READ MORE

answered Apr 3, 2018 in AWS by brat_1
• 7,200 points
2,015 views
+3 votes
6 answers

Renaming files in S3

You can either use AWS CLI or ...READ MORE

answered Oct 16, 2018 in AWS by petter dj
31,281 views
0 votes
1 answer

S3 Static Website Hosting Route All Paths to Index.html

Yes there is an easy way to ...READ MORE

answered Apr 8, 2018 in AWS by code_ninja
• 6,300 points
3,450 views
+1 vote
4 answers

Can a URL be directly uploaded to S3 using POST?

You can read this blog and get ...READ MORE

answered Oct 25, 2018 in AWS by chamunda
3,316 views
0 votes
1 answer

How do I read a csv stored in S3 with csv.DictReader?

Hey, I have attached code line by ...READ MORE

answered Nov 27, 2018 in AWS by Archana
• 5,640 points
3,016 views
0 votes
1 answer

How to pull container instances stored in Amazon Elastic Container Registry?

Amazon ECR is integrated with Amazon ECS ...READ MORE

answered Nov 9, 2018 in AWS by Archana
• 5,640 points
764 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP