How to read a csv file stored in Amazon S3 using csv DictReader

+1 vote

I have code that fetches an AWS S3 object. How do I read this StreamingBody with Python's csv.DictReader?

import boto3, csv

session = boto3.session.Session(aws_access_key_id=<>, aws_secret_access_key=<>, region_name=<>)
s3_resource = session.resource('s3')
s3_object = s3_resource.Object(<bucket>, <key>)
streaming_body = s3_object.get()['Body']
Oct 25, 2018 in AWS by datageek
• 3,070 points
33,113 views

2 answers to this question.

+2 votes

The code would be something like this:

import boto3
import csv

# get a handle on s3
s3 = boto3.resource(u's3')

# get a handle on the bucket that holds your file
bucket = s3.Bucket(u'bucket-name')

# get a handle on the object you want (i.e. your file)
obj = bucket.Object(key=u'test.csv')

# get the object
response = obj.get()

# read the contents of the file and split it into a list of lines

lines = response[u'Body'].read().split()

# now iterate over those lines
for row in csv.DictReader(lines):

    # here you get a sequence of dicts
    # do whatever you want with each line here
    print(row)

You can compact this a bit in actual code, but I tried to keep it step-by-step to show the object hierarchy with boto3.

For detailed know-how on AWS check this out! https://www.youtube.com/watch?v=XjPUyGKRjZs

answered Oct 25, 2018 by Archana
• 5,640 points
how to skip the header while reading
Hey, which header are you referring to? what's your query?
next(read, None)
Can you explain your query?
0 votes

Hi,

You can use .get() on an s3.Object() will retrieve the object information and metadata from S3. One of the keys in that dict is Body as shown below.

import boto3

s3 = boto3.resource('s3')
# .get()['Body'] returns a generator!
fileobj = s3.Object('mybucket', 'hello.txt').get()['Body']
fileobj.read(1)  # h
fileobj.read(1)  # e
fileobj.read(1)  # l
fileobj.read(1)  # l
fileobj.read(1)  # o
answered Dec 16, 2020 by MD
• 94,990 points

Related Questions In AWS

0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14, 2019 in AWS by athenauserz
4,865 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,150 points
10,900 views
+4 votes
5 answers

Usage of Amazon Cloudfront or S3

When to use S3? S3 is like many ...READ MORE

answered Apr 3, 2018 in AWS by brat_1
• 7,180 points
373 views
+3 votes
6 answers

Renaming files in S3

You can either use AWS CLI or ...READ MORE

answered Oct 16, 2018 in AWS by petter dj
7,953 views
0 votes
1 answer

S3 Static Website Hosting Route All Paths to Index.html

Yes there is an easy way to ...READ MORE

answered Apr 8, 2018 in AWS by code_ninja
• 6,280 points
1,806 views
+1 vote
4 answers

Can a URL be directly uploaded to S3 using POST?

You can read this blog and get ...READ MORE

answered Oct 25, 2018 in AWS by chamunda
295 views
0 votes
1 answer

How do I read a csv stored in S3 with csv.DictReader?

Hey, I have attached code line by ...READ MORE

answered Nov 27, 2018 in AWS by Archana
• 5,640 points
1,449 views
0 votes
1 answer

How to pull container instances stored in Amazon Elastic Container Registry?

Amazon ECR is integrated with Amazon ECS ...READ MORE

answered Nov 9, 2018 in AWS by Archana
• 5,640 points
62 views