How to read a csv file stored in Amazon S3 using csv.DictReader

–1 vote

I have code that fetches an AWS S3 object. How do I read this StreamingBody with Python's csv.DictReader?

import boto3, csv

session = boto3.session.Session(aws_access_key_id=<>, aws_secret_access_key=<>, region_name=<>)
s3_resource = session.resource('s3')
s3_object = s3_resource.Object(<bucket>, <key>)
streaming_body = s3_object.get()['Body']
Oct 25, 2018 in AWS by datageek
• 3,050 points

1 answer to this question.

0 votes

The code would be something like this:

import boto3
import csv

# get a handle on s3
s3 = boto3.resource(u's3')

# get a handle on the bucket that holds your file
bucket = s3.Bucket(u'bucket-name')

# get a handle on the object you want (i.e. your file)
obj = bucket.Object(key=u'test.csv')

# get the object
response = obj.get()

# read the contents of the file and split it into a list of lines

lines = response[u'Body'].read().split()

# now iterate over those lines
for row in csv.DictReader(lines):

    # here you get a sequence of dicts
    # do whatever you want with each line here

You can compact this a bit in actual code, but I tried to keep it step-by-step to show the object hierarchy with boto3.

For detailed know-how on AWS check this out!

answered Oct 25, 2018 by Archana
• 5,600 points
how to skip the header while reading
Hey, which header are you referring to? what's your query?
next(read, None)
Can you explain your query?

Related Questions In AWS

0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14, 2019 in AWS by athenauserz
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,150 points
+4 votes
5 answers

Usage of Amazon Cloudfront or S3

When to use S3? S3 is like many ...READ MORE

answered Apr 3, 2018 in AWS by brat_1
• 7,170 points
+3 votes
6 answers

Renaming files in S3

You can either use AWS CLI or ...READ MORE

answered Oct 16, 2018 in AWS by petter dj
0 votes
1 answer

S3 Static Website Hosting Route All Paths to Index.html

Yes there is an easy way to ...READ MORE

answered Apr 8, 2018 in AWS by code_ninja
• 6,270 points
+1 vote
4 answers

Can a URL be directly uploaded to S3 using POST?

You can read this blog and get ...READ MORE

answered Oct 25, 2018 in AWS by chamunda
0 votes
1 answer

How do I read a csv stored in S3 with csv.DictReader?

Hey, I have attached code line by ...READ MORE

answered Nov 27, 2018 in AWS by Archana
• 5,600 points
0 votes
1 answer

How to pull container instances stored in Amazon Elastic Container Registry?

Amazon ECR is integrated with Amazon ECS ...READ MORE

answered Nov 9, 2018 in AWS by Archana
• 5,600 points