How to directly read excel file from s3 with pandas in airflow dag

0 votes

Python is not working when I try to read an excel file from S3 inside of an AI flow dag. 

It is quite strange because it works when I read it using pd.read excel(s3 excel path) from outside airflow.


 

How I acted:



 

Create an AWS account in Airflow (this works well as I can list my s3 bucket)

In my Docker environment, where I run Airflow, instal pandas and s3fs.

Pd.read excel(s3 excel path) should be used to attempt to read the file.

I've tried it outside of Airflow, and it functions as stated. 

Furthermore, even after waiting 20 minutes, nothing happens; the dag just keeps running indefinitely (at the stage where it is meant to read the file).

(I am attempting to read the file directly from S3 since I want to have to download it from S3, process it, and then upload it back to S3)

Note that I do not also deal with CSV.

EDIT: In a same vein, df.to csv('s3 path') in the airflow dag does not allow me to save my dataframe directly to S3, but it does in Python.

Jan 3 in AWS by Tejashwini
• 3,770 points
356 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.

Related Questions In AWS

+1 vote
2 answers

How to read a csv file stored in Amazon S3 using csv.DictReader

The code would be something like this: import ...READ MORE

answered Oct 25, 2018 in AWS by Archana
• 5,640 points
51,267 views
0 votes
1 answer

How to upload a file from S3 in Lambda?

Yes there is an option where you ...READ MORE

answered Jan 24, 2019 in AWS by Priyaj
• 58,100 points
4,249 views
0 votes
0 answers

I want to get file name from key in S3 bucket wanted to read single file from list of file present in bucket

1 <class 'boto.s3.key.Key'> <Key: numbers-email, staging/Procured_Numbers_Status/procured_numbers_status_2019-05-15:06:09:04.csv> I ...READ MORE

May 15, 2019 in AWS by anonymous
5,547 views
0 votes
1 answer

AWS S3 uploading hidden files by default

versioning is enabled in your bucket. docs.aws.amazon.com/AmazonS3/latest/user-guide/….... the ...READ MORE

answered Oct 4, 2018 in AWS by Priyaj
• 58,100 points
3,987 views
–1 vote
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 58,100 points
4,035 views
0 votes
1 answer

Import my AWS credentials using python script

Using AWS Cli  Configure your IAM user then ...READ MORE

answered Nov 16, 2018 in AWS by Jino
• 5,810 points
2,113 views
0 votes
2 answers
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP