How to read data from S3 in a regular inetrval using Spark Scala

0 votes

I was trying to find application for my need and found one java application which dump data(in csv file) to s3 on daily basis.

This application create folder in S3 based on system date like(MM-DD-YYYY format) and then add files to the folder created.

Now i want to read those files from S3 on regular interval like

val fileFromS3= sc.textFile("s3a://digital/MM-DD-YYYY/abc.csv")

Now the script should replace 'MM-DD-YYYY' with the system date.

Please suggest possible solution or any other way to achieve this.

Oct 31, 2018 in AWS by findingbugs
• 3,140 points
674 views

1 answer to this question.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
0 votes

Inorder to get it done first you need to import calender

First you need some imports:

import java.util.Calendar
import java.text.SimpleDateFormat
import java.util.Date
val now = Calendar.getInstance().getTime()  //you can get the current Time:
val formatter = new SimpleDateFormat("MM-dd-yyyy")
val dateAsString = formatter.format(now)   //use your formatter to get the date as String

Then you can load your resource with the dateAsString value using String interpolation:

val fileFromS3= sc.textFile(s"s3a://digital/${dateAsString}/abc.csv")
answered Oct 31, 2018 by Priyaj
• 56,140 points

Related Questions In AWS

–1 vote
1 answer

How to read a csv file stored in Amazon S3 using csv.DictReader

The code would be something like this: import ...READ MORE

answered Oct 25, 2018 in AWS by Archana
• 5,360 points
4,210 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,090 points
1,856 views
0 votes
1 answer

AWS S3 uploading hidden files by default

versioning is enabled in your bucket. docs.aws.amazon.com/AmazonS3/latest/user-guide/….... the ...READ MORE

answered Oct 4, 2018 in AWS by Priyaj
• 56,140 points
107 views
0 votes
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 56,140 points
219 views
0 votes
1 answer

Import my AWS credentials using python script

Using AWS Cli  Configure your IAM user then ...READ MORE

answered Nov 16, 2018 in AWS by Jino
• 5,520 points
133 views
0 votes
2 answers
0 votes
1 answer

© 2018 Brain4ce Education Solutions Pvt. Ltd. All rights Reserved.
"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.