I want to read .csv file from S3 and load/write the same data to cassandra.

0 votes
I want to read the .csv file from S3 and load/write the same data to Cassandra.
FYI: I have Access Key and Secret Key to connect to S3.
Any help?
Sep 4 in AWS by Raj
• 120 points

recategorized Sep 4 by MD 112 views

1 answer to this question.

0 votes

Hi@Raj,

You need to use the connector that connects your S3 bucket to your database. After that, you can use the commands.

First, read the file from s3 into an RDD in your system.

val rdd = sc.textFile("s3n://mybucket/file_name")

After that, load the red to Cassandra table. Also, check the proper format.

rdd.saveToCassandra()
answered Sep 4 by MD
• 64,820 points
@MD, I am running my code on EC2. Using Spark, How do I write it to Cassandra, Please share your thoughts.

Hi,

As you said your spark is in EC2 instance. So first read the dataset in a data frame and then use the format of Cassandra to save the dataset in Cassandra table. You can use the below command.

val DF = sqlContext.applySchema(rowRDD, schema)
DF.write.format("org.apache.spark.sql.cassandra").options(Map( "table" -> "name", "ID" -> "test")).save()

Related Questions In AWS

0 votes
0 answers

I want to get file name from key in S3 bucket wanted to read single file from list of file present in bucket

1 <class 'boto.s3.key.Key'> <Key: numbers-email, staging/Procured_Numbers_Status/procured_numbers_status_2019-05-15:06:09:04.csv> I ...READ MORE

May 15, 2019 in AWS by anonymous
1,434 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer