Google Hadoop Filesystem Encryption

0 votes

In normal operation one can provide encryption keys to the google storage api to encrypt a given bucket/blob: https://cloud.google.com/compute/docs/disks/customer-supplied-encryption

Is this possible for the output of spark/hadoop jobs "on the fly"?

Say we wanted to encrypt the output of a spark write

 df.write().format("json").save("gs:///somebucket/output");

In https://storage.googleapis.com/hadoop-conf/gcs-core-default.xml there is no way to specify an encryption key.

Is this possible to do?

Oct 18, 2018 in Big Data Hadoop by Neha
• 6,300 points
620 views

1 answer to this question.

0 votes

If you're asking if customer-supplied encryption keys are currently available on Cloud Dataproc, the answer is no. Here is a list of current products options for encryption at rest at Google.

If you were just looking to encrypt the output of a Spark write, you could still encrypt this at the application layer using Google's Cloud KMS. Here's a codelab for doing so in Google Cloud Storage (which looks like what you're doing with the command above). Note that customer content is encrypted at rest on Google cloud platform by default at the storage layer, so this is another layer of protection.

answered Oct 18, 2018 by Frankie
• 9,830 points

Related Questions In Big Data Hadoop

0 votes
1 answer

HortonWorks Hadoop encryption tools and data security

There are many tools available for encrypting ...READ MORE

answered Apr 20, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
1,067 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer

The file exists before processing with hadoop command

Took session and it got resolved. READ MORE

answered Dec 18, 2017 in Big Data Hadoop by Sudhir
• 1,610 points
1,000 views
+1 vote
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
10,927 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 11,380 points
2,454 views
+2 votes
11 answers

hadoop fs -put command?

Hi, You can create one directory in HDFS ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by nitinrawat895
• 11,380 points
108,322 views
–1 vote
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
4,547 views
0 votes
1 answer

Hadoop fs mkdir and testing existence FileSystem.exists

FileSystem.get(conf) may return the local file system where ...READ MORE

answered Sep 18, 2018 in Big Data Hadoop by Frankie
• 9,830 points
3,293 views
+1 vote
1 answer
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP