Google Hadoop Filesystem Encryption

0 votes

In normal operation one can provide encryption keys to the google storage api to encrypt a given bucket/blob: https://cloud.google.com/compute/docs/disks/customer-supplied-encryption

Is this possible for the output of spark/hadoop jobs "on the fly"?

Say we wanted to encrypt the output of a spark write

 df.write().format("json").save("gs:///somebucket/output");

In https://storage.googleapis.com/hadoop-conf/gcs-core-default.xml there is no way to specify an encryption key.

Is this possible to do?

Oct 18, 2018 in Big Data Hadoop by Neha
• 6,280 points
58 views

1 answer to this question.

0 votes

If you're asking if customer-supplied encryption keys are currently available on Cloud Dataproc, the answer is no. Here is a list of current products options for encryption at rest at Google.

If you were just looking to encrypt the output of a Spark write, you could still encrypt this at the application layer using Google's Cloud KMS. Here's a codelab for doing so in Google Cloud Storage (which looks like what you're doing with the command above). Note that customer content is encrypted at rest on Google cloud platform by default at the storage layer, so this is another layer of protection.

answered Oct 18, 2018 by Frankie
• 9,810 points

Related Questions In Big Data Hadoop

0 votes
1 answer

HortonWorks Hadoop encryption tools and data security

There are many tools available for encrypting ...READ MORE

answered Apr 20, 2018 in Big Data Hadoop by nitinrawat895
• 10,510 points
97 views
0 votes
1 answer
0 votes
1 answer

The file exists before processing with hadoop command

Took session and it got resolved. READ MORE

answered Dec 18, 2017 in Big Data Hadoop by Sudhir
• 1,610 points
59 views
0 votes
1 answer

How to sync Hadoop configuration files to multiple nodes?

For syncing Hadoop configuration files, you have ...READ MORE

answered Jun 21, 2018 in Big Data Hadoop by HackTheCode
147 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,510 points
2,397 views
0 votes
1 answer

hadoop.mapred vs hadoop.mapreduce?

org.apache.hadoop.mapred is the Old API  org.apache.hadoop.mapreduce is the ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,510 points
244 views
0 votes
10 answers

hadoop fs -put command?

put syntax: put <localSrc> <dest> copy syntax: copyFr ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Aditya
12,200 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
896 views
0 votes
1 answer

Hadoop fs mkdir and testing existence FileSystem.exists

FileSystem.get(conf) may return the local file system where ...READ MORE

answered Sep 18, 2018 in Big Data Hadoop by Frankie
• 9,810 points
301 views
+1 vote
1 answer