What is the difference between S3n S3a and S3

0 votes

I have been through this page related to Amazon S3.

Also, I have read the documentation that states as follows:

S3 Native FileSystem (URI scheme: s3n) A native filesystem for reading and writing regular files on S3. The advantage of this filesystem is that you can access files on S3 that were written with other tools. Conversely, other tools can access files written using Hadoop. The disadvantage is the 5GB limit on file size imposed by S3.

S3A (URI scheme: s3a) A successor to the S3 Native, s3n fs, the S3a: the system uses Amazon's libraries to interact with S3. This allows S3a to support larger files (no more 5GB limit), higher performance operations and more. The filesystem is intended to be a replacement for/successor to S3 Native: all objects accessible from s3n:// URLs should also be accessible from s3a simply by replacing the URL schema.

S3 Block FileSystem (URI scheme: s3) A block-based filesystem backed by S3. Files are stored as blocks, just like they are in HDFS. This permits efficient implementation of renames. This filesystem requires you to dedicate a bucket for the filesystem - you should not use an existing bucket containing files, or write other files to the same bucket. The files stored by this filesystem can be larger than 5GB, but they are not interoperable with other S3 tools.

Yet, my doubt is, how does a single word in the URL could make such a big difference?

I would drop down a code for better understandability.

val data = sc.textFile("s3n://bucket-name/key")


val data = sc.textFile("s3a://bucket-name/key")

Can anyone help me to understand the technical difference underlying this change? Can you suggest any good articles that I can read on this?

Jul 30, 2019 in Big Data Hadoop by nitinrawat895
• 11,380 points

1 answer to this question.

0 votes

Your doubt is quite an interesting one.

Yes, the difference between one single word can make a huge difference. To understand this in a better way, you can consider the difference between HTTP and HTTPS.

Similarly, S3a, S3n, and S3 work with different interfaces. Hence, one single word can make a huge difference.

S3a and S3n are an Object-Based overlay on top of Amazon S3, while, on the other hand, S3 is a Block-Based overlay on top of Amazon S3.

  1. S3n is capable to support up to 5Gigabytes sized objects.
  2. S3a is capable to support up to 5Terrabytes sized objects. It is the successor of S3n.
For more detailed information on (Amazon EMR) S3, S3n, and S3a, you can go through this article.
The net is: use s3:// because s3:// and s3n:// are functionally interchangeable in the context of EMR, while s3a:// is not compatible with EMR.
For more detailed information, you can go to Amazon official Documentation.
I hope this helps. Happy Learning...;-)
answered Jul 30, 2019 by ravikiran
• 4,620 points

Related Questions In Big Data Hadoop

0 votes
10 answers

What is the difference between Mongodb and Hadoop?

MongoDB is a NoSQL database, whereas Hadoop is ...READ MORE

answered Jun 20, 2018 in Big Data Hadoop by jenny_code
0 votes
1 answer
0 votes
1 answer
0 votes
2 answers

What is the difference between the Smart Data Access of SAP HANA and SAP HANA Vora?

Hadoop: Used to store Big Data in ...READ MORE

answered Jun 21, 2018 in Big Data Hadoop by kurt_cobain
• 9,390 points
0 votes
1 answer

AWS S3 uploading hidden files by default

versioning is enabled in your bucket. docs.aws.amazon.com/AmazonS3/latest/user-guide/….... the ...READ MORE

answered Oct 4, 2018 in AWS by Priyaj
• 58,090 points
–1 vote
1 answer

How to decrypt the encrypted S3 file using aws-encryption-cli --decrypt

Use command : aws s3 presign s3://mybucket/abc_count.png you get ...READ MORE

answered Oct 22, 2018 in AWS by Priyaj
• 58,090 points
0 votes
1 answer

Import my AWS credentials using python script

Using AWS Cli  Configure your IAM user then ...READ MORE

answered Nov 16, 2018 in AWS by Jino
• 5,820 points
0 votes
2 answers
0 votes
2 answers

What is the relationship between Hadoop and Database?

Hadoop software framework work is very well ...READ MORE

answered Aug 6, 2019 in Big Data Hadoop by Dinesh
0 votes
1 answer

Explain to me the difference between HBase and HDFS.

Hadoop generally consists of three major components: HDFS It ...READ MORE

answered Apr 12, 2019 in Big Data Hadoop by ravikiran
• 4,620 points
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP