How compression works in Hadoop

Question

In my Map Reduce job, let us say, I specify the compression for either the map or reduce output to LZO, how does it get compressed? Is the entire data from map or reduce task first obtained without compression  and  then the uncompressed data gets compressed, or does it get incrementally gets  compressed and written. If it gets incrementally compressed and written, then how is it done?

Answer 1

It basically depends on the file type you use. If it is a text file then compression happens at the file level. 

But if it is SequenceFile then compression could be at record level or block level. 

Note that here block means a buffer in using sequence file and not the hdfs block.

If it is block compression then multiple records are compressed into a block at once. Records are added to a block until it reaches a minimum size in bytes. The maximum size of input data to be compressed at a time is calculated by subtracting the maximum overhead of the compression algorithm from the buffer size. The default buffer size is 512 bytes and for compression overhead it's 18 bytes(1% of bufferSize + 12 bytes) for zlib algorithm. 

Then a BlockCompressorStream is created with given output-stream and compressor and the compressed data gets written.

If you specify compression for the map stage (mapreduce.map.output.compress=true) the intermediate map output data will be compressed using whatever code you’ve specified (mapreduce.map.ouput.compress.codec=org.apache.hadoop.io.compress.*) and saved to disk at the conclusion of each map task’s completion (or earlier if the map task exceeds it serialization buffer limit, and begins to spill to the disk). The compressed data is then read from the disk and sent to the appropriate nodes during the Shuffle & Sort stage of your mapreduce job.

I hope this answer helps :)