Output Splitting problem in Hadoop

Question

I ran the following script with two files as input, the output was split into two file part-m-00000 and part-m-00001. I couldn't understand why, please assist me? Note: The size in only 8.2 MB for each file.

REGISTER PIG/PigUDF.jar;

A = LOAD "PIG/HealthCare/Input/healthcare_Sample_dataset1.csv" USING PigStorage(",") AS (patientID:int, name:chararray, date:chararray, phoneNumber:chararray, eMail:chararray, SSN:chararray, gender:chararray, disease:chararray, age:chararray);

B = LOAD "PIG/HealthCare/Input/healthcare_Sample_dataset2.csv" USING PigStorage(",") AS (patientID:int, name:chararray, date:chararray, phoneNumber:chararray, eMail:chararray, SSN:chararray, gender:chararray, disease:chararray, age:chararray);

C = UNION A, B;

D = FOREACH C GENERATE patientID, com.kamran.pig.udf.encryptField(name,"12345678abcdefgh"), com.kamran.pig.udf.encryptField(date,"12345678abcdefgh"), com.kamran.pig.udf.encryptField(phoneNumber,"12345678abcdefgh"), com.kamran.pig.udf.encryptField(eMail,"12345678abcdefgh"), com.kamran.pig.udf.encryptField(SSN,"12345678abcdefgh"), com.kamran.pig.udf.encryptField(gender,"12345678abcdefgh"), com.kamran.pig.udf.encryptField(disease,"12345678abcdefgh"), age;

STORE D INTO "PIG/HealthCare/Output/HealthCareOutput.csv";

Answer 1

When you are loading two different files, it is not mandatory that the files are getting loaded into the same data block. It might get loaded into different data blocks and for each block, separate mappers might be running on them. Since the data might be present in different nodes, it can easily create different part files.

You can check by loading a small file to pig and try processing it, this is going to create a single part file in the output.

Refer below:

A = load 'weatherPIG.txt' using TextLoader as (date:chararray);

AF = foreach A generate TRIM(SUBSTRING(data, 6, 14)), TRIM(SUBSTRING(data, 46, 53)), TRIM(SUBSTRING(data, 38, 45));

store AF into 'pigudf32' using PigStorage(',');

You can check pigudf32, this folder is supposed to consist of a single part file.