Question

What are all the Data quality checks we do in our real time Bigdata projects.
Example1: How can we find out the count of records loaded in hdfs and source are same?
Example2: How can we know the loaded records in hdfs are proper?

Answer 1

You can use a checksum to compare the file in the source and the file uploaded on the hdfs. 

Try this: 

$ hdfs dfs -cat /file/in/hdfs | md5sum

$ hdfs dfs -cat /file/at/source | md5sum

If these two commands return the same value, then the file is not corrupted. 

Comments

Thanks for the help, but i am copying from mysql table to hdfs in that scenario, if one record corrupted then how can we know that?

---

I'm not a 100% sure but I think you can use crc32 checksum as follows:

For the mysql table, use the below command to get the checksum:

CHECKSUM TABLE <tablename>;

And in hdfs, use this:

hdfs dfs -Ddfs.checksum.combine.mode=COMPOSITE_CRC -checksum /path/to/file