Update hdfs data before stroring in MySql

Question

Hi guys. I am new to Hadoop and I have some doubt. I am learning hdfs and Sqoop. I created a table in SQL and then imported it to hdfs using Sqoop. Now I want to store the data in SQL from hdfs again. But the problem is that I want to make some changes to the data before storing it in SQL. How can I do this? Is intermediate data updation possible?

Answer 1

Yes, you can update the data before storing it in SQL table. In this approach, you will have to update the column using hive and store the hive output to hdfs using INSERT OVERWRITE DIRECTORY. And after that, to store it in SQL, you can use the Sqoop command. Example:

sqoop export \
--connect jdbc:mysql://master/poc \
--username root \
--table employee \
--export-dir /user/hdfs/mysql/export.txt \
--update-key id \
--update-mode allowinsert \
--fields-terminated-by '\t' \
-m 1