Synchronizing sqoop imported data in HDFS

0 votes
How can we synchronize the incoming data from sqoop in HDFS, if it's getting updated now and then?
Jul 19, 2018 in Big Data Hadoop by shams
• 3,580 points
69 views

1 answer to this question.

0 votes

Data can be synchronised using incremental parameter with data import –

--Incremental parameter can be used with one of the two options-

i) append-If the table is getting updated continuously with new rows and increasing row id values then incremental import with append option should be used where values of some of the columns are checked (columns to be checked are specified using –check-column) and if it discovers any modified value for those columns then only a new row will be inserted.

ii) lastmodified – In this kind of incremental import, the source has a date column which is checked for. Any records that have been updated after the last import based on the lastmodifed column in the source, the values would be updated.

Hope this helps

answered Jul 19, 2018 by kurt_cobain
• 9,240 points

Related Questions In Big Data Hadoop

+1 vote
1 answer

What is the process to perform an incremental data load in Sqoop?

The process to perform incremental data load ...READ MORE

answered Dec 17, 2018 in Big Data Hadoop by Frankie
• 9,810 points
239 views
0 votes
1 answer

How to securely transfer data from rdms to hdfs using sqoop?

Sqoop stores metadata in a repository and ...READ MORE

answered Dec 18, 2018 in Big Data Hadoop by Omkar
• 67,120 points
62 views
0 votes
1 answer

How to send curl downloaded data in hdfs?

Try using put command with stdin as ...READ MORE

answered Jan 25 in Big Data Hadoop by Omkar
• 67,120 points
18 views
0 votes
1 answer

Update hdfs data before stroring in MySql

Yes, you can update the data before ...READ MORE

answered Jan 27 in Big Data Hadoop by Omkar
• 67,120 points
20 views
0 votes
1 answer

Hadoop Mapreduce word count Program

Firstly you need to understand the concept ...READ MORE

answered Mar 16, 2018 in Data Analytics by nitinrawat895
• 10,030 points
2,027 views
0 votes
10 answers

hadoop fs -put command?

copy command can be used to copy files ...READ MORE

answered Dec 7, 2018 in Big Data Hadoop by Sujay
10,261 views
0 votes
1 answer

Hadoop dfs -ls command?

In your case there is no difference ...READ MORE

answered Mar 16, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
753 views
0 votes
1 answer
0 votes
1 answer

How to transfer data from Netezza to HDFS using Apache Sqoop?

Remove the --direct option. It gives issue ...READ MORE

answered Apr 23, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
230 views
0 votes
1 answer

Why Java Code in Hadoop uses own Data Types instead of basic Data types?

Hadoop provides us Writable interface based data ...READ MORE

answered Apr 15, 2018 in Big Data Hadoop by kurt_cobain
• 9,240 points
57 views