Filtering Out the Unique records and populating the records with Max count using Talend

Question

I have a real-time data table in which data is updated in every 24 hours. From this table, I am pulling in the data using an excel sheet. Sometimes with regular updates, it leads to duplication of data along with null values at some places. Can someone help me in filtering out the unique rows with most number of columns?

Answer 1

Hi...I think in your case you can make use of tUniqRow component. In order to get the required output, you can make use of a tJavaRow component to specify the conditions and then sort the incoming rows and then pass them to the tUniqRow component.

Your job flow should look somewhat like below:

In the tJavaRow component, you need to add the below code to filter out the required data:

// Code generated according to input schema and output schema
output_row.id = input_row.id;
output_row.name       = input_row.name;
output_row.address = input_row.address;
// End of pre-generated code

int i = 0;
if (input_row.id == null) { i++; }
if (input_row.name       == null) { i++; }
if (input_row.address  == null) { i++; }
output_row.priority = i;

Hope this helps!!

Join the Talend course today to know more about and become expert.

Thanks.