How to perform Near-real time analytics using Hadoop

Question

We have PostgreSQL in our production server. We have to analyse it on a regular basis, for which we are planning to use Hadoop. Each query which we are executing should be executed on the updated data present in our PostgreSQL database. All this process can only handle a little delay.
Can anyone help me in understanding how to perform this near-real time analytics using Hadoop?

Answer 1

To perform Near-real time analytics, you have to use Storage Handler API of Hive. Storage Handler API basically keeps the metadata of the external table in the Hive metastore, whereas the tables reside in the external data source(in your case it is postgreSQL).

But the Hive metastore will have all the information of the data (i.e. metadata) i.e. coulnm & format details. So, whenever we are reading data from Hive table with storage handler then the data is fetched from the external table.

You can go to this link & check the Hive handler for PostgreSQL.

https://github.com/myui/HiveJdbcStorageHandler

To know more about handlers, check this link:

https://cwiki.apache.org/confluence/display/Hive/StorageHandlers