Comparing technologies for Data Ingestion

0 votes

I'm trying to build an IoT + data analytics system and I'm having trouble deciding on what technology or service to use for ingestion.

A high level description of the end goal is:

  1. IoT devices push data to an IoT gateway (using Zigbee, Z-wave, Bluetooth, etc)
  2. IoT gateway (which is connected to the internet) pushes data to a pub/sub system
  3. Backend services process the data coming out of the pub/sub system, updates dashboards and sends out alerts

My question is what kind of pubsub system should we use if we only need ~10 second response time? (E.g. The following is acceptable: IoT device senses event and then about 10 seconds later it shows up on a user's dashboard or sets off an alert)

Below are some questions I have:

  1. I see PubNub advertised a lot for use in IoT. My understanding is that PubNub is basically a very fast pubsub system that guarantees less than 1/4 second delivery- is this a correct understanding of it? But PubNub is a bit expensive compared to using Google's Pub/Sub or maintaining our own Kafka.
  2. Is Google Pub/Sub in a sense similar to PubNub, because unlike being a single self-managed Kafka cluster running in a single data center, Google Pub/Sub has its own network because it's a part of Google? (similar to how PubNub is a "data-streams-network"?)
  3. If I use Kafka, should the producer be in the gateway devices?
    1. If the producer isn't in the gateway devices, then should the Kafka producers be on our servers and have a REST API to accept messages from the gateway devices?
    2. If the Kafka producer IS in the gateway devices, does there need to be anything special in front of the Kafka brokers for them to accept messages from the gateway devices?
  4. PubNub can be used to send commands back to the IoT devices. Can this also be done with Google Pub/Sub or Kafka?
    1. To push commands out to the IoT devices With Kafka, would every Gateway device need a consumer that is waiting for messages from the topics it's subscribed to? (e.g. the commands)

Also, not sure if it's worth mentioning, but currently, the team is just me and maybe 2 other full stack developers. We've read up on Kafka and Zookeeper but none of us have gone past rolling out a tutorial example of it.

Sep 25, 2018 in IoT (Internet of Things) by Matt
• 2,260 points
22 views

1 answer to this question.

0 votes

All great questions. (Full disclosure I work for PubNub)

PubNub is way more then just pub/sub (I will get to that later on). First PubNub was built as a global distributed network for a reason - so we could provide low-latency connectivity for all devices around the world through local Points of Presence. Because of this distributed architecture your devices will always connect to the POP closest to them and because we replicate messages globally, if a server or node goes down you will automatically be reconnected to the next closest node with no message lose. Because of this, PubNub provides 99.999% uptime SLA's to all customers.

When it comes to build vs. buy decision, I can tell you that many of our customers started by thinking they could build it themselves but quickly realized the undertaking was way more then they expected - read more here. Building and maintaining the client libraries, scaling the backend, 24-7 monitoring, and security are all things that you would need to have in-house expertise. If you take into consideration the upfront development costs, on-going maintenance - vs. downloading an SDK today, start coding today so you can get to market faster with a known scalable solution.

PubNub is priced based on transactions, so depending on how many devices and the level of traffic, I would bet the total cost would still be less than one full time employee. And for that you get to choose from over 70+ client SDK's, tap into a proven scalable architecture, take advantage of the security features already built in, have a full team ready and waiting to help 24/7, so you can focus on innovation and not infrastructure.

PubNub is way more then just Pub/Sub. PubNub provides not only realtime messaging, but also state management and serverless compute via their programmable netowrk. PubNub allows you to write and deploy functions in the network, in fact there are already 30+ prebuilt functions available in the PubNub BLOCKS Catalog that allow you to send SMS, emails, and more, when data changes.

PubNub has also created an open source project for building realtime dashboards called Project EON. This makes it super easy to provide realtime visualization for all of your device data.

You are also correct in stating that PubNub can be used for remote device control. And is being used today for just that by Insteon, Logitech, Samsung, Wink and many more.

If you have any additional questions, PubNub has an amazing support staff available 24/7 - support@pubnub.com or click chat on the website.

answered Sep 25, 2018 by anonymous2
• 4,260 points

Related Questions In IoT (Internet of Things)

0 votes
1 answer

Data Modelling for IoT based application in Apache Cassandra

I think creating a single table that has ...READ MORE

answered Dec 3, 2018 in IoT (Internet of Things) by nirvana
• 3,060 points
256 views
0 votes
1 answer

What is the maximum message length for a MQTT broker?

It's not entirely clear what you're asking ...READ MORE

answered Jul 17, 2018 in IoT (Internet of Things) by anonymous2
• 4,260 points
601 views
0 votes
1 answer

Connect Android Things based Raspberry Pi 3 to wifi network for the first time!

Hey, I think its alright!  Your Raspberry Pi ...READ MORE

answered Jul 18, 2018 in IoT (Internet of Things) by nirvana
• 3,060 points
194 views
0 votes
1 answer
0 votes
1 answer

Is Kafka and Zookeeper are required in a Big Data Cluster?

Apache Kafka is one of the components ...READ MORE

answered Mar 22, 2018 in Big Data Hadoop by nitinrawat895
• 10,670 points
374 views
0 votes
1 answer

Getting error while connecting zookeeper in Kafka - Spark Streaming integration

I guess you need provide this kafka.bootstrap.servers ...READ MORE

answered May 24, 2018 in Apache Spark by Shubham
• 13,290 points
548 views
0 votes
1 answer

Kafka Feature

Here are some of the important features of ...READ MORE

answered Jun 7, 2018 in Apache Spark by Data_Nerd
• 2,360 points
77 views
0 votes
1 answer

How to reset the offset of messages consumed from Kafka?

The reset option only prints the result ...READ MORE

answered Jul 9, 2018 in Apache Kafka by Shubham
• 13,290 points
2,174 views
0 votes
1 answer

Azure - What service to use for Arduino data (iot)

That's because the two services do completely ...READ MORE

answered Aug 1, 2018 in IoT (Internet of Things) by anonymous2
• 4,260 points
86 views