Spark Ecosystem is still in the stage of work-in-progress with Spark components, which are not even in their beta releases. It is still in their alpha release stage, and are being tested by their respective developers.
Components of Spark Ecosystem
The components of Spark ecosystem are getting developed and several contributions are being made every now and then. Primarily, Spark Ecosystem comprises the following components:
- Shark (SQL)
- Spark Streaming (Streaming)
- MLLib (Machine Learning)
- GraphX (Graph Computation)
- SparkR (R on Spark)
- BlindDB (Approximate SQL)
These components are built on top of Spark Core Engine. Spark Core Engine allows writing raw Spark programs and Scala programs and launch them; it also allows writing Java programs before launching them. All these are being executed by Spark Core Engine. To top it all, there are various projects that have come up very fast and efficient.
Shark is one of the Spark Ecosystem components. It is used to perform structured data analysis, especially if the data is too voluminous. Shark also allows running unmodified Hive queries on existing Hadoop deployment.
BlindDB or Blind Database is also known as an Approximate SQL database. If there is a huge amount of data barraging and you are not really interested in exactitude, or in exact results, but just want to have a rough or an approximate picture, BlindDB gets you the same. Firing a query, doing some sort of sampling, and giving out some output is called Approximate SQL. Isn’t it a new and interesting concept? Many a time, when you do not require accurate results, sampling would certainly do.
Spark Streaming is one of those unique features, which have empowered Spark to potentially take the role of Apache Storm. Spark Streaming mainly enables you to create analytical and interactive applications for live streaming data. You can do the streaming of the data and then, Spark can run its operations from the streamed data itself.
MLLib is a machine learning library like Mahout. It is built on top of Spark, and has the provision to support many machine learning algorithms. But the point difference with Mahout is that it runs almost 100 times faster than MapReduce. It is not yet as enriched as Mahout, but it is coming up pretty well, even though it is still in the initial stage of growth.
For graphs and graphical computations, Spark has its own Graph Computation Engine, called GraphX. It is similar to other widely used graph processing tools or databases, like Neo4j, Girafe, and many other distributed graph databases.
There are many people from data science track, who must be aware that for statistical analysis, R is among the best. There is already an integration of R with Hadoop. Now, SparkR is a package for R language to enable R users to leverage the power of Spark from R shell.
Got a question for us? Mention them in the comments section and we will get back to you.