As far as I know only spark Actions can trigger a spark job, not any transformation as Spark follows lazy evaluation.
So when I execute the sortBy command, the spark web UI shows it as a spark job trigger.
Any idea why it's happening?
Firstly, it's the In-memory computation, if the file ...READ MORE
// Collect data from input avro file ...READ MORE
Yes, you can go ahead and write ...READ MORE
By default a partition is created for ...READ MORE
For accessing Hadoop commands & HDFS, you ...READ MORE
No, you can run spark without hadoop. ...READ MORE
CDH is basically a packaged deal, where ...READ MORE
The reason you are not able to ...READ MORE
You can use the following command. This ...READ MORE
Hadoop 3 is not widely used in ...READ MORE