questions/apache-spark
You can do this by turning off ...READ MORE
Try df.where($"cola".isNotNull && $"cola" =!= "" && !$"colb".isin(2,3)) your ...READ MORE
I found the following solution to be ...READ MORE
In spark 2.0.+ it should look something ...READ MORE
What is the major use case for ...READ MORE
Try including the package while starting the ...READ MORE
Hey, you can use "contains" filter to extract ...READ MORE
Hello, Your problem is here: val df_merge_final = df_merge .withColumn("version_key", ...READ MORE
Yes we can add columns to the ...READ MORE
Spark-submit jobs are also run from client/edge ...READ MORE
import sqlContext.implicits._ import org.apache.spark.sql.Row import org.apache.spark.sql.types.{StructType, StructField, LongType} val df ...READ MORE
Hey, You can try this: from pyspark import SparkContext SparkContext.stop(sc) sc ...READ MORE
Hi, You can use this command to start ...READ MORE
Hey, Jobs- to view all the spark jobs Stages- ...READ MORE
Hi, You can do it using map partition ...READ MORE
How can I import zip files and ...READ MORE
Hi, foreach() operation is an action. It does not ...READ MORE
Hey, sortByKey() is a transformation. It returns an RDD sorted ...READ MORE
Hey, You can launch spark application in four ...READ MORE
Hi, To launch spark application in cluster mode, ...READ MORE
Hi, If you have a file with id ...READ MORE
Hi, Paired RDD is a distributed collection of ...READ MORE
Well, it depends on the block of ...READ MORE
Suppose you have two dataset results( id, ...READ MORE
Try this code, it worked for me: val ...READ MORE
Hi, Hive contains significant support for Apache Spark, ...READ MORE
Hey, A sparse vector is used for storing ...READ MORE
Can anyone explain how to define SparkConf? READ MORE
You need to declare the variable which ...READ MORE
Hey @Esha, you can use this code. ...READ MORE
The problem is probably with the command. ...READ MORE
Converting text file to Orc: Using Spark, the ...READ MORE
Hi, This reverses the order of elements in ...READ MORE
Hey, The method List.fill() creates a list and ...READ MORE
Hey, Scala uses the method copy() to carry ...READ MORE
Hey, You can use below code to access variables ...READ MORE
Hey, Java’s “If. Else”: In Java, “If. Else” is a statement, ...READ MORE
Hey, You can follow this below solution for ...READ MORE
Hey, It already has SparkContent.union and it does know how to ...READ MORE
Hey, Use SparkContext.union(...) instead to union many RDDs at once You ...READ MORE
Hi, Traits are basically Scala's workaround for the ...READ MORE
Refer to the below command: val input_df = ...READ MORE
Hi, Scala provides a helper class, called App, that ...READ MORE
As is widely used, and has different ...READ MORE
Hey, Scala introduces a new object keyword, which is used ...READ MORE
Since the file is in HDFS so ...READ MORE
1) First we loaded the data to ...READ MORE
Hey, To format a string, use the .format ...READ MORE
Hey, Scala allows the definition of a higher-order function. These ...READ MORE
OR
At least 1 upper-case and 1 lower-case letter
Minimum 8 characters and Maximum 50 characters
Already have an account? Sign in.