Trending questions in Apache Spark

+5 votes
11 answers

Concatenate columns in apache spark dataframe

its late but this how you can ...READ MORE

Mar 21 in Apache Spark by anonymous
 • 22,925 views
0 votes
11 answers

How to create new column with function in Spark Dataframe?

val coder: (Int => String) = v ...READ MORE

Apr 4 in Apache Spark by anonymous
edited Apr 5 by Omkar • 14,614 views
0 votes
6 answers

How to replace null values in Spark DataFrame?

Hi i hope this will help for ...READ MORE

Feb 5 in Apache Spark by Srinivasreddy
 • 140 points • 13,900 views
0 votes
1 answer

Filtering a row in Spark DataFrame based on matching values from a list

Use the function as following: var notFollowingList=List(9.8,7,6,3, ...READ MORE

Jun 5, 2018 in Apache Spark by Shubham
 • 13,190 points • 18,703 views
0 votes
1 answer

Spark Null Pointer Exception.

I used Spark 1.5.2 with Hadoop 2.6 ...READ MORE

16 minutes ago in Apache Spark by ravikiran
 • 3,560 points • 4 views
0 votes
1 answer

Spark on Yarn

If you just want to get your ...READ MORE

23 hours ago in Apache Spark by ravikiran
 • 3,560 points • 6 views
0 votes
1 answer

Spark, Scala: Load custom delimited file

You can load a DAT file into ...READ MORE

3 days ago in Apache Spark by Shri
 • 16 views
0 votes
1 answer

Pyspark is taking default path

The HDFS path for MyLab is /user/edureka_id. ...READ MORE

3 days ago in Apache Spark by Khushi
 • 12 views
0 votes
0 answers

Need to load 40 GB data to elasticsearch using spark

I am working in psedo distributed spark ...READ MORE

2 days ago in Apache Spark by Amit
 • 120 points • 8 views
0 votes
0 answers

Spark: java.io.FileNotFoundException

While executing a query I am getting ...READ MORE

3 days ago in Apache Spark by Tilka
 • 13 views
0 votes
1 answer

Spark Processing Internals

Spark uses a master/slave architecture. As you ...READ MORE

4 days ago in Apache Spark by Jimmy
 • 13 views
0 votes
1 answer

Spark-shell not working

First, reboot the system. And after reboot, ...READ MORE

4 days ago in Apache Spark by Mahesh
 • 13 views
0 votes
1 answer

org.apache.spark.sql.AnalysisException: cannot resolve "`id`" given input columns

I have used a header-less csv file ...READ MORE

5 days ago in Apache Spark by Puneet
 • 50 views
0 votes
1 answer

What does the command df.registerTempTable() do?

df.registerTempTable(“airports”) This command is used to register ...READ MORE

5 days ago in Apache Spark by James
 • 19 views
0 votes
1 answer

Difference between cogroup and full outer join in spark

Please go through the below explanation : Full ...READ MORE

5 days ago in Apache Spark by Kiran
 • 22 views
0 votes
1 answer

Can we change the path where the Hive data is stored in HDFS?

Yes, you can but it has to ...READ MORE

5 days ago in Apache Spark by Yogi
 • 15 views
0 votes
1 answer

Spark: How can i create temp views in user defined database instead of default database?

You can try the below code: df.registerTempTable(“airports”) sqlContext.sql(" create ...READ MORE

5 days ago in Apache Spark by Ishan
 • 12 views
0 votes
1 answer

Spark memory processing on a not temporary table

Temporary table is more like an index ...READ MORE

5 days ago in Apache Spark by Suri
 • 10 views
0 votes
1 answer

How do I turn off INFO Logging in Spark?

Execute this command in the spark directory: cp ...READ MORE

Jul 12 in Apache Spark by ravikiran
 • 3,560 points • 21 views
0 votes
1 answer

Spark error: Caused by: java.lang.RuntimeException: The root scratch dir: /tmp/hive on HDFS should be writable.

Give  read-write permissions to  C:\tmp\hive folder Cd to winutils bin folder ...READ MORE

Jul 11 in Apache Spark by Rajiv
 • 25 views
0 votes
1 answer

org.apache.spark.sql.AnalysisException: cannot resolve given input columns

The string Productivity has to be enclosed between single ...READ MORE

Jul 10 in Apache Spark by Tina
 • 63 views
0 votes
1 answer

Working of map function on data

The map function creates an array of ...READ MORE

Jul 11 in Apache Spark by Krish
 • 16 views
0 votes
1 answer

Why do we use sc.parallelize?

Spark revolves around the concept of a ...READ MORE

Jul 11 in Apache Spark by Suman
 • 17 views
0 votes
1 answer

Is fetching data from apache flume webcrawling?

Web crawling is a program or automated ...READ MORE

Jul 11 in Apache Spark by Esha
 • 16 views
0 votes
1 answer

Error Loading data to mysql in Spark

You have to use sqoop to export data ...READ MORE

Jul 11 in Apache Spark by Jishan
 • 14 views
0 votes
1 answer

Spark Streaming Pyspark code not working

The address you are using in the ...READ MORE

Jul 11 in Apache Spark by Shir
 • 14 views
0 votes
1 answer

Error : split value is not a member of org.apache.spark.sql.Row

spark.read.csv is used when loading into a ...READ MORE

Jul 10 in Apache Spark by Rishi
 • 29 views
0 votes
1 answer

How to add package com.databricks.spark.avro in spark?

Start spark shell using below line of ...READ MORE

Jul 10 in Apache Spark by Jishnu
 • 19 views
0 votes
1 answer

Query regarding Appending " to a string in Scala

You can perform this task in two ...READ MORE

Jul 10 in Apache Spark by Esha
 • 14 views
0 votes
1 answer

Query regarding Operator Overloading in Scala

All prefix operators' symbols are predefined: +, -, ...READ MORE

Jul 10 in Apache Spark by Karan
 • 10 views
0 votes
1 answer

Scala join comma delimited file as tables

Dataframe creation commands:​ Now we will register them ...READ MORE

Jul 9 in Apache Spark by Suraj
 • 15 views
0 votes
1 answer

How to use yield keyword in scala and why it is used instead of println?

Hi, The yield keyword is used because the ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 27 views
0 votes
1 answer

How to print string text in scala?

Hi, You can see this example to see ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 24 views
0 votes
1 answer

How can you use "for" statement in scala to print list from collection?

Hi, You can use for loop in scala using ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 22 views
0 votes
1 answer

How can we iterate any function using "foreach" function in scala?

Hi, Yes, "foreach" function you use because it will ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 21 views
0 votes
1 answer

Which syntax to use to take the sum of list of collection in scala?

Hi, You can see this example to get ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 21 views
0 votes
1 answer

How to assign block expression in scala?

Hi, You can follow this example to know ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 18 views
0 votes
1 answer

How to implement two level loop in scala?

Hi, You can use two level loops using the ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 15 views
0 votes
1 answer

How to print loop with condition in scala?

Hi, Yes, in scala there is a guard condition where ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 14 views
0 votes
1 answer

How can we optimize and minimize the memory when work with scala use case?

Hi, There is a term in Scala that is ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 15 views
0 votes
1 answer

Spark Installation problem

After downloading Spark, you need to set ...READ MORE

Jul 5 in Apache Spark by Rishi
 • 17 views
0 votes
1 answer

How to create dataframe for the comma delimited file?

 Refer to the below command used: val df ...READ MORE

Jul 5 in Apache Spark by karan
 • 16 views
0 votes
1 answer

load/save in spark

The reason why you are able to ...READ MORE

Jul 5 in Apache Spark by Firoz
 • 14 views
0 votes
1 answer

Is it mandatory to start Hadoop to run spark application?

Hi, No, not mandatory, but there is no ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 18 views
0 votes
1 answer

Which File System is supported by Apache Spark?

Hi, Apache Spark is an advanced data processing ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 18 views
0 votes
1 answer

How to run spark in Standalone client mode?

Hi, These are the steps to run spark in ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 15 views
0 votes
1 answer

Do real-time data processing is possible with Spark SQL?

Hey, Real-time data processing is not possible directly ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 14 views
0 votes
1 answer

How to create scala project in intellij?

You have to install Intellij with scala plugin. ...READ MORE

Jul 5 in Apache Spark by Jimmy
 • 13 views
0 votes
1 answer

How will you explain yield keyword in Scala?

Hi, Yield keyword can be used either before ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 13 views
0 votes
1 answer

When we create an RDD, does it bring the data and load it into the memory?

Hi, No. An RDD is made up of ...READ MORE

Jul 5 in Apache Spark by Gitika
 • 19,720 points • 12 views
Page:

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.