Recent questions tagged apache-spark

0 votes

1 answer

Can number of Spark task be greater than the executor core?

Jun 16 in Apache Spark by Rishi
• 160 points • 45 views

0 votes

1 answer

Can the executor core be greater than the total number of spark tasks?

Jun 16 in Apache Spark by Rishi
• 160 points • 43 views

0 votes

1 answer

after installing hadoop 3.0.1 I can's access spark shell or hive shell.

Jun 16 in Apache Spark by abdul
• 120 points • 61 views

0 votes

1 answer

Where can I get best spark tutorials for beginners?

May 14 in Apache Spark by akhtar
• 14,950 points • 105 views

0 votes

1 answer

Py4JJavaError: An error occurred while calling o310.csv. : java.net.ConnectException: Call From master/192.168.56.101 to master:9000

May 7 in Apache Spark by akhtar
• 14,950 points • 109 views

+1 vote

1 answer

How to convert pyspark Dataframe to pandas Dataframe?

May 7 in Apache Spark by akhtar
• 14,950 points • 783 views

0 votes

1 answer

How to integrate Machine Learning with Spark?

May 6 in Machine Learning by akhtar
• 14,950 points • 43 views

0 votes

1 answer

ERROR thriftserver.SparkExecuteStatementOperation: Error executing query, currentState RUNNING, org.apache.spark.sql.catalyst.errors.package$TreeNodeException

Apr 29 in Apache Spark by akhtar
• 14,950 points • 65 views

0 votes

1 answer

Error: sql.out:Error: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 381610 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)

Apr 29 in Apache Spark by akhtar
• 14,950 points • 267 views

0 votes

1 answer

"java.lang.ClassNotFoundException" in Spark on Amazon EMR

Apr 29 in Apache Spark by akhtar
• 14,950 points • 151 views

0 votes

1 answer

error: Caused by: org.apache.spark.SparkException: Failed to execute user defined function.

Apr 22 in Apache Spark by akhtar
• 14,950 points • 172 views

+1 vote

2 answers

py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getEncryptionEnabled does not exist in the JVM

Apr 7 in Apache Spark by akhtar
• 14,950 points • 1,986 views

0 votes

1 answer

env: ‘python’: No such file or directory in pyspark.

Apr 7 in Apache Spark by akhtar
• 14,950 points • 398 views

0 votes

0 answers

How to parse an S3 XML file to find tags using apache spark

Mar 18 in Apache Spark by anonymous
• 120 points • 114 views

+2 votes

1 answer

FileStreamSink: Error while looking for metadata directory. java.lang.IllegalArgumentException: java.net.UnknownHostException: hive

Feb 13 in Big Data Hadoop by akhtar
• 14,950 points • 1,401 views

0 votes

1 answer

apache.spark.sql.AnalysisException: Text data source does not support int data type.;

Feb 13 in Big Data Hadoop by akhtar
• 14,950 points • 429 views

+1 vote

1 answer

What is the importance of Kafka bootstrap.servers?

Feb 11 in Apache Kafka by akhtar
• 14,950 points • 2,674 views

0 votes

1 answer

org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.

Feb 10 in Apache Kafka by akhtar
• 14,950 points • 638 views

0 votes

0 answers

What is the difference between point to point and publish-subscribe messaging system?

Feb 6 in Apache Kafka by akhtar
• 14,950 points • 36 views

+1 vote

1 answer

ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain) java.net.BindException: Address already in use

Feb 5 in Apache Kafka by akhtar
• 14,950 points • 1,280 views

0 votes

1 answer

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

Feb 5 in Big Data Hadoop by akhtar
• 14,950 points • 621 views

0 votes

1 answer

Does spark streaming provides checkpoint?

Feb 4 in Apache Spark by akhtar
• 14,950 points • 116 views

0 votes

0 answers

not able to get output in spark streaming??

Feb 4 in Apache Spark by akhtar
• 14,950 points • 82 views

0 votes

1 answer

Is Spark Sql provides indexing to improve processing speed?

Feb 4 in Apache Spark by akhtar
• 14,950 points • 92 views

0 votes

1 answer

What is the difference between spark streaming and spark structured streaming?

Feb 4 in Apache Spark by akhtar
• 14,950 points • 372 views

0 votes

1 answer

What are Dstreams?

Feb 4 in Apache Spark by akhtar
• 14,950 points • 43 views

0 votes

1 answer

Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@2c7bde26

Feb 3 in Apache Spark by akhtar
• 14,950 points • 505 views

0 votes

1 answer

Cannot create directory /hive/xzxz/_temporary/0. Name node is in safe mode.

Feb 3 in Apache Spark by akhtar
• 14,950 points • 68 views

+1 vote

1 answer

is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [51, 53, 10, 10]

Feb 3 in Apache Spark by akhtar
• 14,950 points • 1,881 views

0 votes

0 answers

What is pageRank in graphX??

Jan 31 in Apache Spark by akhtar
• 14,950 points • 116 views

0 votes

0 answers

env : R : No such file or directory

Jan 31 in Apache Spark by Hasid
• 370 points • 141 views

0 votes

2 answers

java.lang.StringIndexOutOfBoundsException: String index out of range: 1

Jan 29 in Apache Spark by akhtar
• 14,950 points • 1,088 views

0 votes

0 answers

Not enough space to cache rdd_80_1 in memory!

Jan 29 in Apache Spark by akhtar
• 14,950 points • 171 views

0 votes

1 answer

Caused by: java.lang.NumberFormatException: Empty String

Jan 29 in Apache Spark by akhtar
• 14,950 points • 236 views

0 votes

1 answer

Difference between map() and mapPartitions() function in Spark??

Jan 23 in Apache Spark by akhtar
• 14,950 points • 1,071 views

+1 vote

0 answers

How to create a list of RDDs(or RDD of RDDs, if possible) from a single JavaRDD<List<Integers>> in Java?

Jan 11 in Apache Spark by itsroops
• 130 points • 563 views

+1 vote

1 answer

How to assign a column in Spark Dataframe (PySpark) as a Primary Key?

Jan 8 in Apache Spark by anonymous
• 150 points • 968 views

+2 votes

1 answer

Spark code takes too much time to run on cluster

Jan 3 in Apache Spark by asif
• 140 points • 125 views

+1 vote

1 answer

In spark streaming with kafka, i am getting error “Exception in thread ”main“ java.lang.NoSuchMethodError: org.apache.spark.internal.Logging.$init$”

Nov 7, 2019 in Apache Spark by c.kothamasu
• 160 points • 684 views

+1 vote

2 answers

sparkstream.textfilstreaming(localpathdirectory). I am getting empty results

Nov 7, 2019 in Apache Spark by c.kothamasu
• 160 points • 112 views

+1 vote

1 answer

Is there any efficient way of dealing null values during concat functionality of pyspark.sql version 2.3.4?

Nov 5, 2019 in Apache Spark by aizhar
• 130 points • 4,335 views

+1 vote

1 answer

Cannot resolve Error In Spark when filter records with two where condition

Sep 30, 2019 in Apache Spark by anonymous
• 130 points • 528 views

0 votes

1 answer

Primary keys in Apache Spark

Sep 11, 2019 in Apache Spark by nitinrawat895
• 10,920 points • 163 views

0 votes

1 answer

How do I connect to a HIVE Meta store through a program in SparkSQL?

Sep 5, 2019 in Apache Spark by nitinrawat895
• 10,920 points • 1,304 views

+1 vote

1 answer

What are all the Data quality checks we do in our real time Big Data projects.

Sep 4, 2019 in Big Data Hadoop by Madhan
• 130 points • 281 views

+1 vote

1 answer

How to convert JSON file to AVRO file and vise versa

Aug 23, 2019 in Apache Spark by Prashant
• 656 views

+1 vote

1 answer

How to extract record from one RDD using another RDD

Aug 22, 2019 in Apache Spark by anonymous
• 407 views

+2 votes

1 answer

Type mismatch error in scala

Aug 16, 2019 in Apache Spark by anonymous
• 1,370 views

+1 vote

2 answers

Spark: Can we add column to dataframe?

Aug 9, 2019 in Apache Spark by Chirag
• 993 views

0 votes

1 answer

Monitoring Spark application

Aug 9, 2019 in Apache Spark by Rishi
• 129 views

Recent questions tagged apache-spark

Most popular tags

Subscribe to our Newsletter, and get personalized recommendations.

CATEGORIES

TRENDING BLOG ARTICLES