Recent questions tagged apache-spark

0 votes
0 answers

Are there any services like Databricks but without compulsion of AWS/Azure/Google Cloud?

Jan 24, 2023 in AWS by Tejashwini
 • 5,380 points • 976 views
0 votes
1 answer

Azure HD Insight - YARN UI is not showing logs on stderr suddenly

Mar 7, 2022 in Azure by Edureka
 • 12,700 points • 1,335 views
0 votes
1 answer

What is apache hadoop?

Dec 13, 2021 in Big Data Hadoop by CoderGirl
 • 500 points • 1,680 views
0 votes
1 answer

Spark Core How to fetch max n rows of an RDD function without using Rdd.max()

Dec 3, 2020 in Apache Spark by Prashant
 • 120 points • 2,736 views
0 votes
1 answer

In AWS, if user wants to run spark, then on top of which one of the following can the user do it?

Nov 26, 2020 in Apache Spark by ritu
 • 960 points • 1,934 views
0 votes
1 answer

error: value update is not a member of scala.collection.immutable.Map[String, Int]

Nov 17, 2020 in Big Data Hadoop by anonymous
 • 8,870 points • 6,250 views
0 votes
1 answer

How to read Avro Partition Data?

Nov 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 2,318 views
+1 vote
1 answer

How to write Spark DataFrame to Avro Data File?

Nov 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 4,028 views
0 votes
1 answer

How to read a dataframe based on an avro schema?

Oct 30, 2020 in Apache Spark by anonymous
 • 120 points • 4,064 views
0 votes
1 answer

How to implement my clustering algorithm in pyspark (without using the ready library for example k-means)?

Oct 14, 2020 in Apache Spark by dani
 • 160 points • 2,510 views
0 votes
1 answer

How to insert data into Cassandra table using Spark DataFrame?

Sep 21, 2020 in Apache Spark by akhtar
 • 38,260 points • 4,426 views
0 votes
1 answer

How to merge two Spark DataFrames?

Sep 17, 2020 in Big Data Hadoop by akhtar
 • 38,260 points • 2,360 views
0 votes
1 answer

How can browse without port usage (example: "http://<IP>" )on the broswer if the php artisan server on "http://<IP>:8088"

Jul 31, 2020 in Web Development by Raghu
 • 120 points • 1,837 views
0 votes
1 answer

File not found exception while processing the spark job in yarn cluster mode with multinode hadoop cluster

Jul 30, 2020 in Apache Spark by Ganendra
 • 140 points • 5,501 views
0 votes
1 answer

Unable to submit the spark job in deployment mode - multinode cluster(using ubuntu machines) with yarn master

Jul 29, 2020 in Apache Spark by Ganendra
 • 140 points • 3,224 views
0 votes
0 answers

Unable to get the Job status and Group ID java- spark standalone program with databricks

Jul 23, 2020 in Apache Spark by kamboj
 • 140 points • 3,190 views
0 votes
1 answer

Can number of Spark task be greater than the executor core?

Jun 17, 2020 in Apache Spark by Rishi
 • 160 points • 2,575 views
0 votes
1 answer

Can the executor core be greater than the total number of spark tasks?

Jun 17, 2020 in Apache Spark by Rishi
 • 160 points • 2,742 views
0 votes
1 answer

after installing hadoop 3.0.1 I can's access spark shell or hive shell.

Jun 16, 2020 in Apache Spark by abdul
 • 120 points • 1,985 views
0 votes
1 answer

Where can I get best spark tutorials for beginners?

May 14, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,135 views
0 votes
1 answer

Py4JJavaError: An error occurred while calling o310.csv. : java.net.ConnectException: Call From master/192.168.56.101 to master:9000

May 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 8,143 views
+1 vote
1 answer

How to convert pyspark Dataframe to pandas Dataframe?

May 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 8,647 views
0 votes
1 answer

How to integrate Machine Learning with Spark?

May 6, 2020 in Machine Learning by akhtar
 • 38,260 points • 1,259 views
0 votes
1 answer

ERROR thriftserver.SparkExecuteStatementOperation: Error executing query, currentState RUNNING, org.apache.spark.sql.catalyst.errors.package$TreeNodeException

Apr 29, 2020 in Apache Spark by akhtar
 • 38,260 points • 2,469 views
0 votes
1 answer

Error: sql.out:Error: org.apache.spark.SparkException: Job aborted due to stage failure: Total size of serialized results of 381610 tasks (4.0 GB) is bigger than spark.driver.maxResultSize (4.0 GB)

Apr 29, 2020 in Apache Spark by akhtar
 • 38,260 points • 8,865 views
0 votes
1 answer

"java.lang.ClassNotFoundException" in Spark on Amazon EMR

Apr 29, 2020 in Apache Spark by akhtar
 • 38,260 points • 4,346 views
0 votes
1 answer

error: Caused by: org.apache.spark.SparkException: Failed to execute user defined function.

Apr 22, 2020 in Apache Spark by akhtar
 • 38,260 points • 5,077 views
+2 votes
2 answers

py4j.protocol.Py4JError: org.apache.spark.api.python.PythonUtils.getEncryptionEnabled does not exist in the JVM

Apr 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 24,514 views
0 votes
1 answer

env: ‘python’: No such file or directory in pyspark.

Apr 7, 2020 in Apache Spark by akhtar
 • 38,260 points • 7,015 views
–1 vote
0 answers

How to parse an S3 XML file to find tags using apache spark

Mar 18, 2020 in Apache Spark by anonymous
 • 110 points • 2,407 views
+2 votes
1 answer

FileStreamSink: Error while looking for metadata directory. java.lang.IllegalArgumentException: java.net.UnknownHostException: hive

Feb 13, 2020 in Big Data Hadoop by akhtar
 • 38,260 points • 12,125 views
0 votes
1 answer

apache.spark.sql.AnalysisException: Text data source does not support int data type.;

Feb 13, 2020 in Big Data Hadoop by akhtar
 • 38,260 points • 6,744 views
+1 vote
1 answer

What is the importance of Kafka bootstrap.servers?

Feb 11, 2020 in Apache Kafka by akhtar
 • 38,260 points • 19,028 views
0 votes
1 answer

org.apache.kafka.common.config.ConfigException: Missing required configuration "bootstrap.servers" which has no default value.

Feb 10, 2020 in Apache Kafka by akhtar
 • 38,260 points • 8,454 views
0 votes
0 answers

What is the difference between point to point and publish-subscribe messaging system?

Feb 6, 2020 in Apache Kafka by akhtar
 • 38,260 points • 1,106 views
+1 vote
1 answer

ERROR Unexpected exception, exiting abnormally (org.apache.zookeeper.server.ZooKeeperServerMain) java.net.BindException: Address already in use

Feb 5, 2020 in Apache Kafka by akhtar
 • 38,260 points • 13,615 views
0 votes
1 answer

Exception in thread "main" java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute URI: ${system:java.io.tmpdir%7D/$%7Bsystem:user.name%7D

Feb 5, 2020 in Big Data Hadoop by akhtar
 • 38,260 points • 9,552 views
0 votes
1 answer

Does spark streaming provides checkpoint?

Feb 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,957 views
0 votes
0 answers

not able to get output in spark streaming??

Feb 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,261 views
0 votes
1 answer

Is Spark Sql provides indexing to improve processing speed?

Feb 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,404 views
0 votes
1 answer

What is the difference between spark streaming and spark structured streaming?

Feb 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 4,453 views
0 votes
1 answer

What are Dstreams?

Feb 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,609 views
0 votes
1 answer

Caused by: ERROR XJ040: Failed to start database 'metastore_db' with class loader org.apache.spark.sql.hive.client.IsolatedClientLoader$$anon$1@2c7bde26

Feb 3, 2020 in Apache Spark by akhtar
 • 38,260 points • 6,393 views
0 votes
1 answer

Cannot create directory /hive/xzxz/_temporary/0. Name node is in safe mode.

Feb 3, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,222 views
+1 vote
1 answer

is not a Parquet file. expected magic number at tail [80, 65, 82, 49] but found [51, 53, 10, 10]

Feb 3, 2020 in Apache Spark by akhtar
 • 38,260 points • 19,802 views
0 votes
1 answer

What is pageRank in graphX??

Jan 31, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,650 views
0 votes
1 answer

env : R : No such file or directory

Jan 31, 2020 in Apache Spark by Hasid
 • 370 points • 2,217 views
0 votes
2 answers

java.lang.StringIndexOutOfBoundsException: String index out of range: 1

Jan 29, 2020 in