Unanswered questions in Apache Spark

0 votes
0 answers

How to import pyspark in Jupyter

I tried to import pyspark in jupyter ...READ MORE

Apr 3, 2023 in Apache Spark by Navyasilpa
edited Mar 5 • 236 views
0 votes
0 answers

How to import pyspark in Jupyter Notebook

When I tried to import Pyspark I am getting ...READ MORE

Apr 3, 2023 in Apache Spark by Navyasilpa
edited Mar 5 • 273 views
0 votes
0 answers

How to read a nested avro file format in spark dataframe

The avro file format contains nested data. ...READ MORE

Nov 16, 2022 in Apache Spark by Devang
edited Mar 4 • 248 views
0 votes
0 answers

How can i implement corss apply function of TSQL in pyspark

How can i implement corss apply function ...READ MORE

May 30, 2022 in Apache Spark by anonymous
edited Mar 4 • 287 views
0 votes
0 answers

Pyspark: Aggregate and filtering code error

Hi guys, I am a beginner at pyspark ...READ MORE

Apr 22, 2022 in Apache Spark by Saadat
edited Mar 4 • 278 views
0 votes
0 answers

Pyspark: Finding top three countries with covid confirmed covid cases

Hi guys, I have a beginner at pyspark ...READ MORE

Apr 22, 2022 in Apache Spark by Saadat
edited Mar 4 • 206 views
0 votes
0 answers

Scala / SparkSQL dataframes filter issue "data type mismatch"

My probleme is i have a code ...READ MORE

Mar 24, 2022 in Apache Spark by Hamza
edited Mar 4 • 156 views
0 votes
0 answers

Access value in arrays of structs spark scala

Hi, I have a dataset with the ...READ MORE

Mar 24, 2022 in Apache Spark by anonymous
edited Mar 4 • 149 views
0 votes
0 answers

What should I pay attention to when installing smart curtains fabrics?

What should I pay attention to when ...READ MORE

Mar 23, 2022 in Apache Spark by qiansifang
edited Mar 4 • 153 views
0 votes
0 answers

The Batman Movie Online Free HD

dfgsdfg READ MORE

Mar 4, 2022 in Apache Spark by anonymous
edited Mar 4 • 134 views
0 votes
0 answers

How can I use substring to extract the four last characters of a column if I do not know the number of characters that are within a field?

I want to only get the last ...READ MORE

Jan 14, 2022 in Apache Spark by anonymous
edited Mar 4 • 159 views
0 votes
0 answers

Execute Spark.sql query within withColumn clause is Spark Scala

I have a dataframe which has one ...READ MORE

Sep 14, 2021 in Apache Spark by Pinksrider
edited Mar 4 • 183 views
0 votes
0 answers

Aws logs are not writing in cloud watch after certain steps

i have an aws job which reads ...READ MORE

Jul 30, 2021 in Apache Spark by Anjali
edited Mar 4 • 163 views
0 votes
0 answers

Create Hive table using Dataframe getting error

Code: srcDF.write.mode(tblmode).saveAsTable(s"${dbName}.${tgtHiveTableName}") error: 21/06/04 22:11:45 ERROR pa.TrxNbrx: org.apache.spark.SparkException: ...READ MORE

Jun 5, 2021 in Apache Spark by Rajesh
edited Mar 4 • 148 views
0 votes
0 answers

OI JANA TESTE LIVE

OI JANA TESTE LIVE READ MORE

Jun 5, 2021 in Apache Spark by Eufrasia
edited Mar 4 • 138 views
0 votes
0 answers

what parameters are required for a "windowed" operation such as reduceByKeyAndWindow?

a) Window length b) sliding interval c) Window Length ...READ MORE

Jun 4, 2021 in Apache Spark by anonymous
edited Mar 4 • 165 views
0 votes
0 answers

It is a schema for a pyspark nested dataframe ..what I want is to change the name of the ambiguous terms configdata to configdata0, configdata1 and so on . So how can it be achieved using rdd?

root |-- fields: struct (nullable = true) | ...READ MORE

Apr 27, 2021 in Apache Spark by sushant
edited Mar 4 • 176 views
0 votes
0 answers

Real time Project challenges in Spark Data pipeline

Can anybody highlights some challenges they have ...READ MORE

Apr 6, 2021 in Apache Spark by anonymous
edited Mar 4 • 162 views
0 votes
0 answers

ForeachPartition called with python functions doesnot output data to HDFS when executed on YARN mode

Hi, I need help with my code. Trying ...READ MORE

Jan 22, 2021 in Apache Spark by anonymous
edited Mar 4 • 140 views
0 votes
0 answers

17)from the given choices, identify the value returned by $"whatever"?

17)from the given choices, identify the value ...READ MORE

Nov 25, 2020 in Apache Spark by ritu
 • 960 points • 1,816 views
0 votes
0 answers

Unable to get the Job status and Group ID java- spark standalone program with databricks

package com.dataguise.test; import java.io.IOException; import java.util.concurrent.CountDownLatch; import java.util.concurrent.TimeUnit; import org.apache.spark.SparkContext; import org.apache.spark.SparkJobInfo; import ...READ MORE

Jul 23, 2020 in Apache Spark by kamboj
 • 140 points
recategorized Jul 28, 2020 by Gitika • 2,863 views
0 votes
0 answers

Do we have any platform where we can submit spark application.

looking for a platform where we can ...READ MORE

May 12, 2020 in Apache Spark by anonymous
 • 120 points • 1,194 views
–1 vote
0 answers

How to parse an S3 XML file to find tags using apache spark

How can one parse an S3 XML ...READ MORE

Mar 18, 2020 in Apache Spark by anonymous
 • 110 points • 2,293 views
0 votes
0 answers

One Hot Encoding in Apache Spark

The following code that I wrote for ...READ MORE

Feb 11, 2020 in Apache Spark by Manish
 • 120 points • 2,872 views
0 votes
0 answers

not able to get output in spark streaming??

Hi everyone, I tried to count individual words ...READ MORE

Feb 4, 2020 in Apache Spark by akhtar
 • 38,260 points • 1,170 views
0 votes
0 answers

Error: Package: R-core-devel-3.6.0-1el7.x86_64 (epel) Requires: pcre2-devel

Hi, I am getting this error when try ...READ MORE

Jan 31, 2020 in Apache Spark by Hasid
 • 370 points • 1,347 views
+1 vote
0 answers

How to create a list of RDDs(or RDD of RDDs, if possible) from a single JavaRDD<List<Integers>> in Java?

Hi, I have the input RDD as a ...READ MORE

Jan 11, 2020 in Apache Spark by itsroops
 • 130 points • 3,354 views
+1 vote
0 answers

how to access hive view using spark2

We do not have access to hive ...READ MORE

Dec 29, 2019 in Apache Spark by anonymous
 • 130 points • 2,241 views
+1 vote
0 answers

How to combine a nested json file, which is being partitioned on the basis of source tags, and has varying internal structure, into a single json file; ( differently sourced Tag and varying structure)

Source tags are different: {  x : [         { ...READ MORE

Oct 11, 2019 in Apache Spark by anonymous
 • 160 points • 1,305 views
+2 votes
0 answers

How can I import zip files and process the excel files ( inside the zip files ) by using pyspark connecting with pymongo ?

How can I import zip files and ...READ MORE

Aug 6, 2019 in Apache Spark by Ahmed
 • 1,026 views
0 votes
0 answers

How to define SparkConf?

Can anyone explain how to define SparkConf? READ MORE

Aug 1, 2019 in Apache Spark by Danish
 • 674 views
–1 vote
0 answers

what is the benefit of repartition(1) and coalesce(1). When we save data we use df.repartition(1).so how many partition it will create

what is the benefit of repartition(1) and ...READ MORE

Jul 26, 2019 in Apache Spark by kumar Ma
 • 5,761 views
0 votes
0 answers

What is immutabiliity in Spark?

Can anyone explain what is immutability in ...READ MORE

Jul 23, 2019 in Apache Spark by Risha
 • 656 views
0 votes
0 answers

Load data to mysql from local

I'm trying to load data to mysql ...READ MORE

Jul 23, 2019 in Apache Spark by Yogi
 • 1,194 views
0 votes
0 answers

How to create RDD as string file?

Can anyone suggest how to create RDD ...READ MORE

Jul 5, 2019 in Apache Spark by anand
 • 1,260 views
0 votes
0 answers

When we create an RDD, does it bring the data and load it into the memory?

Can anyone suggest when we create an ...READ MORE

Jul 3, 2019 in Apache Spark by monalisa
recategorized Jul 4, 2019 by Gitika • 1,465 views
+1 vote
0 answers

_spark_metadata/0 doesn't exist while Compacting batch 9 Structured streaming error

We have Streaming Application implemented using Spark ...READ MORE

May 31, 2019 in Apache Spark by AzimKangda
 • 130 points • 4,593 views
0 votes
0 answers

what are the memory issues in spark ?

Mar 18, 2019 in Apache Spark by satish kumar
 • 180 points • 2,407 views
0 votes
0 answers

Why doesn't my Spark Yarn client runs on all available worker machines?

I am running an application on Spark ...READ MORE

Feb 22, 2019 in Apache Spark by Uzair Ahmad
edited Feb 22, 2019 by Omkar • 8,873 views

Most popular tags

"PMP®","PMI®", "PMI-ACP®" and "PMBOK®" are registered marks of the Project Management Institute, Inc. MongoDB®, Mongo and the leaf logo are the registered trademarks of MongoDB, Inc.