Execute Spark sql query within withColumn clause is Spark Scala

0 votes

I have a dataframe which has one of the column called "Query" having the select statement present. Want to execute this query and create a new column having actual results from the query within the same dataframe.

+--------------+-----------+-----+--------------------------------+
|DIFFCOLUMNNAME|DATATYPE   |ISSUE|QUERY                           |
+--------------+-----------+-----+--------------------------------+
|Firstname     |StringType |YES  |Select Firstname from TempView  limit 1|
|LastName      |StringType |NO   |Select LastName from TempView  limit 1  |
|Designation   |StringType |YES  |Select Designation from TempView limit 1|
|Salary        |IntegerType|YES  |Select Salary from TempView    limit 1 |
+--------------+-----------+-----+--------------------------------+

When I try below

DF.withColumn("QueryResult", SparkConfiguration.sparkSession.sql(col("QUERY"))) 
Getting error as Type mismatch, Required String found column. 
I am thinking I need to use UDF here. But not sure how to write and use. Please helpspark

Sep 14, 2021 in Apache Spark by Pinksrider

edited Mar 4, 2025 384 views

No answer to this question. Be the first to respond.

Your answer

Your name to display (optional):
Privacy: Your email address will only be used for sending these notifications.
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP