Date formats how to cast string to date

Question

The following date formats not working for me

>>> df = spark.createDataFrame([("1997/02/28",)], ["t"])
>>> df.select(F.date_format(df.t,"yyyy/MM/dd")).show()
+--------------------------+
|date_format(t, yyyy/MM/dd)|
+--------------------------+
| null|
+--------------------------+

>>> df = spark.createDataFrame([["28-02-1997"],["01-02-2000"],["01-02-2022"]], ["t"])
>>> df.select(df.t.cast("date").alias("n_d")).orderBy("n_d").show()
+----+
| n_d|
+----+
|null|
|null|
|null|
+----+

Answer 1

Try this, it should work:

> from pyspark.sql.functions import unix_timestamp

> df = spark.createDataFrame([("11/25/1991",), ("11/24/1991",), ("11/30/1991",)], ['date_str'])

> df2 = df.select('date_str', from_unixtime(unix_timestamp('date_str', 'MM/dd/yyy')).alias('date'))

DataFrame[date_str: string, date: timestamp]

> df2.show()


+----------+--------------------+

| date_str| date|

+----------+--------------------+

|11/25/1991|1991-11-25 00:00:...|

|11/24/1991|1991-11-24 00:00:...|

|11/30/1991|1991-11-30 00:00:...|

+----------+--------------------+