Import zip files and process the excel files inside the zip files by using pyspark connecting with pymongo

Question

How can I import zip files and process the excel files ( inside the zip files ) by using pyspark connecting with pymongo?

I was install spark and mongodb and python to process the files (excel, csv or json)

I used this code to connect pyspark with mmongo :

from pyspark.sql import SparkSession

my_spark = SparkSession \
    .builder \
    .appName("myApp") \
    .config("spark.mongodb.input.uri", "mongodb://127.0.0.1/test.coll") \
    .config("spark.mongodb.output.uri", "mongodb://127.0.0.1/test.coll") \
    .getOrCreate()

but then I was try to import zip files ( I don't need to open every files to process it )

To know more about Pyspark, it's recommended that you join Pyspark Course today.

Answer 1

I found this sample code:

import zipfile
import io

def zip_extract(x):
    in_memory_data = io.BytesIO(x[1])
    file_obj = zipfile.ZipFile(in_memory_data, "r")
    files = [i for i in file_obj.namelist()]
    return dict(zip(files, [file_obj.open(file).read() for file in files]))

zips = sc.binaryFiles("dbfs:/mnt/vedant-demo/ONG/data/las_raw/D-Dfiles.zip")
files_data = zips.map(zip_extract)

Check if this works. Source: https://gist.github.com/vedantja/bd74d0ba7c350dd348af1f92eadd0e76

Hope this helps!

To know more about Pyspark, it's recommended that you join Pyspark course online.

Thanks.