How to execute python script in hadoop file system hdfs

Question

By default, hadoop allows us to run java codes. But now i want to run this python script:

import os.path

def transform():
    inputfolder = "input"
    for filename in os.listdir(inputfolder):
        path = inputfolder + "\\" + filename
        os.remove(path)
def main():
    transform()
if __name__ == "__main__":  main()

How to run .py file instead of .jar file?

Answer 1

If you are simply looking to distribute your python script across the cluster then you want to use Hadoop Streaming.

The basic syntax of the command looks like (from https://hadoop.apache.org/docs/r1.2.1/streaming.html):

$HADOOP_HOME/bin/hadoop  jar $HADOOP_HOME/hadoop-streaming.jar \
-input myInputDirs \
-output myOutputDir \
-mapper myPythonScript.py \
-file myPythonScript.py

This basically creates a map-reduce job for your python script

Hope this helps :-)