How to create EMR cluster using Python boto3

Question

Can someone help me with the python code to create a EMR Cluster? Any help is appreciated.

Answer 1

The python boto3 code for creating a EMR cluster is as follows:-

import boto3

connection = boto3.client('emr',region_name='us-east-1',aws_access_key_id='Access Key',aws_secret_access_key='Secret Key',)

cluster_id = connection.run_job_flow(Name='test_emr_job_boto3',LogUri='s3://priyaj',ReleaseLabel='emr-5.18.0',
    Applications=[
        {
            'Name': 'Spark'
        },
    ],
    Instances={
        'InstanceGroups': [
            {
                'Name': "Master",
                'Market': 'ON_DEMAND',
                'InstanceRole': 'MASTER',
                'InstanceType': 'm1.xlarge',
                'InstanceCount': 1,
            },
            {
                'Name': "Slave",
                'Market': 'ON_DEMAND',
                'InstanceRole': 'CORE',
                'InstanceType': 'm1.xlarge',
                'InstanceCount': 1,
            }
        ],
        'Ec2KeyName': 'Your key name',
        'KeepJobFlowAliveWhenNoSteps': True,
        'TerminationProtected': False,
        'Ec2SubnetId': 'subnet-id',
    },
    Steps=[
        {
            'Name': 'file-copy-step',   
                    'ActionOnFailure': 'CONTINUE',
                    'HadoopJarStep': {
                        'Jar': 's3://Snapshot-jar-with-dependencies.jar',
                        'Args': ['test.xml', 'emr-test', 'kula-emr-test-2']
                    }
        }
    ],
    VisibleToAllUsers=True,
    JobFlowRole='EMR_EC2_DefaultRole',
    ServiceRole='EMR_DefaultRole',
    Tags=[
        {
            'Key': 'tag_name_1',
            'Value': 'tab_value_1',
        },
        {
            'Key': 'tag_name_2',
            'Value': 'tag_value_2',
        },
    ],
)

This way you can create a EMR cluster with 1 master and 1 slave node.

Comments

Hi,

Is there a way to set up a secondary master node and switch over in case of failure? Please can you suggest a alternative. Kindly help

---

I will try and let you know whether it is possible or not.

---

Any way to integrate the lambda function and how can we add an extra step?

---

You can create one lambda function and import your code and try to run your code.