Training data in S3 in AWS Sagemaker

0 votes

I've uploaded my own Jupyter notebook to Sagemaker, and am trying to create an iterator for my training / validation data which is in S3, as follow:

train = mx.io.ImageRecordIter( path_imgrec = ‘s3://bucket-name/train.rec’ …… )

I receive the following exception:

MXNetError: [04:33:32] src/io/s3_filesys.cc:899: Need to set enviroment variable AWS_SECRET_ACCESS_KEY to use S3

I've checked that the IAM role attached with this notebook instance has S3 access. Any clues on what might be needed to fix this?

Jul 27, 2018 in AWS by bug_seeker
• 15,520 points
3,775 views

1 answer to this question.

0 votes

If your IAM roles are setup correctly, then you need to download the file to the AWS Sagemaker instance first and then work on it. Here's how:

# Import roles

import sagemaker

role = sagemaker.get_execution_role()

# Download file locally

s3 = boto3.resource('s3') s3.Bucket(bucket).download_file('your_training_s3_file.rec', 'training.rec') #Access locally train = mx.io.ImageRecordIter(path_imgrec=‘training.rec’ …… )

answered Jul 27, 2018 by Priyaj
• 58,100 points

Related Questions In AWS

0 votes
1 answer

Trying to upload data stream in Amazon S3

Try adding the ContentType & ACL in ...READ MORE

answered Jun 28, 2018 in AWS by Cloud gunner
• 4,670 points
2,541 views
0 votes
1 answer

What happens to data if the data center goes down in AWS?

When it comes to the security and ...READ MORE

answered Jul 13, 2018 in AWS by Priyaj
• 58,100 points

edited Jul 13, 2018 by Priyaj 2,430 views
0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14, 2019 in AWS by athenauserz
11,792 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,170 points
19,811 views
+15 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 27, 2018 in DevOps & Agile by DragonLord999
• 8,450 points
3,990 views
+2 votes
1 answer
0 votes
1 answer

AWS S3 pricing for data in/out.

All traffic from mobile phones to S3 ...READ MORE

answered Jul 6, 2018 in AWS by Priyaj
• 58,100 points
761 views
+1 vote
2 answers

AWS WAF - Auto Save Web Application Firewall logs in S3.

You can do this through the console ...READ MORE

answered Jul 13, 2018 in AWS by Priyaj
• 58,100 points
1,727 views
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP