Training data in S3 in AWS Sagemaker

0 votes

I've uploaded my own Jupyter notebook to Sagemaker, and am trying to create an iterator for my training / validation data which is in S3, as follow:

train = mx.io.ImageRecordIter( path_imgrec = ‘s3://bucket-name/train.rec’ …… )

I receive the following exception:

MXNetError: [04:33:32] src/io/s3_filesys.cc:899: Need to set enviroment variable AWS_SECRET_ACCESS_KEY to use S3

I've checked that the IAM role attached with this notebook instance has S3 access. Any clues on what might be needed to fix this?

Jul 27, 2018 in AWS by bug_seeker
• 15,310 points
730 views

1 answer to this question.

0 votes

If your IAM roles are setup correctly, then you need to download the file to the Sagemaker instance first and then work on it. Here's how:

# Import roles

import sagemaker

role = sagemaker.get_execution_role()

# Download file locally

s3 = boto3.resource('s3') s3.Bucket(bucket).download_file('your_training_s3_file.rec', 'training.rec') #Access locally train = mx.io.ImageRecordIter(path_imgrec=‘training.rec’ …… )

answered Jul 27, 2018 by Priyaj
• 56,160 points

Related Questions In AWS

0 votes
1 answer

Trying to upload data stream in Amazon S3

Try adding the ContentType & ACL in ...READ MORE

answered Jun 28, 2018 in AWS by Cloud gunner
• 4,260 points
257 views
0 votes
1 answer

What happens to data if the data center goes down in AWS?

When it comes to the security and ...READ MORE

answered Jul 13, 2018 in AWS by Priyaj
• 56,160 points

edited Jul 13, 2018 by Priyaj 203 views
0 votes
1 answer

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

This is a known deficiency. The best ...READ MORE

answered Sep 4, 2018 in AWS by Archana
• 4,090 points
752 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,090 points
2,430 views
+13 votes
2 answers
0 votes
1 answer
0 votes
1 answer

AWS S3 pricing for data in/out.

All traffic from mobile phones to S3 ...READ MORE

answered Jul 6, 2018 in AWS by Priyaj
• 56,160 points
31 views
+1 vote
2 answers

AWS WAF - Auto Save Web Application Firewall logs in S3.

You can do this through the console ...READ MORE

answered Jul 13, 2018 in AWS by Priyaj
• 56,160 points
281 views