Training data in S3 in AWS Sagemaker

0 votes

I've uploaded my own Jupyter notebook to Sagemaker, and am trying to create an iterator for my training / validation data which is in S3, as follow:

train = mx.io.ImageRecordIter( path_imgrec = ‘s3://bucket-name/train.rec’ …… )

I receive the following exception:

MXNetError: [04:33:32] src/io/s3_filesys.cc:899: Need to set enviroment variable AWS_SECRET_ACCESS_KEY to use S3

I've checked that the IAM role attached with this notebook instance has S3 access. Any clues on what might be needed to fix this?

Jul 27, 2018 in AWS by bug_seeker
• 15,350 points
1,108 views

1 answer to this question.

0 votes

If your IAM roles are setup correctly, then you need to download the file to the Sagemaker instance first and then work on it. Here's how:

# Import roles

import sagemaker

role = sagemaker.get_execution_role()

# Download file locally

s3 = boto3.resource('s3') s3.Bucket(bucket).download_file('your_training_s3_file.rec', 'training.rec') #Access locally train = mx.io.ImageRecordIter(path_imgrec=‘training.rec’ …… )

answered Jul 27, 2018 by Priyaj
• 56,900 points

Related Questions In AWS

0 votes
1 answer

Trying to upload data stream in Amazon S3

Try adding the ContentType & ACL in ...READ MORE

answered Jun 28, 2018 in AWS by Cloud gunner
• 4,280 points
430 views
0 votes
1 answer

What happens to data if the data center goes down in AWS?

When it comes to the security and ...READ MORE

answered Jul 13, 2018 in AWS by Priyaj
• 56,900 points

edited Jul 13, 2018 by Priyaj 342 views
0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14 in AWS by athenauserz
1,426 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,090 points
4,826 views
+13 votes
2 answers

Git management technique when there are multiple customers and need multiple customization?

Consider this - In 'extended' Git-Flow, (Git-Multi-Flow, ...READ MORE

answered Mar 26, 2018 in DevOps & Agile by DragonLord999
• 8,380 points
205 views
0 votes
1 answer
0 votes
1 answer

AWS S3 pricing for data in/out.

All traffic from mobile phones to S3 ...READ MORE

answered Jul 6, 2018 in AWS by Priyaj
• 56,900 points
56 views
+1 vote
2 answers

AWS WAF - Auto Save Web Application Firewall logs in S3.

You can do this through the console ...READ MORE

answered Jul 13, 2018 in AWS by Priyaj
• 56,900 points
342 views