How to create Athena tables for dynamic S3 paths using AWS Crawler

0 votes

Below are given my S3 paths under which multiple folders are present. Each folder contains a CSV file each with a different schema.

The values within the curly braces {} will be dynamic.

s3://test_bucket/{val1}/data/{val2}/input/latest/

s3://test_bucket/{val1}/data/{val2}/input/archived/timestamp={val3}/

I want to create the Athena tables using AWS Glue Crawler. We can have a separate database for input_data both for current and archive.

The tables formed should be such that it's partitioned over val1 and val2 both for the current and archive. And, an additional partition should be present in the table, that is, val3, in the case of the archived.

Feb 16 in AWS by Rahul
• 2,080 points
25 views

1 answer to this question.

0 votes
The simplest and most efficient way to use partition projection. It speeds up the query processing of highly partitioned tables and automate partition management. Partition values and locations are calculated from configuration rather than the repository like AWS Glue Data Catalogue. In memory operations are faster than remote operations
answered Feb 16 by anonymous

Related Questions In AWS

+1 vote
3 answers

How to get ARN for s3 Bucket using aws cli .

An ARN is a non-opaque, constructible identifier, ...READ MORE

answered Aug 16, 2018 in AWS by Priyaj
• 58,060 points
3,736 views
0 votes
1 answer

How to create AWS S3 api logs using Cloud Trail?

CloudTrail basically captures the part of API calls ...READ MORE

answered Dec 6, 2018 in AWS by Nitesh
200 views
0 votes
1 answer

How to create a S3 bucket using AWS CLI?

You can use the following command:- C:\Users\priyj_kumar>aws s3api ...READ MORE

answered Feb 15, 2019 in AWS by Priyaj
• 58,060 points
4,613 views
0 votes
1 answer
0 votes
2 answers

How to skip headers when reading a CSV file in S3 and creating a table in AWS Athena?

Thanks for the answer. This should be clear ...READ MORE

answered Aug 14, 2019 in AWS by athenauserz
8,022 views
0 votes
2 answers

How to display just the name of files using aws s3 ls command?

aws s3 ls s3://<your_bucket_name>/ | awk '{print ...READ MORE

answered Mar 17, 2019 in AWS by anonymous
14,857 views
0 votes
1 answer

How to download the latest file in a S3 bucket using AWS CLI?

You can use the below command $ aws ...READ MORE

answered Sep 6, 2018 in AWS by Archana
• 4,150 points
14,421 views
0 votes
1 answer

How to create new stack using create-change-set in AWS Cloudformation

The type of change set operation. To ...READ MORE

answered Sep 18, 2018 in AWS by Priyaj
• 58,060 points
2,805 views
0 votes
1 answer

How to use AWS S3 REST API ListObjects(v2) with multiple parent/child bucket directories?

Here you have to use the ListObjectsV2 ...READ MORE

answered Feb 15 in AWS by anonymous
34 views
0 votes
1 answer

Add a new item to a Dynamodb using a AWS lambda function each time a function is executed with Cloudwatch

The problem here is that the PutItem ...READ MORE

answered Feb 14 in AWS by Rahul
• 2,080 points
43 views
webinar REGISTER FOR FREE WEBINAR X
Send OTP
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP