Update Resource URL automatically when replicating CKAN to another AWS environment via CloudFront

0 votes

We are implementing CKAN on AWS with the DataStore extension and interact with it via the python CKAN API. AWS is essentially split into two environments:

PUB is created via CloudFront and uses Read Replica of the PRIV database. It is basically the same as PRIV except that it is read-only.

Challenge

Resource URLs in PUB point to the PRIV environment. For example, running PUB_ckan.resource_show(id='123') api call in the public environment returns the following:

{ ...

'datastore_active': False,

'id': '123',

'name': 'Resource 1',

'package_id': 'abc',

'state': 'active',

'url': 'https://private.company.com/dataset/f688/resource/e3c785/download/file.zip',

'url_type': 'upload'

... }

This is the same for files uploaded through the CKAN API or the DataStore extension (in which case they are labeled 'url_type': 'datastore).

Expectation

All of the package/resource metadata should be the same between environments, with the exception of Resource URL which must reflect the PUB URL so end-users make api calls against the highly available, secure, environment. i.e.:

'url': 'https://public.company.com/dataset/f688/resource/e3c785/download/file.zip'

So far, I have looked into whether the config file contains a setting for using relative URLs and also tried updating the URLs manually via a Python script, both without success. Any help with this would be greatly appreciated.

Aug 21, 2018 in AWS by bug_seeker
• 15,510 points
887 views

1 answer to this question.

0 votes

If you want to keep using the same underlying database then the easiest way is probably a small plugin on the public instance which implements the IResourceController interface and uses the before_show method of that interface to change the resource URL.

Note, however, the following warning from the documentation of before_show:

Be aware that this method is not only called for UI display, but also in other methods like when a resource is deleted because showing a package is used to get access to the resources in a package.

You should therefore definitely test that your URL modifications do not affect other parts of CKAN.

answered Aug 21, 2018 by Priyaj
• 58,020 points

Related Questions In AWS

0 votes
0 answers

Error when creating AWS Budget Actions to automatically stop EC2 and RDS

I use one EC2 instance and one ...READ MORE

Apr 21, 2022 in AWS by Rahul
• 9,680 points
862 views
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
0 votes
1 answer
webinar REGISTER FOR FREE WEBINAR X
REGISTER NOW
webinar_success Thank you for registering Join Edureka Meetup community for 100+ Free Webinars each month JOIN MEETUP GROUP