Update Resource URL automatically when replicating CKAN to another AWS environment via CloudFront

Question

We are implementing CKAN on AWS with the DataStore extension and interact with it via the python CKAN API. AWS is essentially split into two environments:

• PRIV: for internal and QA/Staging purposes, hosted in https://private.company.com

• PUB: for external end-users, hosted in https://private.company.com

PUB is created via CloudFront and uses Read Replica of the PRIV database. It is basically the same as PRIV except that it is read-only.

Challenge

Resource URLs in PUB point to the PRIV environment. For example, running PUB_ckan.resource_show(id='123') api call in the public environment returns the following:

{ ...

'datastore_active': False,

'id': '123',

'name': 'Resource 1',

'package_id': 'abc',

'state': 'active',

'url': 'https://private.company.com/dataset/f688/resource/e3c785/download/file.zip',

'url_type': 'upload'

... }

This is the same for files uploaded through the CKAN API or the DataStore extension (in which case they are labeled 'url_type': 'datastore).

Expectation

All of the package/resource metadata should be the same between environments, with the exception of Resource URL which must reflect the PUB URL so end-users make api calls against the highly available, secure, environment. i.e.:

'url': 'https://public.company.com/dataset/f688/resource/e3c785/download/file.zip'

So far, I have looked into whether the config file contains a setting for using relative URLs and also tried updating the URLs manually via a Python script, both without success. Any help with this would be greatly appreciated.

Answer 1

If you want to keep using the same underlying database then the easiest way is probably a small plugin on the public instance which implements the IResourceController interface and uses the before_show method of that interface to change the resource URL.

Note, however, the following warning from the documentation of before_show:

Be aware that this method is not only called for UI display, but also in other methods like when a resource is deleted because showing a package is used to get access to the resources in a package.

You should therefore definitely test that your URL modifications do not affect other parts of CKAN.