GCP VM Instance is not able to access secrets from Secret Manager despite of appropriate Roles

Question

In GCP's Secret Manager service, I just made a few secrets. I then generated a service account and a JSON key to authenticate that service account from my local PC in order to access those secrets locally. To allow the service account to access the values of secrets from Secret Manager, I additionally awarded it the role Secret Manager Secret Accessor. Currently, everything is operating flawlessly on my local machine.

The GCP Compute Instance is where I want to deploy this code, furthermore. As a result, I built one and sent the source code to it. I also gave the compute instance's default service account the same privilege, Secret Manager Secret Accessor. Now, when I execute this code on the instance, I receive the following error message: "Permission is refused."

The above exception was the direct cause of the following exception:
ibdax            | 
ibdax            | Traceback (most recent call last):
ibdax            |   File "manage.py", line 22, in <module>
ibdax            |     main()
ibdax            |   File "manage.py", line 18, in main
ibdax            |     execute_from_command_line(sys.argv)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 419, in execute_from_command_line
ibdax            |     utility.execute()
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/core/management/__init__.py", line 363, in execute
ibdax            |     settings.INSTALLED_APPS
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 82, in __getattr__
ibdax            |     self._setup(name)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 69, in _setup
ibdax            |     self._wrapped = Settings(settings_module)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/django/conf/__init__.py", line 170, in __init__
ibdax            |     mod = importlib.import_module(self.SETTINGS_MODULE)
ibdax            |   File "/usr/local/lib/python3.7/importlib/__init__.py", line 127, in import_module
ibdax            |     return _bootstrap._gcd_import(name[level:], package, level)
ibdax            |   File "<frozen importlib._bootstrap>", line 1006, in _gcd_import
ibdax            |   File "<frozen importlib._bootstrap>", line 983, in _find_and_load
ibdax            |   File "<frozen importlib._bootstrap>", line 967, in _find_and_load_unlocked
ibdax            |   File "<frozen importlib._bootstrap>", line 677, in _load_unlocked
ibdax            |   File "<frozen importlib._bootstrap_external>", line 728, in exec_module
ibdax            |   File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
ibdax            |   File "/ibdax/ibdax/settings.py", line 19, in <module>
ibdax            |     from ibdax.constants import (
ibdax            |   File "/ibdax/ibdax/constants.py", line 30, in <module>
ibdax            |     DEV_DATABASE_HOST=secrets.get_secrets("dev-database-host")
ibdax            |   File "/ibdax/ibdax/gcp_secret_manager.py", line 23, in get_secrets
ibdax            |     response = self.client.access_secret_version(request)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/cloud/secretmanager_v1/services/secret_manager_service/client.py", line 1155, in access_secret_version
ibdax            |     response = rpc(request, retry=retry, timeout=timeout, metadata=metadata,)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/gapic_v1/method.py", line 145, in __call__
ibdax            |     return wrapped_func(*args, **kwargs)
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 286, in retry_wrapped_func
ibdax            |     on_error=on_error,
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/retry.py", line 184, in retry_target
ibdax            |     return target()
ibdax            |   File "/usr/local/lib/python3.7/site-packages/google/api_core/grpc_helpers.py", line 75, in error_remapped_callable
ibdax            |     six.raise_from(exceptions.from_grpc_error(exc), exc)
ibdax            |   File "<string>", line 3, in raise_from
ibdax            | google.api_core.exceptions.PermissionDenied: 403 Request had insufficient authentication scopes.

I looked at the IAM role for the service account for Compute Instance, and I couldn't comprehend some of the messages there. This is a screenshot of it:

Answer 1

Please refrain from downloading a JSON service account key file to your local computer to start with. At https://cloud.google.com/sdk, you can download and install the gcloud command-line programme. You can then log in using your user account:

$ gcloud auth login && gcloud auth application-default login

There is no need for a service account because this will prompt you to connect in to your Google account online. From a security and auditing perspective, this is considerably better.

Similar to GKE, Cloud Functions, Cloud Run, etc., you should create a service account and run the instance as that service account when operating on GCE (or any "compute-based" platform). Use of the Default Compute Engine service account is prohibited. Additionally, avoid granting access to all secrets to the Default Compute Engine service account as this poses a serious security risk. Please go to Secret Manager Best Practices for further information.

Cloud-platform is not one of the OAuth scopes included by default for GCE. The scopes must be updated to include cloud-platform:

$ gcloud compute instances set-service-account "my-instance" --service-account "...@..." --scopes "cloud-platform"

Hope this helps!

Check out google cloud platform certification program here to learn from the expert!