Ranger Kms having multiple instance enabled

Question

Hi team,

I was installing multiple ranger KMS service on the Ambari. When I tried with a single instance say suppose node1 we are able to create the KMS key successfully.

After that, I added one more instance .and try to create key say suppose node2 

ITS throwing error as:

[user@data01 ~]$ hadoop key create keytrustee_test2
20/10/22 14:52:19 WARN kms.LoadBalancingKMSClientProvider: KMS provider at [http://node2::19292/kms/v1/] threw an IOException!! java.io.FileNotFoundException: http://node2:9292/kms/v1/keys?user.name=opc
    at org.apache.hadoop.security.authentication.client.AuthenticatedURL.extractToken(AuthenticatedURL.java:275)
    at org.apache.hadoop.security.authentication.client.PseudoAuthenticator.authenticate(PseudoAuthenticator.java:77)
    at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:133)
    at org.apache.hadoop.security.authentication.client.KerberosAuthenticator.authenticate(KerberosAuthenticator.java:212)
    at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticator.authenticate(DelegationTokenAuthenticator.java:133)
    at org.apache.hadoop.security.authentication.client.AuthenticatedURL.openConnection(AuthenticatedURL.java:216)
    at org.apache.hadoop.security.token.delegation.web.DelegationTokenAuthenticatedURL.openConnection(DelegationTokenAuthenticatedURL.java:322)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:542)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider$1.run(KMSClientProvider.java:537)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1869)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createConnection(KMSClientProvider.java:536)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKeyInternal(KMSClientProvider.java:730)
    at org.apache.hadoop.crypto.key.kms.KMSClientProvider.createKey(KMSClientProvider.java:740)
    at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$13.call(LoadBalancingKMSClientProvider.java:300)
    at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider$13.call(LoadBalancingKMSClientProvider.java:296)
    at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.doOp(LoadBalancingKMSClientProvider.java:95)
    at org.apache.hadoop.crypto.key.kms.LoadBalancingKMSClientProvider.createKey(LoadBalancingKMSClientProvider.java:296)
    at org.apache.hadoop.crypto.key.KeyShell$CreateCommand.execute(KeyShell.java:483)
    at org.apache.hadoop.crypto.key.KeyShell.run(KeyShell.java:79)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
    at org.apache.hadoop.crypto.key.KeyShell.main(KeyShell.java:515)

Same for node 2 

What exactly I am missing here. Please pour some suggestions.

Answer 1

Hi@Shllpa,

First, you need to check the network connectivity between your instance and all the configuration files of KMS. Another reason maybe if the KMS (Hadoop Key Management Server) is down and Kerberos is not able to talk to this KMS service to get the delegation token to submit the ingestion map-reduce job.

Comments

Hi,
Thanks for inputs but connectivity is fine "

[user@data05 ~]$ curl -v -u  "keyadmin:keyadmin1" http://URL:9292/kms
* About to connect() to data05.data.cxidev.oraclevcn.com port 9292 (#0)
*   Trying <ipaddr>
* Connected to data05.data.cxidev.oraclevcn.com (172.16.8.6) port 9292 (#0)
* Server auth using Basic with user 'keyadmin'
> GET /kms HTTP/1.1
> Authorization: Basic <auth>
> User-Agent: curl/7.29.0
> Host: urlm:9292
> Accept: */*
>
< HTTP/1.1 404 Not Found
< Server: Apache-Coyote/1.1
< Content-Length: 0
< Date: Fri, 23 Oct 2020 06:38:06 GMT
<
* Connection #0 to host user left intact"

---

Hi@Shilpa,

It is related to network connectivity. In your second instance check all the configuration files.