Azure KeyVault Active Directory AcquireTokenAsync timeout when called asynchronously

0 votes

I have setup Azure Keyvault on my ASP.Net MVC web application by following the example in Microsoft's Hello Key Vault sample application.

Azure KeyVault (Active Directory) AuthenticationResult by default has a one hour expiry. So after one hour, you must get a new authentication token. KeyVault is working as expected for the first hour after getting my first AuthenticationResult token, but after the 1 hour expiry, it fails to get a new token.

Unfortunately it took a failure on my production environment for me to realize this, as I never tested past one hour in development.

Anyways, after over two days of trying to figure out what was wrong with my keyvault code, I came up with a solution that fixes all of my problems - remove the asynchronous code - but it feels very hacky. I want to find out why it was not working in the first place.

My code looks like this:

public AzureEncryptionProvider() //class constructor
{
   _keyVaultClient = new KeyVaultClient(GetAccessToken);
   _keyBundle = _keyVaultClient
     .GetKeyAsync(_keyVaultUrl, _keyVaultEncryptionKeyName)
     .GetAwaiter().GetResult();
}

private static readonly string _keyVaultAuthClientId = 
    ConfigurationManager.AppSettings["KeyVaultAuthClientId"];

private static readonly string _keyVaultAuthClientSecret =
    ConfigurationManager.AppSettings["KeyVaultAuthClientSecret"];

private static readonly string _keyVaultEncryptionKeyName =
    ConfigurationManager.AppSettings["KeyVaultEncryptionKeyName"];

private static readonly string _keyVaultUrl = 
    ConfigurationManager.AppSettings["KeyVaultUrl"];

private readonly KeyBundle _keyBundle;
private readonly KeyVaultClient _keyVaultClient;

private static async Task<string> GetAccessToken(
    string authority, string resource, string scope)
{
   var clientCredential = new ClientCredential(
       _keyVaultAuthClientId, 
       _keyVaultAuthClientSecret);
   var context = new AuthenticationContext(
       authority, 
       TokenCache.DefaultShared);
   var result = context.AcquireToken(resource, clientCredential);
   return result.AccessToken;
}

The GetAccessToken method signature has to be asynchronous to pass into the new KeyVaultClient constructor, so I left the signature as async, but I removed the await keyword.

With the await keyword in there (the way it should be, and is in the sample):

private static async Task<string> GetAccessToken(string authority, string resource, string scope)
{
   var clientCredential = new ClientCredential(_keyVaultAuthClientId, _keyVaultAuthClientSecret);
   var context = new AuthenticationContext(authority, null);
   var result = await context.AcquireTokenAsync(resource, clientCredential);
   return result.AccessToken;
}

The program works fine the first time I run it. And for an hour, AcquireTokenAsync returns the same original authentication token which is great. But once the token expires, AcquiteTokenAsync should get a new token with a new expiry date. And it doesn't - the application just hangs. No error returned, nothing at all.

So calling AcquireToken instead of AcquireTokenAsync solves the problem, but I have no idea why. You'll also notice that I'm passing 'null' instead of 'TokenCache.DefaultShared' into the AuthenticationContext constructor in my sample code with async. This is to force the toke to expire immediately instead of after one hour. Otherwise, you have to wait an hour to reproduce the behavior.

I was able to reproduce this again in a brand new MVC project, so I don't think it has anything to do with my specific project. Any insight would be appreciated. But for now, I'm just not using async.

Aug 24, 2018 in Azure by null_void
• 3,220 points
462 views

1 answer to this question.

0 votes

Problem: deadlock

Your EncryptionProvider() is calling GetAwaiter().GetResult(). This blocks the thread, and on subsequent token requests, causes a deadlock. The following code is the same as yours is but separates things to facilitate explanation.

public AzureEncryptionProvider() // runs in ThreadASP
{
    var client = new KeyVaultClient(GetAccessToken);

    var task = client.GetKeyAsync(KeyVaultUrl, KeyVaultEncryptionKeyName);

    var awaiter = task.GetAwaiter();

    // blocks ThreadASP until GetKeyAsync() completes
    var keyBundle = awaiter.GetResult();
}

In both token requests, the execution starts in the same way:

  • AzureEncryptionProvider() runs in what we'll call ThreadASP.
  • AzureEncryptionProvider() calls GetKeyAsync().

Then things differ. The first token request is multi-threaded:

  1. GetKeyAsync() returns a Task.
  2. We call GetResult() blocking ThreadASP until GetKeyAsync() completes.
  3. GetKeyAsync() calls GetAccessToken() on another thread.
  4. GetAccessToken() and GetKeyAsync() complete, freeing ThreadASP.
  5. Our web page returns to the user. Good.

GetAccessToken is running on its own thread.

The second token request uses a single thread:

  1. GetKeyAsync() calls GetAccessToken() on ThreadASP (not on a separate thread.)
  2. GetKeyAsync() returns a Task.
  3. We call GetResult() blocking ThreadASP until GetKeyAsync() completes.
  4. GetAccessToken() must wait until ThreadASP is free, ThreadASP must wait until GetKeyAsync() completes, GetKeyAsync() must wait until GetAccessToken() completes. Uh oh.
  5. Deadlock.

GetAccessToken is running on the same thread.

Why? Who knows?!?

There must be some flow control within GetKeyAsync() that relies on the state of our access token cache. The flow control decides whether to run GetAccessToken() on its own thread and at what point to return the Task.

Solution: async all the way down

To avoid a deadlock, it is a best practice "to use async all the way down." This is especially true when we are calling an async method, such as GetKeyAsync(), that is from an external library. It is important not force the method to by synchronous with Wait()Result, or GetResult(). Instead, use async and await because await pauses the method instead of blocking the whole thread.

Async controller action

public class HomeController : Controller
{
    public async Task<ActionResult> Index()
    {
        var provider = new EncryptionProvider();
        await provider.GetKeyBundle();
        var x = provider.MyKeyBundle;
        return View();
    }
}

Async public method

Since a constructor cannot be async (because async methods must return a Task), we can put the async stuff into a separate public method.

public class EncryptionProvider
{
    //
    // authentication properties omitted

    public KeyBundle MyKeyBundle;

    public EncryptionProvider() { }

    public async Task GetKeyBundle()
    {
        var keyVaultClient = new KeyVaultClient(GetAccessToken);
        var keyBundleTask = await keyVaultClient
            .GetKeyAsync(KeyVaultUrl, KeyVaultEncryptionKeyName);
        MyKeyBundle = keyBundleTask;
    }

    private async Task<string> GetAccessToken(
        string authority, string resource, string scope)
    {
        TokenCache.DefaultShared.Clear(); // reproduce issue 
        var authContext = new AuthenticationContext(authority, TokenCache.DefaultShared);
        var clientCredential = new ClientCredential(ClientIdWeb, ClientSecretWeb);
        var result = await authContext.AcquireTokenAsync(resource, clientCredential);
        var token = result.AccessToken;
        return token;
    }
}

Mystery solved. :) Here is a final reference that helped my understanding.

Console App

My original answer had this console app. It worked as an initial troubleshooting step. It did not reproduce the problem.

The console app loops every five minutes, repeatedly asking for a new access token. At each loop, it outputs the current time, the expiry time, and the name of the retrieved key.

On my machine, the console app ran for 1.5 hours and successfully retrieved a key after expiration of the original.

using System;
using System.Collections.Generic;
using System.Threading.Tasks;
using Microsoft.Azure.KeyVault;
using Microsoft.IdentityModel.Clients.ActiveDirectory;

namespace ConsoleApp
{
    class Program
    {
        private static async Task RunSample()
        {
            var keyVaultClient = new KeyVaultClient(GetAccessToken);

            // create a key :)
            var keyCreate = await keyVaultClient.CreateKeyAsync(
                vault: _keyVaultUrl,
                keyName: _keyVaultEncryptionKeyName,
                keyType: _keyType,
                keyAttributes: new KeyAttributes()
                {
                    Enabled = true,
                    Expires = UnixEpoch.FromUnixTime(int.MaxValue),
                    NotBefore = UnixEpoch.FromUnixTime(0),
                },
                tags: new Dictionary<string, string> {
                    { "purpose", "StackOverflow Demo" }
                });

            Console.WriteLine(string.Format(
                "Created {0} ",
                keyCreate.KeyIdentifier.Name));

            // retrieve the key
            var keyRetrieve = await keyVaultClient.GetKeyAsync(
                _keyVaultUrl,
                _keyVaultEncryptionKeyName);
answered Aug 24, 2018 by club_seesharp
• 3,450 points

Related Questions In Azure

0 votes
1 answer

Can we have a custom login page for Azure Active Directory?

Hey there! Feel Lucky. Custom company branding is ...READ MORE

answered Aug 8, 2018 in Azure by null_void
• 3,220 points
773 views
0 votes
1 answer

Why as a co-administrator of a subscription am I unable to edit the Active Directory?

The answer was that I needed to ...READ MORE

answered Aug 22, 2018 in Azure by null_void
• 3,220 points
27 views
0 votes
1 answer

Friendly filename when downloading Azure blob.

Azure blobs can be downloaded in four ...READ MORE

answered Mar 15 in Azure by Prerna
• 1,940 points
63 views
0 votes
1 answer

Chrome's default HTML5 player is not working when video is uploaded to Azure Storage.

I don't know much about Windows Storage, ...READ MORE

answered Apr 23 in Azure by Prerna
• 1,940 points
95 views
0 votes
1 answer

Getting an error when trying to authenticate using REST API Dynamics CRM with Azure AD.

Dynamics 365 has started to support Server-to-Server ...READ MORE

answered Apr 25 in Azure by Prerna
• 1,940 points
201 views
0 votes
2 answers
+1 vote
2 answers

How do I establish an connection to Azure IoT Hub? Say a connection like MQTT/AMQP using Python.

Azure IoT Hub comes with 3 SDK's: Service SDK Device ...READ MORE

answered Apr 13, 2018 in Azure by null_void
• 3,220 points
621 views
0 votes
1 answer

How to upload a file on to Azure Blob storage without writing a code?

You can find the below tools useful ...READ MORE

answered Apr 13, 2018 in Azure by club_seesharp
• 3,450 points
170 views
+1 vote
2 answers

Getting an App pool timeout for azure web sites

Windows Azure Web Sites supports two modes, ...READ MORE

answered Aug 21, 2018 in Azure by Priyaj
• 56,920 points
843 views
+1 vote
4 answers

How do Connect to Azure website via FTP?

First set up your FTP credentials are ...READ MORE

answered Oct 23, 2018 in Azure by abc
353 views