Blog / Development

Clearing Obsolete Cache Entries with Webhooks


by Jan Lenoch

Sep 21, 2017

Do you store your Kentico Cloud content in your app's cache? Have you ever wanted to keep the cache clutter free, with just up-to-date entries? In this how-to article I'll show how to go about clearing the obsolete cache entries upon a webhook call made from Kentico Cloud.

In a step-by-step manner, I'll configure a webhook in Kentico Cloud and create a simple ASP.NET Core MVC app that listens to webhook calls to clear its cache entries appropriately.

In general, apps that utilize Kentico Cloud don't need much caching. Kentico Cloud Delivery/Preview API service exposes your content through a worldwide-operating CDN network with response latencies measured in milliseconds. So your web apps don't necessarily need to cache the content and can refer to Kentico Cloud directly.

But, there might be a reason for you to implement caching: you may wish to save on API requests made to Kentico Cloud. Our free plan includes 50,000 API requests per month and our paid plans include even higher amounts of prepaid requests, so you don't have to feel compelled to cache your content, but if you wish to do so, here's how you'd set up webhooks to keep the cache up to date.

The Problem

So you have an app that caches Kentico Cloud content items for a fixed period of time. But what happens when a content contributor alters the content item in Kentico Cloud before the cache entry expires? The app will serve an obsolete version of the content item. Assuming that this is something you don't wish to happen, we've designed the webhooks feature. It's not just for cache invalidation; webhooks cater for various other cases in which apps wish to get signals of events happening in Kentico Cloud. See the article updating an Azure Search index with webhooks by Kentico Technical Evangelist, Bryan Soltis, for an example.

The Solution

Webhooks are nothing but HTTP POST requests that Kentico Cloud sends to the preconfigured, publicly routable URL addresses of your apps. The content of the webhook request (i.e. the payload) always contains detailed information about what has happened. This way, your app can react appropriately—e.g. clear an obsolete cache item.

Set Up Kentico Cloud

As advertised above, the first place I went to set things up was Kentico Cloud. Here, I went to the main menu and chose "Webhooks". This menu item is accessible to the "Developer" and "Project manager" roles.

Thereafter, the steps were pretty straightforward.

I clicked "Create new Webhook" and was presented with a simple dialog.


I entered "Cache Webhook" as the name and then filled in the public URL of my future app's webhook endpoint ("http://example.com/webhook"). Then, once I had noted down the generated secret, I was ready to create my app.

Create the Example App

If you expected this part to be complicated, I'll disappoint you now: it’s fairly easy.

As before, I deployed our boilerplate code template via the "Dotnet New" command:

dotnet new kentico-cloud-mvc --name "WebhookCacheInvalidationMvc" --output "WebhookCacheInvalidationMvc"


From the highest-level perspective, I implemented the logic in the following steps:

  1. I created a class called CacheManager which is responsible solely for managing the state of the MemoryCache.
  2. I slightly modified the code of the existing  CachedDeliveryClient class so that it just prepares identifiers (codenames) for newly created cache items and its dependencies. It then calls the CacheManager class to work with the cache.
  3. I added a new MVC controller called WebhookController to represent the webhook endpoint mentioned above. The controller calls the CacheManager to invalidate cache entries.

The CacheManager Class

Apart from holding the reference to the MemoryCache object, this singleton class has two significant methods: GetOrCreateAsync<T> and InvalidateEntry. 

The first method is called by the CachedDeliveryClient's GetItem(s)Async methods. The method accepts "identifierTokens", "valueFactory", and "dependencyListFactory" as input parameters. 

The "valueFactory" delegate is used to build a fresh cache entry (by calling the Delivery/Preview API endpoint) should the cache entry be expired. The "dependencyListFactory" is then responsible for reading that entry and returning a collection of identifiers of entries that the current entry depends upon. The CachedDeliveryClient has specific methods for both the "valueFactory" and "dependencyListFactory" parameters.

public async Task<T> GetOrCreateAsync<T>(IEnumerable<string> identifierTokens, Func<Task<T>> valueFactory, Func<T, IEnumerable<IdentifierSet>> dependencyListFactory)
{
    // Check existence of the cache entry.
    if (!_memoryCache.TryGetValue(StringHelpers.Join(identifierTokens), out T entry))
    {
        // If it doesn't exist, get it via valueFactory.
        T response = await valueFactory();

        // Create it. (Could be off-loaded to a background thread.)
        CreateEntry(identifierTokens, response, dependencyListFactory);

        return response;
    }

    return entry;
}


The CreateEntry<T> method invokes the "dependencyListFactory" and uses the identifiers to create dummy cache items of all the dependencies. Each dummy entry holds  a CancellationTokenSource object that advertises invalidation of its corresponding cache entry to other entries that subscribe to its cancellation token. This is how dependencies are dealt with in .NET Core.

Finally, the InvalidateEntry method simply invalidates (clears) a specific cache entry, along with all entries that depend upon it. The InvalidateEntry method is called by the WebhookController class.

public void InvalidateEntry(IdentifierSet identifiers)
{
    var typeIdentifiers = new List<string>();

    // Aggregate several types that appear in webhooks into one.
    if (identifiers.Type.Equals(CacheHelper.CONTENT_ITEM_TYPE_CODENAME, StringComparison.Ordinal) || identifiers.Type.Equals(CacheHelper.CONTENT_ITEM_VARIANT_TYPE_CODENAME, StringComparison.Ordinal))
    {
        typeIdentifiers.AddRange(new[] { CacheHelper.CONTENT_ITEM_TYPE_CODENAME, string.Join(string.Empty, CacheHelper.CONTENT_ITEM_TYPE_CODENAME, "_variant"), string.Join(string.Empty, CacheHelper.CONTENT_ITEM_TYPE_CODENAME, "_typed"), string.Join(string.Empty, CacheHelper.CONTENT_ITEM_TYPE_CODENAME, "_runtime_typed") });
    }
    else if (identifiers.Type.Equals(CacheHelper.CONTENT_ITEM_LISTING_IDENTIFIER, StringComparison.Ordinal))
    {
        typeIdentifiers.AddRange(new[] { string.Join(string.Empty, CacheHelper.CONTENT_ITEM_LISTING_IDENTIFIER, "_typed"), string.Join(string.Empty, CacheHelper.CONTENT_ITEM_LISTING_IDENTIFIER, "_runtime_typed") });
    }
    else
    {
        typeIdentifiers.Add(identifiers.Type);
    }

    foreach (var typeIdentifier in typeIdentifiers)
    {
        if (_memoryCache.TryGetValue(StringHelpers.Join("dummy", typeIdentifier, identifiers.Codename), out CancellationTokenSource dummyEntry))
        {
            // Mark all subscribers to the CancellationTokenSource as invalid.
            dummyEntry.Cancel();
        }
    }
}

The CachedDeliveryClient Class

The GetItem(s)Async method now just calls the CacheManager. For instance, the following is the code of the strongly typed overload of the GetItemAsync method.

public async Task<DeliveryItemResponse<T>> GetItemAsync<T>(string codename, IEnumerable<IQueryParameter> parameters)
{
    var identifierTokens = new List<string> { string.Join(string.Empty, CacheHelper.CONTENT_ITEM_TYPE_CODENAME, "_typed"), codename };
    AddIdentifiersFromParameters(parameters, identifierTokens);

    return await _cacheManager.GetOrCreateAsync(identifierTokens, () => _deliveryClient.GetItemAsync<T>(codename, parameters), GetDependencies);
}


And here's what the "dependencyListFactory" instance looks like (the GetDependencies<T> method):

public static IEnumerable<IdentifierSet> GetDependencies<T>(T response)
{
    var dependencies = new List<IdentifierSet>();

    // Both single-item and listing responses depend on their modular content items. Create dummy items for all modular content items.
    AddModularContentDependencies(response, dependencies);

    // Single-item responses
    if (response is DeliveryItemResponse || (response.GetType().IsConstructedGenericType && response.GetType().GetGenericTypeDefinition() == typeof(DeliveryItemResponse<>)))
    {
        // Create dummy item for the content item itself.
        var ownDependency = new IdentifierSet
        {
            Type = CacheHelper.CONTENT_ITEM_TYPE_CODENAME,
            Codename = GetContentItemCodenameFromResponse(response)
        };

        if (!dependencies.Contains(ownDependency))
        {
            dependencies.Add(ownDependency);
        }
    }

    // Listing responses
    else if (response is DeliveryItemListingResponse || (response.GetType().IsConstructedGenericType && response.GetType().GetGenericTypeDefinition() == typeof(DeliveryItemListingResponse<>)))
    {
        // Create dummy item for each content item in the listing.
        foreach (var codename in GetContentItemCodenamesFromListingResponse(response))
        {
            var dependency = new IdentifierSet
            {
                Type = CacheHelper.CONTENT_ITEM_TYPE_CODENAME,
                Codename = codename
            };

            if (!dependencies.Contains(dependency))
            {
                dependencies.Add(dependency);
            }
        }
    }

    return dependencies;
}


The AddModularContentDependencies method and other backend methods simply rely on type dynamic to extract codenames from the ModularContent property.

The WebhookController Class

The job of the controller is simply to:

• check the signature (via the KenticoCloudSignatureActionFilter)
• check the type of the Kentico Cloud artifacts and their codenames
• invoke the InvalidateEntry method, if needed

[ServiceFilter(typeof(KenticoCloudSignatureActionFilter))]
public IActionResult Index([FromBody] KenticoCloudWebhookModel model)
{
    switch (model.Message.Type)
    {
        case CacheHelper.CONTENT_ITEM_TYPE_CODENAME:
        case CacheHelper.CONTENT_ITEM_VARIANT_TYPE_CODENAME:
            switch (model.Message.Operation)
            {
                case "archive":
                case "publish":
                case "unpublish":
                case "upsert":
                    foreach (var item in model.Data.Items)
                    {
                        _cacheManager.InvalidateEntry(new IdentifierSet
                        {
                            Type = model.Message.Type,
                            Codename = item.Codename
                        });
                    }

                    break;
                default:
                    return Ok();
            }

            return Ok();
        default:
            return Ok();
    }
}

Testing Time

There were basically two things to test:

  • whether the cache gets populated with all its dependencies
  • if the call to the WebhookController invalidates the cache entries

The cache was correctly populated, including the dummy entries of all the ModularContent items. In the case of listing responses, all content items in the listing produced their corresponding dummy entries. So, if any of the items in either the modular content or the listing were obsolete, the whole listing would have been properly invalidated. 


Upon a WebhookController call, the cached items and listings (i.e. the cache entry) were first marked invalid and then purged. The MemoryCache's Get method correctly returned “null” instead of the stale cache entry. That in turn caused the app to fetch fresh content. The cache entry was also purged when one of the dependencies was invalidated.


All was set and done.

Get the Code

Of course, you can get the code of the example app from a common article examples repository in GitHub. We're also planning on merging the functionality into our Boilerplate template.

Moving Forward

There are a few things that could be improved. E.g., the content items in the cache could be invalidated when their underlying content type is altered in Kentico Cloud. If you wish to see that, just tell us, either through the comments below or in our forums!