Tech Blog | Elakkuvan Rajamani - Software Development & Technology Insights

The Sitecore Layout Service

The Sitecore layout service is a headless API end point that provides Sitecore content as a JSON response. This JSON response can be used in the rendering engine to render the content. We can use the LayoutService.Client to request the Sitecore Layout service for the response.

More details about the Sitecore Layout Request and Response Models can be read from the link:-

https://doc.sitecore.com/en/developers/100/developer-tools/sitecore-layout-service.html

Each time a page gets requested, the Sitecore RenderingEngine Middleware internally makes a request to the Sitecore Layout Service and fetch the JSON response of the page, deserialize the response as a strongly typed model SitecoreLayoutResponse and it will be set in the HttpContext and hence it will be available throughout the page life cycle. This response will be having details such as RequestInformation, Errors if any, Route Information etc., This is the in-short working flow between the request and responses in the headless approach.

And the aforementioned bindings will be happening via this [UseSitecoreRendering] middleware when it is decorated to any controller action as an attribute.

What we are going to achieve ?

As per the above flow, each time when a page gets requested the rendering engine middleware will make a call to the sitecore layout service for the JSON formatted response of the page regardless of whether there is a change in the page or not.

If we check the execution time in the middleware of each request/response is around 31656ms.

When the same page is requested for the second time ( even when there is no change), then it will be around the same as still there will be an API end point call to the Sitecore Layout Service.

So to achieve a significant improvement in this flow, we will be introducing a Distributed Caching Layer to the RenderingEngineMiddleware with the help of the custom request handlers.

How we are going to achieve it ?

Let us see how we will be introducing the Distributed Caching.

Adding a Distributed Caching Foundation Layer
Adding the CustomRequestHandlers in the Foundation (which will be used by the RenderingEngineMiddleware) and to use the caching in it
Registering the CustomRequestHandlers

For this POC I have used the SitecoreMVP site. Thank you for all the contributors of this repository…!!! Please feel free to clone the repository and follow the installation steps. This repository is build against Sitecore 10.1 with Asp.Net Core Rendering Host and Sitecore Headless Services and it will be running in the docker.

Step 1 – Adding a Distributed Caching Foundation Layer

Once the solution is cloned, add a new Foundation Layer as Mvp.Foundation.Caching.Rendering. This will be a project of type Class Library of Asp.Net Core framework as it will be used in the rendering host part.

We will be using Redis as a distributed caching for this demo. The cloned MVP site will be having a Redis container running within it on the port 6379.

We will be using the RedisStackExchange as the RedisClient. So please install the latest version via nuget package manager to this foundation layer.

We need set the redis connection string in the appsettings.json file as below.

	{
	"Sitecore": {
	"InstanceUri": "http://mvp-cd",
	"LayoutServicePath": "/sitecore/api/layout/render/jss",
	"DefaultSiteName": "mvp-site",
	"ApiKey": "{E2F3D43E-B1FD-495E-B4B1-84579892422A}"
	},
	"Logging": {
	"LogLevel": {
	"Default": "Debug",
	"Microsoft": "Information",
	"Microsoft.Hosting.Lifetime": "Information"
	}
	},
	"Redis": {
	"ConnectionString": "redis:6379,ssl=False,abortConnect=False"
	},
	"AllowedHosts": "*"
	}

view raw gistfile1.txt hosted with ❤ by GitHub

Next we will be creating an interface ICachingProvider as below.

	namespace Mvp.Foundation.Caching.Providers
	{
	using System.Threading.Tasks;
	using Microsoft.Extensions.Caching.Distributed;
	using Sitecore.LayoutService.Client.Response;

	public interface ICachingProvider
	{
	Task GetFromCacheAsync(string key) where T : class;
	T GetFromCache(string key) where T : class;
	Task SetCache(string key, T value, DistributedCacheEntryOptions options = null) where T : class;
	Task ClearCache(string key);
	string GetCacheValue(string key);
	void SetCacheAsString(string key, string value, DistributedCacheEntryOptions options = null);
	}
	}

view raw ICachingProvider.cs hosted with ❤ by GitHub

Next we will be implementing the DistributdCachingProvider type as below.

	namespace Mvp.Foundation.Caching.Providers
	{
	using System;
	using System.Threading.Tasks;
	using Microsoft.Extensions.Caching.Distributed;
	using Newtonsoft.Json;
	using Sitecore.LayoutService.Client;

	public class DistributedCachingProvider : ICachingProvider
	{
	private readonly IDistributedCache _distributedCache;
	private readonly ISitecoreLayoutSerializer _layoutSerializer;

	public DistributedCachingProvider(IDistributedCache distributedCache, ISitecoreLayoutSerializer layoutSerializer)
	{
	_distributedCache = distributedCache;
	_layoutSerializer = layoutSerializer;
	}

	public async Task GetFromCacheAsync(string key) where T : class
	{
	var cachedValue = await _distributedCache.GetStringAsync(key);
	return cachedValue != null ? JsonConvert.DeserializeObject(cachedValue) : null;
	}

	public T GetFromCache(string key) where T : class
	{
	var cachedValue = _distributedCache.GetString(key);
	return cachedValue != null ? JsonConvert.DeserializeObject(cachedValue) : null;
	}

	public async Task SetCache(string key, T value, DistributedCacheEntryOptions options = null) where T : class
	{
	var serializedValue = JsonConvert.SerializeObject(value);
	options ??= new DistributedCacheEntryOptions()
	{ AbsoluteExpiration = DateTimeOffset.Parse(DateTime.Now.AddDays(1).ToString()) };
	await _distributedCache.SetStringAsync(key, serializedValue, options);
	}

	public async Task ClearCache(string key)
	{
	await _distributedCache.RemoveAsync(key);
	}

	///
	/// Get the cached value from the Distributed cache
	///
	/// Cache key
	/// Cached values corresponds to the key passed
	public string GetCacheValue(string key)
	{
	return _distributedCache.GetString(key);
	}

	///
	/// Set the input string to Distributed Cache
	///
	/// Cache key
	/// Cache Value
	/// Cache entry options
	public void SetCacheAsString(string key, string value, DistributedCacheEntryOptions options = null)
	{
	options ??= new DistributedCacheEntryOptions()
	{
	AbsoluteExpiration = DateTimeOffset.Parse(DateTime.Now.AddDays(1).ToString())
	};
	_distributedCache.SetString(key, value, options);
	}
	}
	}

view raw DistributedCachingProvider.cs hosted with ❤ by GitHub

Finally registering this service to the ServiceCollection as below.

	public static class ServiceCollectionExtension
	{
	public static void AddCachingServices(this IServiceCollection services)
	{
	services.AddSingleton();
	services.AddSingleton();
	}
	}

view raw ServiceCollectionExtension.cs hosted with ❤ by GitHub

In RenderingHost project we need to call this method in the Startup.cs’s ConfigureServices() method. Now we will have this caching objects ready to be injected whenever needed. Also we need to register the Redis Distributed caching services as well.

Now we are having the Foundation Caching service ready.

Step 2: Creating a CustomRequestHandlers in the Foundation and to use the caching in it

The LayoutRequest from the RenderingEngineMiddleware will be handled by the default handler which is registered as below.

If we deep dive into this handler, we can notice that this handler functions cannot be overriden.

So we need to create CustomHandlers where we will applying the Caching Layer and we need to register the Custom Handler as the default LayoutRequestHandler in the application start.

Let us see how we will be creating this CustomHanlders.

In the same way we created the Caching project, we will be creating this CustomMiddleware in the Foundation Layer of the solution. Create a new dotnetcore class library named as Mvp.Foundation.CustomMiddleware.

Next we will be creating couple of class files which will be holding some extension methods.

CustomLayoutRequestBuilderExtension.cs

This will be holding set of functions which will be used to build the LayoutRequest obj from the CustomRequestHandler which we will be creating shortly. This will be creating and registering the Handler of type CustomRequestHandler to handle the layout requests.

	using Microsoft.Extensions.Configuration;
	using Microsoft.Extensions.DependencyInjection;
	using Mvp.Foundation.CustomMiddleware.Handlers;
	using Sitecore.LayoutService.Client;
	using Sitecore.LayoutService.Client.Extensions;
	using Sitecore.LayoutService.Client.Request;
	using System;
	using System.Collections.Generic;
	using System.Net.Http;
	using System.Net.Http.Headers;
	using System.Text;

	namespace Mvp.Foundation.CustomMiddleware.Extension
	{
	public static class CustomLayoutRequestBuilderExtension
	{
	///
	/// Registers a CustomHttpHandler for the layout service client
	///
	///
	///
	///
	///
	public static ILayoutRequestHandlerBuilder AddCustomLayoutRequestHandler(
	this ISitecoreLayoutClientBuilder builder,
	string handlerName,
	Uri url)
	{
	return builder.AddHttpHandler(handlerName, url);
	}

	///
	/// Sets the LayoutRequest address
	///
	///
	///
	///
	///
	public static ILayoutRequestHandlerBuilder AddHttpHandler(this ISitecoreLayoutClientBuilder builder, string handlerName, Uri uri)
	{
	Uri uriInternal = uri;
	return builder.AddHttpHandler(handlerName, delegate (HttpClient client)
	{
	client.BaseAddress = uriInternal;
	});
	}

	///
	/// Registers the HttpClient
	///
	///
	///
	///
	///
	public static ILayoutRequestHandlerBuilder AddHttpHandler(this ISitecoreLayoutClientBuilder builder, string handlerName, Action configure)
	{
	string handlerNameInternal = handlerName;
	builder.Services.AddHttpClient(handlerNameInternal, configure).ConfigurePrimaryHttpMessageHandler(() => new HttpClientHandler
	{
	UseCookies = false
	});
	return builder.AddHttpHandler(handlerNameInternal, (IServiceProvider sp) => sp.GetRequiredService().CreateClient(handlerNameInternal));
	}

	///
	/// Registers a HTTP request handler for the Sitecore layout service client.
	///
	///
	///
	///
	///
	public static ILayoutRequestHandlerBuilder AddHttpHandler(this ISitecoreLayoutClientBuilder builder, string handlerName, Func resolveClient)
	{
	Func resolveClientInternal = resolveClient;
	ILayoutRequestHandlerBuilder layoutRequestHandlerBuilder = builder.AddHandler(handlerName, delegate (IServiceProvider sp)
	{
	HttpClient httpClient = resolveClientInternal(sp);
	return ActivatorUtilities.CreateInstance(sp, new object[1] { httpClient });
	});
	layoutRequestHandlerBuilder.MapFromRequest(delegate (SitecoreLayoutRequest request, HttpRequestMessage message)
	{
	message.RequestUri = request.BuildDefaultSitecoreLayoutRequestUri(message.RequestUri);
	if (request.TryReadValue("sc_auth_header_key", out var value))
	{
	message.Headers.Authorization = new AuthenticationHeaderValue("Bearer", value);
	}
	if (request.TryGetHeadersCollection(out var headers))
	{
	foreach (KeyValuePair item in headers)
	{
	message.Headers.Add(item.Key, item.Value);
	}
	}
	});
	return layoutRequestHandlerBuilder;
	}
	///
	/// Registers the HttpLayoutRequestHandlerOptions
	///
	///
	///
	///
	public static ILayoutRequestHandlerBuilder MapFromRequest(this ILayoutRequestHandlerBuilder builder, Action configureHttpRequestMessage)
	{
	Action configureHttpRequestMessageInternal = configureHttpRequestMessage;
	builder.Services.Configure(builder.HandlerName, delegate (HttpLayoutRequestHandlerOptions options)
	{
	options.RequestMap.Add(configureHttpRequestMessageInternal);
	});
	return builder;
	}
	}
	}

view raw CustomLayoutRequestBuilderExtension.cs hosted with ❤ by GitHub

CustomSitecoreLayoutRequestHttpExtensions.cs

This class will be holding set of functions to build the LayoutRequest Uri with parameters like API Key, Item Path, Language and Sitename etc.,

	using Sitecore.LayoutService.Client.Request;
	using System;
	using System.Collections.Generic;
	using System.Linq;
	using System.Net;
	using System.Text;

	namespace Mvp.Foundation.CustomMiddleware
	{
	public static class CustomSitecoreLayoutRequestHttpExtensions
	{
	private static readonly List _defaultSitecoreRequestKeys = new List { "sc_site", "item", "sc_lang", "sc_apikey", "sc_mode", "sc_date" };

	///
	/// Build an URI using the default Sitecore layout entries in the provided request.
	///
	/// The request object.
	/// The base URI used to compose the final URI.
	/// An URI containing the base URI and the relevant entries in the request object added as query strings.
	public static Uri BuildDefaultSitecoreLayoutRequestUri(this SitecoreLayoutRequest request, Uri baseUri)
	{
	return BuildUri(request, baseUri, _defaultSitecoreRequestKeys);
	}

	///
	/// Build an URI using the default Sitecore layout entries in the provided request.
	///
	/// The request object.
	/// The base URI used to compose the final URI.
	/// The additional URI query parameters to get from the request.
	/// An URI containing the base URI and the relevant entries in the request object added as query strings.
	public static Uri BuildDefaultSitecoreLayoutRequestUri(this SitecoreLayoutRequest request, Uri baseUri, IEnumerable additionalQueryParameters)
	{
	List list = new List(_defaultSitecoreRequestKeys);
	list.AddRange(additionalQueryParameters);
	return BuildUri(request, baseUri, list);
	}

	///
	/// Build an URI using all the entries in the provided request.
	///
	/// The request object.
	/// The base URI used to compose the final URL.
	/// The URI query parameters to get from request.
	/// An URI containing the base URI and all the valid entries in the request object added as query strings.
	public static Uri BuildUri(this SitecoreLayoutRequest request, Uri baseUri, IEnumerable queryParameters)
	{
	IEnumerable queryParameters2 = queryParameters;
	string[] array = (from entry in request.Where((KeyValuePair entry) => queryParameters2.Contains(entry.Key)).ToList()
	where entry.Value is string && !string.IsNullOrWhiteSpace(entry.Value.ToString())
	select entry into kvp
	select WebUtility.UrlEncode(kvp.Key) + "=" + WebUtility.UrlEncode(kvp.Value.ToString())).ToArray();
	if (!array.Any())
	{
	return baseUri;
	}
	string query = "?" + string.Join("&", array);
	return new UriBuilder(baseUri)
	{
	Query = query
	}.Uri;
	}
	}
	}

view raw CustomSitecoreLayoutRequestHttpExtensions.cs hosted with ❤ by GitHub

HttpRequestExtension.cs

This class contains an extension method for SitecoreLayoutRequest class to generate the cache keys based on the request.

	using Microsoft.AspNetCore.Http;
	using Sitecore.LayoutService.Client.Request;

	namespace Mvp.Foundation.CustomMiddleware.Extension
	{
	public static class HttpRequestExtension
	{
	public static string GenerateCacheKeyForHttpRequest(this SitecoreLayoutRequest request)
	{
	string format = "{0}-{1}-{2}";
	object item = string.Empty;
	object lang = string.Empty;
	object site = string.Empty;

	if (request.TryGetValue("item", out item)
	&& request.TryGetValue("sc_lang", out lang)
	&& request.TryGetValue("sc_site", out site))
	return string.Format(format, site, item, lang).ToLower().Replace("/", "-");
	return string.Empty;
	}
	}
	}

view raw HttpRequestExtension.cs hosted with ❤ by GitHub

CustomHttpRequestHandler.cs

This is our custom layout request handler which will be handling the layout request from the RenderingEngineMiddleware.

	#nullable enable
	using Microsoft.Extensions.Logging;
	using Microsoft.Extensions.Options;
	using Mvp.Foundation.Caching.Providers;
	using Mvp.Foundation.CustomMiddleware.Extension;
	using Newtonsoft.Json;
	using Sitecore.LayoutService.Client;
	using Sitecore.LayoutService.Client.Exceptions;
	using Sitecore.LayoutService.Client.Request;
	using Sitecore.LayoutService.Client.RequestHandlers;
	using Sitecore.LayoutService.Client.Response;
	using System;
	using System.Collections.Generic;
	using System.Diagnostics;
	using System.Linq;
	using System.Net;
	using System.Net.Http;
	using System.Resources;
	using System.Text;
	using System.Threading.Tasks;

	namespace Mvp.Foundation.CustomMiddleware.Handlers
	{
	public class CustomHttpRequestHandler : ILayoutRequestHandler
	{
	private readonly ISitecoreLayoutSerializer _serializer;

	private readonly HttpClient _client;

	private readonly IOptionsSnapshot _options;

	private readonly ILogger _logger;

	private readonly IOptions _layoutClientOptions;

	private readonly IOptionsSnapshot _layoutRequestOptions;

	private readonly ICachingProvider _cachingProvider;

	public CustomHttpRequestHandler(HttpClient client, ISitecoreLayoutSerializer serializer, IOptionsSnapshot options, ILogger logger, IOptions layoutClientOptions, IOptionsSnapshot layoutRequestOptions, ICachingProvider cachingprovider)
	{
	_client = client;
	_serializer = serializer;
	_options = options;
	_logger = logger;
	_layoutClientOptions = layoutClientOptions;
	_cachingProvider = cachingprovider;
	}

	///
	public async Task Request(SitecoreLayoutRequest request, string handlerName)
	{
	Stopwatch timer = Stopwatch.StartNew();
	SitecoreLayoutResponseContent content = null;
	string cacheKey = request.GenerateCacheKeyForHttpRequest();
	ILookup metadata = null;
	List errors = new List();
	try
	{
	HttpLayoutRequestHandlerOptions options = _options.Get(handlerName);
	HttpRequestMessage httpRequestMessage;
	try
	{
	httpRequestMessage = BuildMessage(request, options);
	if (_logger.IsEnabled(LogLevel.Debug))
	{
	_logger.LogDebug($"Layout Service Http Request Message : {httpRequestMessage}");
	}
	}
	catch (Exception ex)
	{
	errors = AddError(errors, new SitecoreLayoutServiceMessageConfigurationException(ex));
	if (_logger.IsEnabled(LogLevel.Debug))
	{
	_logger.LogDebug($"An error configuring the HTTP message : {ex}");
	}
	return new SitecoreLayoutResponse(request, errors);
	}
	// Read the layout response from Distributed Cache
	string contentText = _cachingProvider.GetCacheValue(cacheKey);
	if (!string.IsNullOrEmpty(contentText))
	{
	content = _serializer.Deserialize(contentText);
	timer.Stop();
	_logger.LogDebug($"Total Execution Time : {timer.ElapsedMilliseconds}");
	return new SitecoreLayoutResponse(request, errors)
	{
	Content = content,
	Metadata = metadata
	};

	}
	HttpResponseMessage httpResponse = await GetResponseAsync(httpRequestMessage).ConfigureAwait(continueOnCapturedContext: false);
	if (_logger.IsEnabled(LogLevel.Debug))
	{
	_logger.LogDebug($"Layout Service Http Response : {httpResponse}");
	}
	int responseStatusCode = (int)httpResponse.StatusCode;
	if (!httpResponse.IsSuccessStatusCode)
	{
	int num = responseStatusCode;
	List list;
	if (responseStatusCode == 404)
	{
	list = AddError(errors, new ItemNotFoundSitecoreLayoutServiceClientException(), responseStatusCode);
	}
	else
	{
	list = ((responseStatusCode >= 400 && responseStatusCode < 500) ? AddError(errors, new InvalidRequestSitecoreLayoutServiceClientException(), responseStatusCode) : ((responseStatusCode < 500) ? AddError(errors, new SitecoreLayoutServiceClientException(), responseStatusCode) : AddError(errors, new InvalidResponseSitecoreLayoutServiceClientException(new SitecoreLayoutServiceServerException()), responseStatusCode)));
	}
	errors = list;
	}
	if (httpResponse.IsSuccessStatusCode \|\| httpResponse.StatusCode == HttpStatusCode.NotFound)
	{
	try
	{
	string text = await httpResponse.Content.ReadAsStringAsync().ConfigureAwait(continueOnCapturedContext: false);
	string newone = await Task.Run(() =>
	newone = text
	);
	content = _serializer.Deserialize(text);
	await Task.Run(() => _cachingProvider.SetCacheAsString(cacheKey, text));
	if (_logger.IsEnabled(LogLevel.Debug))
	{
	object arg = JsonConvert.DeserializeObject(text);
	_logger.LogDebug($"Layout Service Response JSON : {arg}");
	}
	}
	catch (Exception innerException)
	{
	errors = AddError(errors, new InvalidResponseSitecoreLayoutServiceClientException(innerException), responseStatusCode);
	}
	}
	try
	{
	metadata = httpResponse.Headers.SelectMany((KeyValuePair> x) => x.Value.Select((string y) => new
	{
	Key = x.Key,
	Value = y
	})).ToLookup(k => k.Key, v => v.Value);
	}
	catch (Exception innerException2)
	{
	errors = AddError(errors, new InvalidResponseSitecoreLayoutServiceClientException(innerException2), responseStatusCode);
	}
	}
	catch (Exception innerException3)
	{
	errors.Add(new CouldNotContactSitecoreLayoutServiceClientException(innerException3));
	}
	timer.Stop();
	_logger.LogDebug($"Total Execution Time : {timer.ElapsedMilliseconds}");
	return new SitecoreLayoutResponse(request, errors)
	{
	Content = content,
	Metadata = metadata
	};
	}

	protected virtual HttpRequestMessage BuildMessage(SitecoreLayoutRequest request, HttpLayoutRequestHandlerOptions options)
	{
	HttpRequestMessage httpRequestMessage = new HttpRequestMessage(HttpMethod.Get, _client.BaseAddress);
	if (options != null)
	{
	foreach (Action item in options.RequestMap)
	{
	item(request, httpRequestMessage);
	}
	return httpRequestMessage;
	}
	return httpRequestMessage;
	}

	protected virtual async Task GetResponseAsync(HttpRequestMessage message)
	{
	return await _client.SendAsync(message).ConfigureAwait(continueOnCapturedContext: false);
	}

	private static List AddError(List errors, SitecoreLayoutServiceClientException error, int statusCode = 0)
	{
	if (statusCode > 0)
	{
	error.Data.Add("HttpStatusCode_KeyName", statusCode);
	}
	errors.Add(error);
	return errors;
	}


	private SitecoreLayoutRequestOptions MergeLayoutRequestOptions(string handlerName)
	{
	SitecoreLayoutRequestOptions value = _layoutRequestOptions.Value;
	SitecoreLayoutRequestOptions sitecoreLayoutRequestOptions = _layoutRequestOptions.Get(handlerName);
	if (AreEqual(value.RequestDefaults, sitecoreLayoutRequestOptions.RequestDefaults))
	{
	return value;
	}
	SitecoreLayoutRequestOptions sitecoreLayoutRequestOptions2 = value;
	SitecoreLayoutRequest requestDefaults = value.RequestDefaults;
	SitecoreLayoutRequest requestDefaults2 = sitecoreLayoutRequestOptions.RequestDefaults;
	foreach (KeyValuePair item in requestDefaults2)
	{
	if (requestDefaults.ContainsKey(item.Key))
	{
	requestDefaults[item.Key] = requestDefaults2[item.Key];
	}
	else
	{
	requestDefaults.Add(item.Key, requestDefaults2[item.Key]);
	}
	}
	sitecoreLayoutRequestOptions2.RequestDefaults = requestDefaults;
	return sitecoreLayoutRequestOptions2;
	}


	private static bool AreEqual(IDictionary dictionary1, IDictionary dictionary2)
	{
	if (dictionary1.Count != dictionary2.Count)
	{
	return false;
	}
	foreach (string key in dictionary1.Keys)
	{
	if (!dictionary2.TryGetValue(key, out var value) \|\| dictionary1[key] != value)
	{
	return false;
	}
	}
	return true;
	}
	}
	}

view raw CustomHttpRequestHandler.cs hosted with ❤ by GitHub

In the above custom handler, Request is the function which will actually handle the layout request. This works as per the below logic,

When a page is first requested, we will form the Redis cache key from the LayoutRequest info with the help of the extension method that we created above
Then we will check whether the page response information are already exists in the Redis Distributed Cache. If exists, then we will deserilize the value as SitecoreLayoutResponseContent and respond back ( skipping the actual Layout Request)
If not exists in the redis, then actual LayoutRequestService call will be made to get the JSON response from the headless service. And eventually the response will be saved into Redis cache with the key. So when the same page is requested next time, it will served from Redis.

Now we have created our CustomLayoutRequest handler. We need to register this a default handler in the startup.

Demo

When a page from the Mvp-Site is accessed first time. You can notice that there is an actual call to the Sitecore headless service and the time taken in ms.

When the same page is accessed second time and you can notice there is no call to the headless service and great improvement in the response time.

What is next ?

To purge the redis cache whenever a page is changed and published. So that the page response will be intact.

Happy Sitecoring…!!!