Automating the Sitecore Content Migration: Choosing between SCS and SPE Remote Package Creation for seamless integration with Azure Pipeline

Problem

Sitecore Content Serialization serves below multiple purposes,

• To serialize the content items
• Enabling seamless transfer between environments
• Also to facilitate effective version control

However, an issue arises when dealing with a large volume of items with the scope ItemAndDescendants, leading to BadGateway issues and CM also runs out of memory. Despite attempting various solutions, such as adjusting the apiClientTimeoutInMinutes, disabling the versionComparision in user.json file, utilizing the latest version of Sitecore CLI, and extending the CDN timeouts, none of these measures proved effective. So the last option was to raise a support ticket and still the issue is under Sitecore’s investigation.

And other limitation of SCS: its inability to serialize only the Delta changes. Instead it exports all the content even if there are only few item changes. And SCS by default serialises all the language versions of an item. It results in larger processing time especially when dealing with larger amount of content items and loss of existing data in the environment.

To address the timeouts with the “ItemAndDescendants” scope, the workaround involves excluding item tree with larger number of items from the main modules json file and creating a seperate module json file specifically for that item tree. The individual module json file can then include multiple items as rules within that tree. It adds additional complexity in creating these modules.json with individual entries, leading it to extra overhead.

The individual module json of the larger content tree is depicted below and this would circumvent the BadGateway issue.

What approach can be taken to address this issue ?

The approach would be to use SPE Scripts to create the dynamic packages and execute them with PSE Remoting in order to integrate it with the Azure pipeline to automate it for the seamless Content migration between environments.

SPE offers packaging modules to pack the sitecore items without having to open the Packaging Designer everytime and there are number of comments available for generating the packages. However there is no straight forward approach to create the dynamic package using these available commands.

This blog will explain into the following topics,

• Creating Dynamic Packages using SPE
• Leveraging the SPE Remoting with the Dynamic Packages
  • Uploading the packages to the desired environment
  • Installing the dynamic package
• Integrating the solution with Azure DevOps Pipeline for seamless content migration across environments

Creating Dynamic Packages using SPE:

By default, the existing packaging commands, such as New-ItemSource and New-ExplicitItemSource includes all the item’s languages and there is no option available to exclusively include the deltas. If we examine the SPE’s binary closely, we can observe it.

Therefore the solution would be to apply the same logic used by Sitecore out of the box to create the Dynamic Packages. The below given packaging template xml will be used with appropriate placeholders to replace the Root Item ID, Package Name, Date Filters and Languages and other relevant parameters., from the Azure Pipeline during the run.

The full template xml file can be downloaded from the gist.

The placeholders can be replaced from the Azure Pipeline with relavant values and this XML file can be used. The next step is to use the same logic of the dynamic package creation alongside this template xml file.

Upon examining the Sitecore’s PackageGenerator assembly, it is evident that it offers a static method which accepts the template XML File, package name and processing context as parameters. This is the same method being used by the Sitecore Dynamic Package creator by default. So we have the freedom to utilize the same in our scenario.

The following Powershell script utilizes the aforementioned method with the necessary parameters. Pleae be mindful, to sure that you upload the template XML File to the package folder of CMS instance for the powershell script to function standalone.

$packageFile = "$SitecorePackageFolder\PACKAGE_NAME"
$solutionFile = "$SitecorePackageFolder\BASE_TEMPLATE_XML_FILE_NAME"
$simpleProcessingContext = New-Object -TypeName Sitecore.Install.Framework.SimpleProcessingContext
[Sitecore.Install.PackageGenerator]::GeneratePackage($solutionFile, $packageFile, $simpleProcessingContext);

The above code creates the required package file based on the parameters specified in the template xml file. It then uploads the package to the package folder located within the CM instance.

Next let us explore how SPE Remoting can be leveraged alongside with the aformentioned script and the reason for its necessity.

Configuring Remote Powershell Executions in CM instance:

To enable Remote Powershell executions on Sitecore CM instance, the following steps can be followed:

• Create a patch file to enable the below services under <powershell>/<services>
  • remoting
  • fileDownload
  • fileUpload
• Create a user ‘sitecore/speremote’ for instance and assign the role ‘sitecore\PowerShell Extensions Remoting’
• Within the same patch file, ensure to authorize the newly created user foreach of the aformentioned service
• Optionally, you can choose to set the <detailedAuthenticationErrors> attribute to true. Enabling this option would provide detailed information about the authentication errors encountered for better troubleshooting

The final /showconfig.aspx will be like this,

Leveraging the SPE Remoting to create the Dynamic Packages:

Before proceeding, it is essential to setup the PSE Remoting module by following the instructions provided in the link.

In order to automate the Content Migration across environments, we will be integrating our dynamic package PSE script alongside the PSE Remoting modules. This integration allows us to trigger the package creation on the Remote Sitecore instance from Azure build agents using the newly created remote user context.

The complete script, which facilitates remote execution, is depicted below and can be downloaded from the gist.

This script will create content package using the template XML file on the remote Sitecore CM and upload the same to the package folder. Kindly note that the package creation steps are being executed as Job. This approach ensures continuous polling until the job is successfully completed. This helps to avoid any timeouts that may occur during the execution in the Azure pipeline.

To achieve end-to-end automation of the Content Migration, we would also need the following steps to be carried out:

• Dowloading the content package from Remote CM instance to the local folder within the build agent
• Uploading the package to the targeted CM where this package needs to be installed
• Executing remote scripts to install the Sitecore Content Package

The overall high-level flow can be illustrated as follows, and let us explore the process of configuring the Azure DevOps pipeline for the smooth Content Migration across environments.

Configuring the Azure DevOps pipeline:

Stage – Build Content Package

We can include the base Content Template xml file discussed earlier as part of the solution. During the Azure Pipeline execution, the Root Item ID, Language, Package Name and rest of the required parameters can be read as pipeline parameters. Then these values can be substituted into the respective placeholders within the source xml file.

Subsequently, we are required to upload the source xml file to the CM instance from where the packages will be generated. Once uploaded, the next step involves invoking the remote package creation script.

The following pipline yaml snippet shows these tasks,

        steps:
        - checkout: self
          persistCredentials: true
        - task: PowerShell@2
          name: Rename_XML_Template
          inputs:
            targetType: 'inline'
            script: |
                Rename-Item -Path 'Content_Package_Template.xml.template' -NewName 'Content_Package_Template.xml'
        - task: PowerShell@2
          name: Print_Param_Values
          inputs:
            targetType: 'inline'
            script: |
                Write-Host "rootItemId '$(rootItemId)'"
                Write-Host "rootItemId '$(languages)'"
                Write-Host "rootItemId '$(packageName)'"
        - task: ReplaceInFilesTextByText@2
          displayName: 'Set Root Item ID'
          inputs:
            parameterSearchDirectory: '$(Build.SourcesDirectory)'
            parameterSearchText: '$(Root_Item_ID)'
            parameterReplaceText: '$(rootItemId)'
            parameterTypeOfSearch: 'filesSearchPattern'
            parameterFilesPattern: '*.xml'
        - task: ReplaceInFilesTextByText@2
          displayName: 'Set PackageName'
          inputs:
            parameterSearchDirectory: '$(Build.SourcesDirectory)'
            parameterSearchText: '$(Package_Name)'
            parameterReplaceText: '$(packageName)'
            parameterTypeOfSearch: 'filesSearchPattern'
            parameterFilesPattern: '*.xml'
        - task: ReplaceInFilesTextByText@2
          displayName: 'Set Languages'
          inputs:
            parameterSearchDirectory: '$(Build.SourcesDirectory)'
            parameterSearchText: '$(Language_Sources)'
            parameterReplaceText: '$(languages)'
            parameterTypeOfSearch: 'filesSearchPattern'
            parameterFilesPattern: '*.xml'
        - task: PowerShell@2
          name: Upload_Base_XML_RLT
          inputs:
            filePath: '$(System.DefaultWorkingDirectory)/scripts/remote-PSE/upload-remote.ps1'
            arguments: >
                -remoteCMUrl '$(Source_CM)'
                -username '$(SPS_Remote_User_Name)'
                -password '$(SPS_Remote_User_Password)'
                -sourceFilePath '$(System.DefaultWorkingDirectory)\Content_Package_Template.xml'
                -destinationFilePath 'Content_Package_Template.xml'
        - task: PowerShell@2
          name: Create_Content_Package
          inputs:
            filePath: '$(System.DefaultWorkingDirectory)/scripts/remote-PSE/create-package.ps1'
            arguments: >
                -remoteCMUrl '$(Source_CM)'
                -username '$(SPS_Remote_User_Name)'
                -password '$(SPS_Remote_User_Password)'
                -baseXmlFileName 'Content_Package_Template.xml'
                -packageName 'Content_Package_Template.zip'

The /upload-remote.ps1 is depicted below and can be downloaded from the gist.

Stage – Upload and Install Content Package – Destination Environment

This pipeline stage is responsible to download the newly created Sitecore Content package from the source CM instance. Subsequently, upload it to the destination CM instance and invoking the Install Package remote script.

The following ymal snippet shows these tasks,

        steps:
        - checkout: self
          persistCredentials: true
        - task: PowerShell@2
          name: Download_Content_Package_From_Source
          inputs:
            filePath: '$(System.DefaultWorkingDirectory)/scripts/remote-PSE/download-package.ps1'
            arguments: >
                -remoteCMUrl '$(Source_CM)'
                -username '$(SPS_Remote_User_Name)'
                -password '$(SPS_Remote_User_Password)'
                -sourcePackagePath 'Content_Package_Template.zip'
                -destinationLocalFolder '$(System.DefaultWorkingDirectory)/downloads/Content_Package_Template.zip'
        - task: PowerShell@2
          name: Upload_Content_Package_Destination
          inputs:
            filePath: '$(System.DefaultWorkingDirectory)/scripts/remote-PSE/upload-remote.ps1'
            arguments: >
                -remoteCMUrl '$(Destination_CM)'
                -username '$(SPS_Remote_User_Name)'
                -password '$(SPS_Remote_User_Password)'
                -sourceFilePath '$(System.DefaultWorkingDirectory)/downloads/Content_Package_Template.zip'
                -destinationFilePath 'Content_Package_Template.zip'
        - task: PowerShell@2
          name: Install_Content_Package_Destination
          inputs:
            filePath: '$(System.DefaultWorkingDirectory)/scripts/remote-PSE/install-package.ps1'
            arguments: >
                -remoteCMUrl '$(Destination_CM)'
                -username '$(SPS_Remote_User_Name)'
                -password '$(SPS_Remote_User_Password)'
                -packageName 'Content_Package_Template.zip'

The install-package.ps1 script is depicted below and can be downloaded from the gist.

In this way, you can add as many stages as needed in the Azure Pipeline , allowing you to transfer and install the source content packages across various environments as needed.

Here are potential issues you might encounter:

Root Item does not exists

If you encounter the below exception while creating a dynamic package with SPE, please ensure that the user ‘sitecore/speremote‘ has read access to the content tree. If the user lacks the necessary access, please grant the appropriate permissions, as the SPE Remote script will be executed in this user context.

401 Forbidden

If you encounter a 401 Forbidden error while the initialization of remote session via Invoke-RemoteScript -Session $session, please ensure the remote user is created in the CM instance and granted to access specific services such as remote, fileUpload, fileDownload as defined in the patch file.

Key points to consider:

• The primary goal of this approach is to move the Deltas and required language changes to the targeted environment using the same approach that Sitecore Dynamic Package uses to mitigate the Timeout/Memory issues with SCS while serialising larger amount of items. If the amount of items are less and not to consider the Deltas, then we can still consider to use SCS to automate the Content Migration and it is worth reading a blog from my friend and colleague Guido on this.
• Despite attempting to allocate additional memory to the CM instance, the badgateway exception was not able to be resolved
• In the context of SCS, there is no direct command or function that allows for selective serialization of deltas, along with the ability to filter by language version. The serialization process in SCS typically involves serializing the entire item rather than just the changes including all language versions
• To track and serialize deltas specifically, we would need to implement custom logic with SCS APIs. We can avoid these overhead by following the approach with SPE Remote package creation
• In a Sitecore CM instance, remote script executions are disabled by default. To enable the CM to handle remote script executions, it is necessary to setup the required authorizations within the CM. Additionally, it is feasible to implement supplementary security measures at Infra level to avoid potential risk of unauthorized script executions

Information from Sitecore support ticket created on the SCS issue [ update on 4th Jul 2023]:

• The execution of ‘ser pull’ command on larger root item makes the memory to grow exponentially. However, once the operation is finished, the memory usuage returns to normal. This indicates that there are no persistent memory leaks present in the system
• The main cause of the excessive memory usage is due to the larger number of items, each with multiple language versions, getting serialized simultaneously. Upon examining the memory dump, it is evident that a significant portion of the objects are retaining the items slated for serialization
• Taking these points into considerations, they confirmed the workaround would be to divide the modules JSON into multiple modules with individual include rules ( which is discussed in the Problem section of this article)
• Furthermore, they have also added this task into their Product backlog, to ensure that the implemenation of the serialization functionality is reviewed and to check the possibility of avoiding the similtanous retention of all objects being serialized. In addition, they will be revising the current documentation to include this limitation