Azure Pipelines and YAML Templates Optimize Global Operations

About IPG

Interpublic Group (IPG) is a global provider of marketing solutions that operates as a holding company for over 100 marketing, advertising, and public relations companies. With approximately 58,500 employees across all major world markets, they specialize in advertising, digital marketing, communications planning, media, public relations, and marketing.

About | IPG (interpublic.com)

Challenge

IPG operated a large deployment of BizTalk Server and SQL Server Integration Services (SSIS) as a first-tier critical application that included a redundant IT infrastructure for high availability as well as a separate IT infrastructure for disaster recovery.

A complex and dated deployment model
The company had operated a BizTalk Server deployment for more than a decade. As a result, the implementation had grown organically and evolved to over 100 BizTalk applications serving various financial, planning, operational execution, and human resources business processes around the world.

Source code was managed in an on-premises Visual Studio Team Foundation Server (TFS) using a repository in Team Foundation Version Control (TFVC). The BizTalk solutions were built to approximately 300 separate DLL assemblies and the deployment contained approximately 3,100 schemas, 650 maps, 450 pipelines, 125 role links, 550 parties, and 25 Business Rules Engine (BRE) policies.

Assessing requirements

TwoConnect consulted with IPG to determine the necessary product upgrade requirements while also ensuring compliance with Microsoft support.

The evaluation resulted in the following baseline deliverables:

• Moving physical servers to Azure virtualized servers and reducing the upkeep of extensive physical servers.
• Reducing costs by automating the change management of the 100s of BizTalk Applications with Azure DevOps pipelines used to build and deploy software assets into their BizTalk Server environments.
• Automating and improving IPG’s quality assurance practices which were currently conducted manually, thus eliminating cumbersome processes and streamlining operations when coordinating with global business units.

Another consideration for upgrading was the global community of BizTalk Server experts involved in building a shared set of tools in the BizTalk Server Deployment Framework (BTDF), which has made deployment automation often notoriously complicated.

Addressing future requirements

Over time, IPG planned to grow their suite of Azure integration tests to provide the capability to perform regression tests to make certain changes in one component did not break other BizTalk applications. This would be done by implementing unit tests during the build phase and executing integration tests after deployment.

An Azure solution

TwoConnect proposed infrastructure changes of physical servers to Azure virtual machines and addressed all of IPG’s change management and automation goals, which included:

• Infrastructure moves:
• From four physical BizTalk 2020 servers and a cluster of two SQL Server 2019 servers to Azure virtual machines in IPG’s on-premises VMware cluster servers.
• Impacted environments: Production and Disaster Recovery, QA, Pre-QA (Development)
• Azure DevOps upgrades:
• All source code was migrated to a Git repository in Azure DevOps and upgraded with Visual Studio to be compliant with BizTalk Server 2020.
• TwoConnect established a Git repository branching practice aligned with IPG change management and test automation goals.
• Impacted environments: Development, QA, and Master
• Process:
• Feature branches created by developers from Dev
• When achieve is code-complete and has passed testing requirements, merge to Dev branch
• Bug branches created from Dev
• When Dev achieves feature-complete and passed testing requirements, merge Dev to QA
• Bug branches created from QA
• When QA achieved release-complete and passed testing requirements, forward merge QA to Master
• Backward merge QA to dev to pick up any bug fixes

Custom Azure pipelines in YAML

TwoConnect also implemented DevOps automation with custom Azure pipelines using YAML templates to reduce redundancy and streamline efficiencies.

Resources in YAML were used to represent IPG’s sources of pipelines, builds, repositories, containers, etc., as needed. They also provided traceability of services used in the pipelines including information about the version, artifacts, associated commits, and work items. Once resources were defined, they could be consumed anywhere in the pipeline, making the Azure DevOps workflow fully automated by subscribing to trigger events on the resources.

The process included:

• IPG required Builds per BizTalk Applications of which there were over 100
• YAML File per Application referenced common template YAML
• Per Application YAML set core variables that identified the target BizTalk Application name as well as the Visual Studio Solution file name
• YAML referenced Azure DevOps Variable Groups named per Environment and Application and contained environment specific settings for BizTalk configurations
• Azure pipeline triggered by changes committed to branches dedicated for deployment to each environment, including:
• Dev branch to Pre-QA (Development) environment
• QA branch to QA environment
• Master branch to Production environment
• Test Automation Including:
• Database for storing and managing test cases implemented
• Stored input data and files
• Stored copies of expected outcome data and file for comparison validations of test outcome
• Unit Test for Maps and Pipelines implemented using xUnit Test Framework
• Executed by build stage of Azure DevOps pipeline
• Integration Tests implemented using xUnit Test Framework
• Executed after release stage of Azure DevOps pipeline completed deployment of BizTalk Server assets

The result

TwoConnect’s award-winning BizTalk and Azure integration team focused exclusively on building solutions to complete IPG’s upgrade of their BizTalk platform and unify communications, collaboration, and integration by moving physical servers to virtual machines in Azure and reducing costs and cumbersome processes.

TwoConnect implemented guidelines for supporting, maintaining, and adapting all their integration solutions to ensure the seamless continuity of business operations going forward. By implementing YAML templates and Azure solutions, IPG’s code became reusable, simplifying the sharing and use of code across multiple pipelines.

A core part of this solution architecture was achieved through Azure DevOps, which accomplishes more efficient continuity across people, processes, and technology, resulting in a faster means for bringing products to customers using continuous delivery and continuous integration (CI/CD) patterns and practices.

Future needs

It was also critical to IPG that their system be able to support ongoing growth opportunities and their plans for an expanded worldwide presence in the market. Needs assessments were performed to identify requirements for ongoing system effectiveness throughout extended future growth.

TwoConnect continues to provide guidance to IPG on the configuration of dependent software including the Operating System, SQL Server, and Azure Environment segmentation, where applicable.

‍