The Scoular Company

Introduction

The Scoular Company is a global agribusiness firm headquartered in Omaha, Nebraska. The company operates a complex technology ecosystem that includes an IBM i (AS400) system supporting mission-critical operational data. As Scoular expanded its cloud analytics capabilities on Microsoft Azure, the company needed a reliable way to replicate AS400 data to modern data platforms in real time.

Using Etlworks, Scoular implemented a high-performance change data capture (CDC) pipeline that streams data from AS400 to multiple Azure-based destinations, including Azure SQL Server, Azure Synapse Analytics, and Microsoft Fabric Warehouse. The solution captures both transactional data and detailed metadata from AS400 journals while supporting strict operational requirements such as system maintenance windows and high transaction volumes.

The Challenge

Scoular needed a robust data integration platform capable of modernizing access to data stored on IBM i while maintaining operational reliability.

Key challenges included:

• Replicating high-volume transactional data from AS400 to multiple cloud platforms in near real time
• Capturing not only data changes but also journal metadata associated with each record 
• Supporting multiple Azure destinations used for analytics and operational reporting 
• Handling scheduled AS400 maintenance windows without risking data loss or pipeline failure 
• Building a fully automated deployment and operations process around CDC pipelines 
• Ensuring reliable performance under high transaction volume

Several CDC tools evaluated by the Scoular team were able to capture database changes but could not replicate the associated journal metadata, which was required for downstream processing and auditing.

Why Etlworks

After evaluating multiple data integration platforms, Scoular selected Etlworks for its flexibility and deep support for IBM i environments.

Key capabilities included:

• Native CDC support for IBM i (AS400) using transaction logs and journals 
• Ability to capture both record changes and journal metadata for each transaction 
• High-performance replication suitable for large transaction volumes 
• Support for multiple cloud destinations within the same CDC pipeline 
• Built-in automation capabilities to manage operational workflows and maintenance windows 
• Flexible architecture that integrates easily with Azure data platforms

These capabilities allowed Scoular to implement a CDC solution tailored to their operational requirements while maintaining reliability and scalability.

The Solution

Scoular implemented a real-time data integration architecture using Etlworks to stream changes from AS400 to multiple Azure data platforms.

The CDC pipelines capture changes from AS400 journals and replicate both the data and the associated metadata to downstream systems. The data is currently delivered to:

• Managed SQL Server on Azure Azure 
• Synapse Analytics 
• Microsoft Fabric Warehouse

This architecture enables multiple analytics and operational systems to consume near real-time data from the AS400 environment.

Etlworks also enabled the Scoular team to automate operational workflows around the IBM i maintenance schedule. CDC pipelines automatically stop before scheduled maintenance windows and restart afterward without losing data consistency. This automation ensures uninterrupted operations while respecting AS400 system maintenance requirements.

The platform was also designed to support future expansion. Scoular plans to extend the CDC architecture to additional destinations, including Azure Service Bus, enabling event-driven data processing and integration with additional cloud services.

Key Takeaways

• Real-time CDC from IBM i (AS400) to modern Azure data platforms 
• Replication of both data and journal metadata for each transaction 
• Integration with Azure SQL Server, Azure Synapse Analytics, and Microsoft Fabric Warehouse 
• Automated handling of AS400 maintenance windows 
• rchitecture designed to support high transaction volume 
• Future expansion planned to Azure Service Bus for event-driven integrations