Prymat

Introduction

Prymat, a leading food manufacturer in Europe, uses Etlworks to move and synchronize large volumes of data between on-premise SQL Server databases and Google BigQuery. By leveraging automatic partitioning and reverse ETL, Prymat has streamlined analytics and operational reporting across its data stack.

The Challenge

Prymat needed a solution to efficiently manage high-volume data movement between legacy and cloud systems:

• Large Source Tables: Key operational tables in SQL Server contained millions of rows and grew continuously.

• Slow Data Loads: Traditional ETL tools struggled to ingest and transform these tables within acceptable windows.

• Reverse ETL Needs: Required bidirectional flows to push processed data from BigQuery back into operational systems.

• Scalability and Maintenance: Needed a low-maintenance solution that could scale with growing data volumes.

Why Etlworks

Prymat selected Etlworks for its built-in support for parallel data extraction and tight integration with both SQL Server and BigQuery:

• Automatic Partitioning: Etlworks detects physical or logical partitions and distributes processing across parallel threads.

• Optimized SQL Server Integration: High-performance extraction using native drivers and custom queries.

• Native BigQuery Support: Seamless loading and querying with full control over datasets, partitions, and formats.

• Reverse ETL: Easily pushes curated data from BigQuery back to SQL Server and other operational systems.

• Monitoring and Scheduling: Reliable job orchestration and alerting with no need for external schedulers or scripts.

The Solution

Etlworks implemented a scalable integration framework tailored to Prymat’s needs:

• Partitioned Extraction: Large SQL Server tables are split into chunks based on date ranges, IDs, or partitions. Each chunk is processed in parallel for faster throughput.

• BigQuery Loader: Transformed data is loaded into partitioned tables in BigQuery for downstream analytics.

• Reverse ETL Flows: Selected BigQuery datasets are synced back to SQL Server to support operational reporting.

• Fully Automated Pipelines: All workflows are scheduled, monitored, and logged with error handling and recovery built in.

Results

• Faster Processing: Partitioning reduced ETL execution time by over 80% for large tables.

• Scalable Architecture: System handles growing data volumes with no manual reconfiguration.

• Improved Data Availability: Near real-time updates between SQL Server and BigQuery.

• Operational Efficiency: Reduced reliance on manual batch jobs and scripting.

Key Takeaways

• Automatic Partitioning: Etlworks distributes heavy ETL loads across partitions for faster, parallel processing.

• BigQuery Integration: Native connectors enable fast, schema-aware data loading and querying.

• Reverse ETL: Bi-directional flows keep operational and analytical systems in sync.

• Enterprise-Grade Reliability: Scalable, monitored, and fully automated ETL pipelines.

Ready to tackle your most complex data challenges? Discover how Etlworks can transform your data integration workflows. Start your free trial today or request a demo.