High-Volume Data Platform Processing Millions of Files per Day

This case study describes prior operating experience that informs Lirado Tech’s current practice. Lirado Tech was not the original contracting entity.

Business Context

A growing SaaS data platform processed high volumes of compliance-sensitive business documents represented as XML files, while supporting customer integrations, analytics, reporting, and high-volume historical data onboarding.

The existing system had been designed around earlier-stage assumptions and was too dependent on a central database-bound application model.

Platform Bottleneck

The core bottleneck was architectural scalability. The platform needed to move from a centralized, database-bound design toward a decoupled event-driven architecture capable of handling higher ingestion and processing volumes at the millions-of-files-per-day level.

Why It Was Difficult

High-volume ingestion systems require operability, replay, governance, and cost control—not throughput alone. The platform needed to maintain idempotency, partitioning, and operational visibility while scaling to millions of files per day.

Role and Responsibility

Led the architecture and implementation of an event-driven AWS data platform. Responsible for object-storage-based ingestion design, queue-driven processing, governed data architecture, idempotency and replay mechanisms, partitioning, and operational visibility.

Key Decisions

• Chose object storage as the ingestion boundary instead of direct database writes.
• Introduced queue-driven processing to decouple ingestion from transformation.
• Designed for idempotency and replay to support operational recovery and reprocessing.
• Prioritized governance and cost control alongside throughput targets.

Implementation

• Redesigned the platform around looser coupling and cloud-native components.
• Used object storage and event-driven patterns to reduce pressure on the central application and database.
• Introduced pre-signed upload flows, S3 events, Lambda processing, queue-based processing, and governed data architecture.
• Established idempotency, replay, partitioning, and operational visibility mechanisms.
• Built observability and operational foundations for production growth.

Results

• The production event-driven pipeline processed approximately 1M documents in 15–17 minutes and supported volumes of up to 10M XML files/day.
• High-volume historical onboarding moved from weeks or months to approximately 2–3 business days.
• The production platform reduced dependency on a central database-bound model.
• Achieved measurable cost control and operational governance alongside throughput growth.

What It Demonstrates

High-volume systems require operability, replay, governance, and cost control—not throughput alone. The architecture decisions around ingestion boundaries, processing decoupling, and operational visibility determine whether a platform can scale without accumulating unmanageable risk.

Relevant Review Variant

High-Volume Data Platform Review