Operation SwiftShift: Accelerating Data Migrations is a case study that explores how an elective medical software provider addressed the challenges of migrating customer data between two of its core offerings. The company originally offered a Legacy Product, which used a sharded MySQL database and was optimized for multiple segments, alongside a NextGen Product, a single-tenant database better suited for a particular segment. Customers from the segment but still using the Legacy Product needed a streamlined path to move over to the NextGen Product. However, the migration process presented several hurdles: the two products had incompatible database structures, there were large volumes of structured and unstructured data (including images), and manual workflows often stretched migrations over multiple months. The primary goal was to better serve these customers by drastically cutting migration time from months to days—or even aiming for two hours—while ensuring the process was predictable, consistent, and scalable.
To accomplish this, Quest1 carried out several strategic initiatives. First, it developed a DSL (Domain Specific Language) capable of reading and exporting data from the Legacy Product’s sharded MySQL databases, then converting it into standardized CSV files for the NextGen Product. These files, along with associated images and documents, would initially be staged in Amazon S3. The transformation stage took place in AWS Glue, where PySpark scripts cleansed and standardized the data to improve overall quality and consistency. AWS Step Functions oversaw the entire workflow, breaking it into manageable parts and integrating with AWS Lambda for tasks such as loading the transformed data into the NextGen Product’s database and preserving tenant relationships.
In addition, the team introduced a self-service portal specifically designed for the non-technical service teams. Built with an Angular front end and a .NET back end, the portal integrates with Amazon Cognito for authentication, enabling authorized users to manage migrations without extensive engineering support. Amazon DynamoDB stores configurations and tracks migration status, while Amazon EventBridge triggers event-driven updates, ensuring the service teams have real-time insights into the migration process. This automated ecosystem not only streamlined data conversion and loading but also created a replicable platform that could later accommodate migrations from other systems.
The impact of Operation SwiftShift proved significant. Migration times from the Legacy Product to the NextGen Product dropped from months to days, with further enhancements under way to achieve two-hour turnarounds. Standardizing the transformation logic led to a marked improvement in data consistency, reducing errors and overall cleanup efforts. The organization also noted an 80% decrease in technical team involvement, which translated into considerable operational savings. Within the first month after launch, over ten customers were successfully migrated, demonstrating the solution’s reliability and effectiveness. The initiative’s self-service nature not only catered to existing requirements but also set the stage for future integrations with additional electronic medical record platforms.
