If your company hasn't already gone through a major data migration, you will at some point.
We recently finalized migrating the main customer relationship platform for a large retail client in the US. Here’s some lessons that we learned from this project and general practices that we follow at KIS for large migration projects.
1. There is no substitute for reverse engineering. There is rarely a way to get around the hard work.
2. You shouldn’t expect to get a full and complete accounting of all the business rules and logic from your partners and subject matter experts.
3. Systems that run for 10+ years grow organically over time and become complex and hard to understand. It’s unlikely anyone in the organization really knows how the system works.
4. Interfaces can hide migration complexity from dependent systems. Often, many other systems rely on the legacy system. Whenever possible, try to use existing interfaces so that any dependent systems are “shielded” from updates.
5. Preserve the legacy system’s API message format(s) so that API consumers can migrate seamlessly.
6. You may not get to fix everything you want to in the beginning, but that’s okay. Fixing things in phases removes dependencies and decreases complexity.
7. More data makes migration more challenging. Multi-billion record legacy databases require complex coordination for massive data migration.
While this project taught us some specific lessons, over the years, we have also developed a series of best practices. These help us ensure a smooth transition for any data migration project, no matter the size or complexity.
We typically plan for at least three types of dry runs, or practice runs. We thoroughly test the migration to find any and all potential issues. We continue dry runs until we are absolutely sure that the migration process will work, run in reasonable time, and move all the data.
We try to get as much of the new infrastructure in “production” as early as possible. To do this, we run the new system in parallel with the legacy system in soft production, meaning the new system is running against actual data without anyone consuming that data.
Next, we validate our new infrastructure using the soft production environment prior to migrating over any production data. During this time, we also run data comparison reports between the two systems until we are confident both platforms agree with each other.
Finally, we plan for a phased cutover that breaks the data into small pieces that are transitioned one section at a time.
When moving very old systems (10+ years), there is just too much risk in turning it off all at once. We start by picking some of the lower priority and less risky processes or data consumers and migrate them first. We then monitor how they behave on the new system. We gradually move more processes and dependencies from the legacy system to the new system until everything is migrated. This approach allows for a fallback. If we move over a certain process and something goes wrong, we can move it back to the legacy system until we resolve the issues on the new platform.
We also make a data validation plan and post-cutover monitoring plan to confirm that the new system aligns with the legacy system. This requires considerable time writing data validation queries and reports and engaging with business partners on key metrics, but it’s worth it. We continue to run data validation/comparison reports and share the results with business partners until everyone agrees that the two systems align on critical data metrics.
Your first runs of production on your system might have many flaws, and you will probably make a few mistakes. While you can’t always prevent these unexpected issues, you can still prepare for them. You should build a post-production monitoring system.
Each data migration comes with its unique problem, but with these lessons and practices, you can meet the challenge head-on. Remember: every migration is an opportunity to learn and grow.
.png)
