Data Migration – BioGrid helping research programs grow

However, from time to time, clients require their database to be updated or upgraded to newer and improved technology offerings to enable better functionality, features and security. When the update or upgrade involves transitioning the database to newer improved technology, data migration is required.

There are specific data migration considerations relating to clinical data sets where consent for collection and use is required and needs to be accurately maintained across old and new systems.

“Occasionally, a client database upgrade includes changes that impact what participants have consented to previously. In this situation, participants may need to be re-consented by the client which impacts the data migration strategy” says Javier Haurat, Head, Data Science and Registries for BioGrid Australia.

It’s important in these situations to ensure participant consents are carefully and accurately reflected in the destination database design. Sometimes a client will expand a destination database so that as well as migrating data from the old system, they can add in data from other systems or start collecting new data. This makes maintaining relevant participant consents even more complex, so it’s critical to get it right.

“Going through a re-consenting process can have an impact on the time it takes for a migration, because you can’t migrate every participant in one process. We must plan very carefully for this and carry out periodic migrations as re-consent is obtained” says Javier.

Databases that have existed for a long period may also contain participants who are deceased, or who consented at a site that no longer exists, and it’s not possible to obtain consent for use in a new system. BioGrid’s team are well used to providing the extra care needed to handle these special cases.

Well before a migration happens, the BioGrid team spends time with the research team to flesh out the requirements.

“My team works closely with the research team for a significant period of time – it could take months - to review their requirements for a migration,” says Wisam Abdelaziz, Head, Infrastructure & Data Engineering for BioGrid Australia. “We’ll create test environments and do the migration first on the BioGrid servers, where it’s reviewed by our team and Javier’s team. When we have it working as it should, we bring in the client to check it to make sure it’s meeting their expectations.”

It sounds straightforward, but the devil is in the detail.

“The overarching process is straightforward, but the nuance comes in where there are structural and option changes that occur between the origin and the destination system,” says Javier. “There might be interpretation that needs to occur within both systems; patients to be excluded, items to be deleted. There needs to be a clean-up curation process in between and that’s where it can get quite complicated.”

When BioGrid conducted a cancer cohort migration, the way the data was stored in the origin database, created in MS Access, was very different to the way it was going to be stored in the new database in REDCap. The researchers wanted to combine treatments into overarching treatment modalities like immunotherapy, chemotherapy, radiotherapy, and so on, which required a significant degree of curation and processing.

Errors in the origin data can mean that the process takes much longer than anticipated, but it’s worth the time and effort because it means the migrated data is clean and accurate, which is far more useful for research in the long run.

BioGrid recently completed the first stage of a national Sarcoma database migration, moving data from an older system into REDCap. Funding considerations shaped the choice of destination system, and translating the requirements into the new database design gave the client an opportunity to review their existing data collection and decide what to keep, refine or remove before migration began.

Due to the volume of data, and because the site continued entering data into the live system throughout the project, migration is being carried out site by site – the first site completed was Peter MacCallum Cancer Centre, with around 5,500 individuals’ records migrated in full.

One of the challenges was that the two systems handled clinical relationships differently. Sarcoma patients can have multiple tumour occurrences, and each occurrence can be linked to multiple treatments and treatment types – a structure that isn’t natively supported in REDCap.

BioGrid’s team engineered a custom solution within REDCap to preserve these clinical associations accurately in the new database. Clinical classifications had also changed over time: the WHO classification of bone and soft tissue tumours had been updated since the original database was built, so historical entries needed to be carefully mapped against the current classification, and sets of individual yes/no fields in the old system were consolidated into single checklist fields in the new one.

Edge cases required particularly careful handling. In one instance, a case originally diagnosed as benign had since metastasised, which the new data structure hadn’t been designed to allow for. Resolving cases like this involved close, iterative work with the client’s clinical lead to flag cases, confirm how they should be coded, and correct the source data before migration – with extensive testing throughout the curation phase helping catch and resolve issues early. Not all historical data was carried across either; where a data type was no longer needed, the client made the call to leave it out, which kept the new database focused but added some complexity to migration testing.

The change in database structure also changed how the client analyses their data. Moving from the old system’s structure to REDCap’s patient- and episode-based tables, with separate linked tables for multi-select fields, meant the client’s analyst needed to adapt their analytical approach. BioGrid supported this transition directly – a reminder that its role in a migration project doesn’t end when the data lands in its new home.

The key to success with any migration, according to BioGrid, is healthy communication with the client. BioGrid’s team meets regularly with all project teams during both the preparatory and migration phases, ensuring that the researchers are kept well informed about progress and consulted at all decision points.

“There's always a back and forth between us and the client to keep them informed of how we're going. We take them through the data mappings. We show them the issues we’ve identified and how we’re planning to resolve them,” says Javier. “We want them to understand what we’re doing and confirm for us that it meets their expectations.”

The long-term relationships BioGrid has with many clients confirms the robustness of the data migration consultation process and the soundness of the end result, with clients returning to BioGrid as their projects expand and develop. With BioGrid’s help, they can ensure the integrity of their data as it moves into expanded programs.