Situation
Modernising Data Management for Improved Healthcare Insights
NHS Digital, the provider of information, data, and IT systems for health and social care in England, faces immense challenges in managing patient and healthcare data. Their Data Processing Service (DPS) is responsible for transforming the way data is collected, processed, and used to improve health and care. A key strategic objective is to enhance access to rich datasets via the Data Access Environment (DAE), a vital national resource for secure data access. The legacy SQL-based monolithic platform was unable to scale effectively and impacted processing performance. Servita was engaged to re-platform the DAE and build data pipelines for onboarding key national datasets.
Solution
Cutting-Edge Technologies and Upskilling the NHS Digital Workforce
Servita provided a team of advanced data engineers, skilled in cutting-edge design, data analytics, and development technologies. They worked closely with NHS Digital teams, aiming to upskill the workforce in new approaches and technologies. Key initiatives included:
- Highly collaborative ways of working
- Building a knowledge base using Jira-Confluence
- Leading daily stand-ups, retrospectives, and sprint processes
- Conducting drop-in sessions for learning in advanced Python, PySpark, and Databricks
- Representing NHS Digital to customers, including NHS England, for user adoption of new technologies
The state-of-the-art solution, built in Python and Databricks on Amazon Web Services (AWS), greatly surpasses the legacy platform’s processing capability and is highly scalable. Servita also onboarded the main datasets, including Community Services and Improving Access to Psychological Therapies, before handing over the ongoing dataset onboarding activity to the incumbent NHS Digital team.
Results
Unprecedented Data Insights and Cost-Efficient Cloud-Based Solution
The transformed DAE enables NHS Digital to apply AI algorithms to vast amounts of data, providing users with unparalleled data insights. This significantly improves patient health and care while enhancing the nation’s ability to research and diagnose life science challenges. The cloud-hosted solution is cost-effective, and the NHS Digital workforce is now competent in Python and complex dataset analysis using Databricks.
Servita’s work has further solidified their reputation for data engineering and delivery excellence within NHS Digital and NHS England, driving healthcare innovation and outcomes.