Two years of the Health Data Research Innovation Gateway – what have we learned and what does the future hold?
27 May 2022 | Author: Ruth Milne, Communications Manager, Infrastructure and Services
This week we are celebrating two years of the Health Data Research Innovation Gateway – the UK’s only unified platform to search, discover and request access to health data sets for research and innovation. In this post, we reflect on our key learnings so far and offer a snapshot of the road ahead.
The Health Data Research Innovation Gateway (the Gateway) was created to provide a common entry point for researchers to search, discover and request access to health datasets in the UK.
From enhancing search capabilities to supporting COVID-19 research efforts and – most recently – building a data use register, the Gateway has developed at an incredible pace since its first release in early 2020. To date, over 750 datasets from 60+ data custodians have been made available on the Gateway, and more than 2000 people from across the health data community are registered users.
Learn more about the Gateway’s key milestones to date
So, in two years of rapid development and delivery, what have we learned?
Our two biggest lessons
1. Challenges in moving from ‘start-up’ to ‘production’
The Gateway started as a minimum viable product (MVP) where features were added rapidly despite the on-going pandemic and is now a production service platform that facilitates research rather than a research project itself. From the start, we have adopted an agile design and development approach and our delivery has always been open: we share our developed technology assets on GitHub and use open-source software ourselves wherever possible.
This is great, but what we are not doing enough of is working with our wider developer community to ensure the effective reuse of existing outputs and encourage the co-development of new Gateway functionalities. Not everything we need to do in the ecosystem need to be done by us.
We need to be better at documenting our codebase, making our development outputs and assets more visible, and collaborating with the community to achieve true open development, rather than an open-source only approach. This will drive improvements to open-source standards and ultimately a better Gateway experience for all.
2. Build the community and they will come
The Gateway is central to our organisational mission – to unite the UK’s health data to enable discoveries that improve people’s lives – by providing a unified platform to discover and request access to health data in the UK. From core concept to MVP to making over 750 health datasets more discoverable on the Gateway today, we have sought regular consultation from our stakeholders, partners, and the public.
Our initial success with rapidly building a multi-stakeholder product of immediate use and better than what currently existed for researchers and data custodians alike has, in part, also created a hinderance as multiple users and stakeholders started asking us to do more, faster and better rather than focusing on our core mission – changing the ecosystem rather than the product itself.
We also need to be more targeted in our engagements and feedback consultations. For example, making sure that researchers are embedded into our development process to help prioritise features is fundamental to provide the quality and scale of the Gateway.
Gateway users or not, we must engage so much more with academics, clinicians, scientists and other healthcare professionals to better understand the needs of the community when it comes to finding and accessing health data for research. Their experience is essential to improve how we refine our development priorities and deliver a research tool that’s truly valuable to all it serves.
“Our challenges over the next year are to further engage our research communities to deliver user centric enhancements to our search and request features, and to increase integration and self-service capabilities for our data custodians.”
– Elaine Brannigan, Chief Product Owner, HDR Innovation Gateway
Our immediate focus is improving the search functionality and integrating a federated metadata approach to onboarding datasets to the Gateway.
We recently updated the overall text search behaviour on the Gateway – shifting the code command from ‘or’ to ‘and’ – to improve the accuracy of search results containing multiple search items. We are now working to enhance the user interface of the search results page and following feedback from our usability testing we are prioritising the following: keywords and filters refinement, clearer advanced search options, improvements to sort and save, and a cleaner look and feel in general the page. This is moving into development shortly so watch this space!
Federated Metadata Automation
This feature goes to the heart of our mission to improve the ecosystem rather than the product and will enable data custodians more control of the metadata descriptions made available on the Gateway. Second order effects of this work are 1) making metadata equally important as the data for data custodians and 2) allow other ecosystem providers to leverage this resource to build advanced tools and services on top of and beyond those provided by the Gateway.
With our work on federated metadata automation about to be released, a data custodian will soon have the option to sync their metadata catalogue to the Gateway so that data descriptions are automatically pulled through when onboarding datasets, limiting their manual upload time. A change to the custodian catalogue updates the Gateway record too, so researchers have the most up to date information possible when searching for datasets published on the Gateway. We hope that metadata federation will enable us to quickly scale the number of datasets available on the Gateway.
Some of the other development items on the current 2022-2023 Gateway roadmap include:
- Integration of the ICODA Workbench to support the addition of global datasets to the Gateway
- Release of a Data Use Register widget so that data custodians can display data from the Gateway Data Use Register directly on their website
- Simplification of the Data Access Request set-up for data custodians, incorporating a bespoke and self-service approach using a bank of predefined questions
- Explore our development timeline for more!
We look forward to delivering this vision over the next 12 months – see the progress we’re making by reading our Release Notes (page updated monthly).
Celebrating two years of the Health Data Research Innovation Gateway
Two years of the Health Data Research Innovation Gateway
22 May 2022
Earlier this year, we recognised two years of the Health Data Research Innovation Gateway. Funded by UK Research and Innovation's (UKRI) Industrial Strategy Challenge Fund and delivered in...
Data Access and Discovery – March 2023
16 March 2023 at 11:00 am
Come along to our first Data Access and Discovery Webinar of 2023 as we explore some of the data services and expertise on offer from two Health Data Research Hubs: Discover-NOW and PIONEER. The...