Dataset Quality Assurance Procedures
- Michael Rans
- Javier Teran
Introduction
This document presents the current procedures undertaken by the HDX Data team to verify that datasets that are hosted on the Humanitarian Data Exchange (HDX) meet with the minimum quality standards. There are two kinds of datasets on the repository, those created by the community, and those that are created by the HDX team as part of the curation process. This document outlines the quality assurance checks that are to being carried out for each of these two types of datasets. These procedures are conducted twice daily ( morning and evening) on new datasets or existing datasets on HDX that have been changed.
The dataset quality assurance procedures are presented below. These procedures are based on a set of potential dataset quality issues that have been identified. For each potential quality issue, the processes for checking for the existence of the issue, the timing of the checks as well as the corrective action to be taken on datasets that fail to meet the quality standard are listed.
Four tables are presented below, one table for quality issues that are common to both community generated and curated datasets, the second table for dataset quality issues unique to community generated datasets, the third table for dataset quality issues unique to curated datasets and the fourth table lists the potential data quality issues that are checked as part of the HDX Quality Assurance Framework (QAF).
A mindmap graphic visualizes these issues:
Table 1. Common Dataset Quality Assurance Procedures
ID | Potential Quality Issue | When to check | How to check | Corrective action |
101 | Unrelated item in related items tab | -When a related item is added | Using a script, identify datasets with a related item and evaluate the related item for relevance | Delete related item Engage responsible user |
102 | Dataset metadata missing or incomplete | -When dataset is made public -When dataset is revised -During routine data QA process | Manual or scripted evaluation of the dataset | Engage org admin |
ID | Potential Quality Issue | When to check | How to check | Corrective action |
201 | Dataset has no resources (files or links) | -When dataset is made public -During routine data QA process | Manual or scripted count of the number of resources in the dataset | Make dataset private Engage org admin |
202 | Dataset has a broken resource link | -When dataset is made public -During routine data QA process | Manual or scripted check of resource link | Engage org admin Check for 201 |
203 | Dataset contains no relevant humanitarian data | -When dataset is made public | Manual evaluation of the data | Make dataset private Engage org admin |
204 | Dataset contains test data | -When dataset is made public | Manual evaluation of the data | Make dataset private Engage org admin |
205 | Dataset contains sensitive data (PII, DII, CII) | -When dataset is made public | Manual evaluation of the data | Make dataset private/ Remove dataset from platform Engage org admin Refer for management review |
206 | Dataset contains inappropriate or otherwise objectionable content | -When dataset is made public | Manual evaluation of the data | Make dataset private Revoke user editing privileges Refer for management review Engage org admin |
| 207 | Dataset contains the COD tag | -When dataset is made public | Manual or scripted check for the COD tag in the data | Remove the tag Engage the data provider |
| 208 | Dataset contains individual survey data | -When dataset is made public | Manual evaluation of the data, special check for PII, DII or CII Look for data dictionary | Make dataset private Engage the data provider |
| 209 | Dataset contains a PDF resource that is not considered metadata | -When dataset is made public | Manual or scripted check for PDF document | Remove PDF or make dataset private if dataset no longer viable Engage data provider |
| 210 | Dataset source is non existent or unclear | -When dataset is made public | Manual evaluation of the dataset | Make private Engage data provider |