HDX to Frictionless

Owned by Michael Rans
Last updated: 16 Nov 2017

The motivation for this project is to enable taking advantage of third party libraries that have been developed to help with tasks like data validation. These libraries were created by the Open Knowledge Foundation and are built to support a data standard called Frictionless Data. If that standard becomes commonplace then it will be an additional benefit if HDX supports it.

Uptake of Frictionless

Work has been done on assessing the uptake of the Frictionless standard including looking at case studies outlined here: Implementers of Frictionless and monitoring the interest in Frictionless by looking at GitHub Repo Uptake Statistics. These statistics tell us that there seems to be a steady increase in interest in Frictionless tooling which implies a growing adoption of the standard. 

Frictionless Technologies

For now, the main interest is in reusing the technologies underpinning Frictionless. That list has expanded significantly between June and November 2017 with libraries added for all major programming languages. Ones we might use or borrow code from are:

Data Curator

Desktop CSV editor to help describe, validate and share usable open data. 

goodtables-py

Validate and process tabular data in Python.

Stenci.la coming soon

The office suite for reproducible research

Import for Google Spreadsheets experimental

Import Tabular Data Packages into Google Spreadsheets.

Data Package Pipelines

Framework for processing data packages in pipelines of modular components.

datapackage-py/js/...

A library for working with Data Packages.

tableschema-py/js/...

A library for working with Table Schema.

tabulator-py

Consistent interface for stream reading and writing tabular data (csv/xls/json/etc). 

Where to use Frictionless

There are a few areas where Frictionless can be used.

HDX Utilities library

Tabulator-py is already in use in the HDX Utilities library and through that in the HDX Python API for uploading to the HDX datastore and also in the Chatham House project.

HXL Proxy

Tabulator-py could also be used in the HXL Proxy to replace the stream reading code, the advantages including the consequent reduction in the amount of code to be maintained and that improvements coded by others will automatically be available to the HXL Proxy (for example support for zipped csv). The main disadvantages are the time needed to refactor the HXL Proxy to use it and to identify any missing features needed.

Migration Tool for Organisations

Import for Google Spreadsheets could be used to enable organisations to easily move from local Excel spreadsheets to Google Spreadsheets in which we can embed a trigger to determine if the data has changed for freshness purposes.

HDX UI

datapackage-js could be used to enable the export of HDX datasets as Frictionless data packages should the standard take off.

Data Check

goodtables-py, Data Curator and Stenci.la (looks like a cross between Word and Pandas) could provide code and ideas for this tool. This would be the most significant of the areas presented here for where Frictionless could be used in HDX and shapes how much effort should be put into further prototyping. The decision that needs to be made is whether to make improvements to the HXL proxy or to use and contribute to Frictionless libraries to make them either least minimally HXL-aware if used alongside the HXL Proxy or fully aware if used as a replacement.

Advantages to using HXL Proxy

  • Familiarity
  • In house knowledge
  • Speed to get going

Advantages to using Frictionless

  • Less in house code to maintain
  • More contributors to the codebase
  • Access to new tools/libraries

The HXL Proxy already has some validation capabilities built in so the question is whether what it has is already sufficient and if not, whether Frictionless offers significantly more such that it is worth the effort to switch rather than just build on the HXL Proxy. Using a minimally HXL-aware Frictionless for the non HXL-specific validation such as checking types with the HXL Proxy for validation specific to HXL eg. against vocabularies may be a good solution.

HDX to Frictionless Prototype

A prototype has been developed that takes an HDX dataset name and from it produces a Frictionless datapackage. Initially it was designed to convert from HXL to Frictionless but it became apparent that the data package can include metadata unavailable to HXLated resources - metadata contained within the HDX dataset itself.