Skip to end of metadata
Go to start of metadata

You are viewing an old version of this page. View the current version.

Compare with Current View Page History

« Previous Version 18 Next »

Purpose

We would like to be able to determine how fresh is the data on HDX for two purposes. Firstly, we want to be able to encourage data providers to make regular updates of their data where applicable, and secondly, we want to be able to tell users of HDX how up to date are the datasets in which they are interested.

Progress

It was determined that a new field was needed on resources in HDX. This field shows the last time the resource was updated and has been implemented and released to production HDX-4254 - Getting issue details... STATUS . Related to that is ongoing work to make the field visible in the UI HDX-4894 - Getting issue details... STATUS .

Critical to data freshness is having an indication of the update frequency of the dataset. Hence, it was proposed to make the data_update_frequency field mandatory instead of optional and change its name to make it sound less onerous by adding "expected" ie. dataset expected update frequency  HDX-4919 - Getting issue details... STATUS . It was confirmed that this field should stay at dataset level as our recommendation would be that if a dataset has resources with different update frequencies, it should be divided into multiple datasets. Assuming the field is a dropdown, it could have values: daily, weekly, fortnightly, monthly, quarterly, semiannually, annually, never. It would be good to have something pop up if the user chooses "never" making it clear that this is for datasets for which data is static. We will have to audit datasets where people pick this option as we don't want people choosing "never" because they don't want to commit to putting an expected update frequency. The expected update frequency requires further thought particularly on the issue of static datasets, following which there will be interface design and development effort. 

A trigger has been created for Google spreadsheets that will automatically update the resource last modified date when the spreadsheet is edited. This helps with monitoring the freshness of toplines and other resources held in Google spreadsheets and we can encourage data providers to use this where appropriate. Consideration has been given to doing something similar with Excel spreadsheets, but support issues could become burdensome.

Important fields


FieldDescriptionPurpose
data_update_frequencyDataset expected update frequencyShows how often the data is expected to be updated or at least checked to see if it needs updating
revision_last_updatedResource last modified dateIndicates the last time the resource was updated irrespective of whether it was a major or minor change
dataset_dateDataset dateThe date referred to by the data in the dataset. It changes when data for a new date comes to HDX so may not need to change for minor updates

Approach

  1. Determine the scope of our problem by calculating how many datasets are locally and externally hosted. Hopefully we can use the HDX to calculate this number.  
  2. Collect frequency of updates based on interns work? 
  3. Define the age of datasets by calculating: Today's date - last modified date 
  4. Compare age with frequency and define the logic: how do we define an outdated dataset

Determining if a Resource is Updated

The method of determining whether a resource is updated depends upon where the file is hosted. If it is in the file store, then the update time is clear,
If it is hosted externally, then it is not so simple. It may be possible to use HTTP get the last_modified field depending upon whether the server supports it or not.

Number of Files Locally and Externally Hosted

TypeNumber of ResourcesPercentageExample
File Store                                  2,102
22%

CPS                                  2,459
26%

HXL Proxy                                  2,584
27%

ScraperWiki                                     162
2%

Others                                  2,261
24%

Total                                  9,568
100%

Classifying the Age of Datasets


Thought has previously gone into classification of the age of datasets. Reviewing that work, the statuses used (up to date, due, overdue and delinquent) and formulae for determining those statuses is sound and so we will build on that foundation:


Update Frequency

Dataset age state thresholds

(how old must a dataset be for it to have this status)

Up-to-date

Due

Overdue

Delinquent

Daily

0 days old

1 day old

due_age = f

2 days old

overdue_age = f + 2

3 days old

delinquent_age = f + 3

Weekly

0 - 6 days old

7 days old

due_age = f

14 days old

overdue_age = f + 7

21 days old

delinquent_age = f + 14

Fortnightly

0 - 13 days old

14 days old

due_age = f

21 days old

overdue_age = f + 7

28 days old

delinquent_age = f + 14

Monthly

0 -29 days old

30 days old

due_age = f

44 days old

overdue_age = f + 14

60 days old

delinquent_age = f + 30

Quarterly

0 - 89 days old

90 days old

due_age = f

120 days old

overdue_age = f + 30

150 days old

delinquent_age = f + 60

Semiannually

0 - 179 days old

180 days old

due_age = f

210 days old

overdue_age = f + 30

240 days old

delinquent_age = f + 60

Annually

0 - 364 days old

365 days old

due_age = f

425 days old

overdue_age = f + 60

455 days old

delinquent_age = f + 90



Thoughts

There are two aspects of data freshness:
  1. Date of update: The last time the data was was looked at to confirm it is up to date ie. it must be examined according to the update frequency
  2. Date of data: The actual date of the data - an update could consist of just confirming that the data has not changed
We should send an automated mail reminder to data contributors if the update frequency time window is missed by a certain amount. Perhaps we should give the option for contributors to respond directly to that mail to say that data is unchanged so they don't even need to log into HDX in that case, otherwise provide the link to their dataset that needs updating.
The amount of datasets that are outside of HDX is growing. I think we should try to handle this situation now. The simple but perhaps annoying solution is to send a reminder to users according to the update frequency (irrespective of whether they have already updated as we cannot tell).
Another way to do so is to provide guidance to users so that as they consider how to upload resources, we steer them towards a particular technological solution that is helpful to us eg. Google spreadsheet with update trigger, document alerts in OneDrive for Business, macro in Excel spreadsheet. I don't know if this is possible, but complete automation would be if they could click something in HDX that creates a resource pointing to a spreadsheet in Google Drive with the trigger set up that opens automatically once they enter their Google credentials.

Actions

Update frequency needs to be mandatory:  HDX-4919 - Getting issue details... STATUS
Investigate http get last modification date field - 60% in HDX have this according to UofV.

References

Using the Update Frequency Metadata Field and Last_update CKAN field to Manage Dataset Freshness on HDX:

https://docs.google.com/document/d/1g8hAwxZoqageggtJAdkTKwQIGHUDSajNfj85JkkTpEU/edit#

Dataset Aging service:

https://docs.google.com/document/d/1wBHhCJvlnbCI1152Ytlnr0qiXZ2CwNGdmE1OiK7PLzo/edit

https://github.com/luiscape/hdx-monitor-ageing-service


University of Vienna paper on methodologies for estimating next change time for a resource based on previous update history:
University of Vienna presentation of data freshness:
  • No labels