Data Registry

The OCP Data Registry helps you find and access public procurement datasets that are available in the OCDS format. This documentation describes the open-source software itself.

Downloads are served by , thanks to its Fast Forward Program.

(If you are viewing this on GitHub, open the full documentation for additional details.)

How it works

This project is made up of two apps:

• data_registry: Serves the website, metadata API and admin site, and performs orchestration.

• exporter: Creates the bulk downloads in JSON, Excel and CSV format.

The complex part of the project is the orchestration. The tasks to orchestrate are:

• Collect data, via Scrapyd, used as the interface to Kingfisher Collect

• Pre-process data, via Kingfisher Process

• Export JSON files, via the exporter worker

• Calculate coverage, via the coverage worker, using Cardinal

• Export Excel and CSV files, via the flattener worker, using Flatterer

Each task is implemented as a TaskManager under the data_registry/process_manager/task directory. The JOB_TASKS_PLAN setting controls the order and choice of tasks.

The most relevant logic is:

• the manageprocess command, which calls…

• the data_registry.process_manager.process() function with each publication, which calls…

• the data_registry.models.Collection.is_out_of_date() method to decide whether to start a job.

Word choice

“collection” has a different meaning in this project’s code than in Kingfisher Collect or Kingfisher Process. It should be “publication”, as used in the UI and documentation.

Contents:

• Administrator’s guide
  • Site administrators
  • System administrators
• Command-line interface
  • Commands
  • Workers
• Reference
  • Environment variables
• Contributing
  • Setup
  • Development
• API reference
  • Process manager
  • Process manager utilities
  • Exporter utilities
  • Models
  • Template tags
  • Exceptions