Site administrators

Use the Django admin to:

  • Add and edit publications

    Note

    Once a new publication is added, the manageprocess command will collect its data, unless Frozen is checked.

  • Check the status of a job and its tasks

    Note

    The flattener task was added in September 2022, so earlier jobs have a Last completed task of “exporter (4/5)”.

  • Review log entries for other administrators’ actions

Publications and jobs can be searched by publication country and publication title.

Note

The search performs only case normalization. For example, “Montreal” will not match “Montréal” (with an accent).

Add a publication

Refer to our internal documentation (contains links to internal resources).

Review publications

From time to time, use the filters in the right-hand sidebar to:

  • Review publications for out-of-date or missing information:

    • Non-frozen publications that weren’t recently reviewed (By last reviewed: More than a year ago)

    • Publications without all English and Spanish translations (By untranslated: Yes)

    • Publications without licenses (By data license: Empty)

    • Non-frozen publications without quality summaries (By quality summary [en]: Empty)

    • Non-frozen publications without other information (By incomplete: Yes), one or more of:

      • Country flag

      • Country (en)

      • Retrieval frequency

      • Source URL

      • Language (en)

      • Description (en)

      • Data availability (en)

  • Review publications for visibility and processing:

    • Unpublished publications (By public: No)

    • Frozen publications (By frozen: Yes)

    • Historical publications (By retrieval frequency: This dataset is no longer updated by the publisher)

Review jobs

From time to time, use the filters in the right-hand sidebar to:

  • Check for failed jobs, and restart tasks as appropriate (By failed: Yes)

  • Check for completed jobs whose temporary data has not been deleted (By temporary data deleted: No, By status: COMPLETED)

  • Check for running jobs that are old (By status: RUNNING)

Note

The WAITING status is not used.

Troubleshoot a job

A job’s detail page:

  • Displays the status, result and note (e.g. error messages) for each task, in the Job tasks section.

    If a task’s result is FAILED, but process() considers the failure to be temporary, then the manageprocess command retries the task until it succeeds or fails permanently. Read the Note, and judge whether the failure is permanent. If so, you can set the job’s Status to COMPLETED to stop the retries. The manageprocess command will then delete the job’s temporary data. The next job will be scheduled according to the publication’s retrieval status.

    Attention

    If you want it scheduled sooner, prioritize #350.

  • Defines and displays metadata (Context) from its tasks, in the Management section

    Use the metadata to troubleshoot other applications. For example, to check the Scrapy log, replace the hostname and port in the scrapy_log value with collect.data.open-contracting.org.

    See also

    How to check on progress in:

    This project’s RabbitMQ management interface is at rabbitmq.data.open-contracting.org.

Cancel a job

A job can stall (always “running”). The only option is to cancel the Scrapyd job and set the job’s Status to COMPLETED using the Django admin.

Attention

To properly implement this feature, see #352.

Restart a task

You can restart the Exporter and Flattener tasks. Do this only if the data_registry_production_exporter_init and data_registry_production_flattener_init queues are empty in the RabbitMQ management interface.

Note

The Flattener task publishes one message per file. You might receive a Sentry notification about a failed conversion, while other conversions are still enqueued or in-progress.

The Exporter task publishes one message per job. This task can be restarted while the queue is non-empty – as long as another administrator has not restarted it independently.

  1. Access the job

  2. Set only the Exporter and/or Flattener task’s Status to PLANNED

  3. Click SAVE

Any lockfiles are deleted to allow the task to run.

Attention

See #350.

Unpublish or freeze a publication

  1. Find the publication

  2. Uncheck Public, to hide the publication

  3. Check Frozen, to stop jobs from being scheduled

  4. Click Save at the bottom of the page

Only delete a publication if it is a duplicate or if it was otherwise created in error.

Add an administrator

  1. Click Add next to Users in the left-hand menu

  2. Fill in Username and Password, using a strong password

  3. Click Save and continue editing

On the next form:

  1. Fill in First name, Last name and Email address

  2. Check Staff status (only James and Yohanna should have Superuser status)

  3. Assign Groups (multiple can be selected, as they have non-overlapping permissions)

    Viewer

    Can view publications, quality issues, licenses, jobs and job tasks

    Contributor

    Can add/change publications, quality issues and licenses

  4. Click SAVE