giosystem-ecflow

Giosystem-ecflow is a python package that manages the creation of ecflow servers and suites.

ecflow is a software that allows defining complex workflows, grouping them in suites, families and tasks and scheduling them for execution according to predefined rules.

The giosystem-ecflow library hooks up with the online giosystem-settings-django app and uses the information on the defined servers and suites to build the ecflow structure for the processing lines

Installation

Detailed installation instructions are available at the projects source code repository

Defining ecflow servers and suites

Ecflow servers and suites are defined in the giosystem-settings-django app. The settings are available at:

http://<operational_server>/giosystem/admin

There are three sections of the settings administration area that are used to configure suites and ecfflow servers:

  1. packages - Packages define some settings that are used inside suites:
  2. ecflow_servers
  3. suites

Custom suite variables

Giosystem generated suites have some specific variables that are automatically generated by the giosystemecflow code.

  • COMPRESS_OUTPUTS
  • COLLABORATORS_TO_EMAIL
  • COPY_OUTPUTS_TO_ARCHIVE
  • COPY_OUTPUTS_TO_IO_BUFFER
  • ECF_MICRO
  • EXECUTION_HOST
  • EXTRA_HOSTS_TO_COPY_OUTPUTS
  • OPS_DATABASE_API_KEY
  • OPS_DATABASE_URL
  • OPS_DATABASE_USER
  • PYTHON_COMMAND
  • PYTHON_PATH
  • REMOVE_PACKAGE_WORKING_DIR
  • SERVER_NAME
  • SETTINGS_URL
  • URLBASE
  • USE_ARCHIVE_FOR_SEARCHING
  • USE_IO_BUFFER_FOR_SEARCHING
  • PAKAGE

Custom suite families and tasks

Giosystemecflow generated servers and suites automatically add some families and tasks to each server or suite.

  • /server_maintenance
  • /<suite_name>/maintenance
  • /<suite_name>/archive
  • /<suite_name>/clean

Starting suites

Execute the start_ecflow_servers.py script that is available after installation of the giosystemecflow package. This script will take care of starting the predefined servers and suites. When using the script you will have to provide the URL to the settings that you wish to use.

~/dev/venv/bin/python ~/dev/venv/bin/start_giosystem_servers.py <settings_url>

The script starts the server and loads the suites but it does not start them. Use the ecflowview GUI app to access the server and manually start the suite.

Editing suites

Edit the online settings, save them and then run the reload_suite task in the affected suite. It will be reloaded and fetch the new settings.

Monitoring suites

Use the ecflowview GUI app to monitor suites execution in realtime.