Giosystem-orders

Giosystem includes an ordering service that processes order requests for products of the processing lines.

The ordering service uses the following components:

OGC OSEO server

The ordering service implements a subset of the OGC OSEO specification, which defines a standard for ordering earth observation products.

The giosystem-orders server is based on the pyoseo project, while incorporating some custom logic in order to work with the giosystem processing lines.

Note

The OSEO server has the following restrictions:

  • Implements only the following OSEO operations:
    • Submit
    • GetStatus
    • DescribeResultAccess
    • GetOptions
  • Does not implement the asynchronous interface for the Submit operation
  • Does not support FTP push as a delivery method

Supported OSEO operations

Submit

The Submit operation allows a client to place an order on the server. Upon having been successfully authenticated, the order is placed in the order database and is sent to the processing queue, where it will be processed in due time. The following image provides a rough workflow of the Submit operation:

Submitting an order

Since the server does not support the OSEO asynchronous interface, the response to a Submit request is sent only once, immediately after the order has been requested. Further status changes must be explicitly asked for by the user, by issuing Getstatus requests.

Example requests and further information on the Submit operation is available at http://pyoseo.readthedocs.org/en/latest/submit.html

GetStatus

The Getstatus operation allows a client to ask the server what is the status of an order. The server queries its order database and returns the status information back to the client.

Getting the status of an order

As orders get processed by the queue, their status is dynamically updated. After an order has been successfully downloaded, its status is also updated.

Example requests and further information on the GetStatus operation is available at http://pyoseo.readthedocs.org/en/latest/getstatus.html

DescribeResultAccess

When an order has finished processing, a client can issue a DescribeResultAccess request in order to discover the URLs where the ordered items are available. The OSEO server supports both HTTP and FTP pull as delivery methods for an order.

Generally, completed orders will remain available for download for a period of 10 days.

GetOptions

Server configuration

Being based on django, the server is configured by editing the settings.py and settings_local.py files. Besides the main django configuration, these files define the following settings, specific to the OSEO server:

  • GIOSYSTEM_SETTINGS_URL - The full URL to the API for the giosystem settings web application;
  • OSEOSERVER_ORDER_DELETION_THRESHOLD - An integer specifying the number of days that ordered items will remain on the server before being deleted;
  • OSEOSERVER_ONLINE_DATA_ACCESS_HTTP_PROTOCOL_ROOT_DIR - The path on the file system to the root directory for placing ordered items which are to be available via HTTP;
  • OSEOSERVER_ONLINE_DATA_ACCESS_FTP_PROTOCOL_ROOT_DIR - The path on the file system to the root directory for placing ordered items which are available via FTP;
  • OSEOSERVER_AUTHENTICATION_CLASS - The Python import path to the custom authentication class that is used when authenticating incoming requests;
  • OSEOSERVER_PROCESSING_CLASS - The Python import path to the custom module that handles the actual processing of orders
  • OSEOSERVER_ADMIN_MAILS - a list of e-mail addresses that will be notified whenever an order fails
  • OSEOSERVER_MAIL_SERVER - name of the server that is used to send e-mail notifications to the server’s administrators whenever an order fails
  • OSEOSERVER_MAIL_SERVER_PORT - port of the mail server
  • OSEOSERVER_MAIL_DOMAIN - domain of the mail server
  • OSEOSERVER_MAIL_ACCOUNT - user account that is to be used when sending e-mails
  • OSEOSERVER_MAIL_ACCOUNT_PASSWORD - password of the account that is to be used when sending e-mails
  • OSEOSERVER_AUTH_VITO_ATTRIBUTE - Custom setting used in the authentication module;
  • OSEOSERVER_AUTH_VITO_TOKEN - Custom setting used in the authentication module;
  • OSEOSERVER_AUTH_VITO_PASSWORD - Custom setting used in the authentication module.

The settings_local.py file holds site specific settings that override the ones in settings.py.

Order database

The giosystem ordering service is backed by a django application that includes a database. The database stores information on each order’s contents, such as:

  • status
  • requesting user
  • ordered items
  • delivery options

The stored information is suficciently detailed in order to reconstruct an order request and monitor the complete order status life cycle.

The database is dynamically updated as orders are being processed. It is controlled by the OSEO server. Additionally, there is also a web based client used to inspect and manage the database. It can be accessed at

http://geoland2.meteo.pt/giosystem/ordering/admin

User authentication module

The authentication module runs whenever there is a request for ordering. Currently, the implemented authentication scheme uses the SOAP headers to supply authentication information between the ordering client and the server.

The authentication scheme agreed between IPMA and VITO operates as follows:

  • ordering requests must send in a wsse:Security/wsse:Password element with the correct value.
  • ordering requests must send in a wsse:Security/wsse:Username element with the username of the user that made the order on the PDF portal. The user’s password is never sent to the ordering server. When the user tries to access the FTP server to download the data, the FTP server communicates with an LDAP server stored at VITO and authenticates the user using its username and the user supplied password.

Order processing queue

The giosystem ordering service is backed by a celery task queue. This enables the service to manage task execution processes asynchronously. The queue is populated with specialized order processing tasks, which are responsible for doing the actual processing of orders

There are two daemon processes responsible for processing orders and maintenance stuff. They are:

pyoseo-worker

The pyoseo-worker daemon is a celery queue that is responsible for processing orders.

pyoseo-beat

The pyoseo-beat daemon is responsible for performing periodic tasks, such as monitoring the number of FTP downloads for each order item, removing old orders, etc.

Both these daemons are configured to start automatically at boot, so if there ever is a need to reboot the machine, they will be back online as soon as it reboots.

Managing the services

The daemons are managed using SystemV-style init scripts. The scripts are located in the /etc/init.d directory and can be used like this:

sudo service pyoseo-worker status # check if the service is running
sudo service pyoseo-worker start # start the service
sudo service pyoseo-worker stop # stop the service
sudo service pyoseo-worker restart # stop the service and immediately start it again

The previous commands can be used with both daemons. Both init scripts are exact copies of the files supplied by the celery development team. The original can be found at celery’s github repository.

Each service can be customized by editing the files in:

/etc/default/pyoseo-worker
/etc/default/pyoseo-beat

For more information on the options for each service, consult celery’s documentation.

Log files

The log files for these services are located in /var/log/celery. They can be monitored in real time by issuing the command:

tail -f /var/log/celery/worker1.log # for the pyoseo-worker daemon
tail -f /var/log/celery/celeryd.log # for the pyoseo-beat daemon

Order processing module

The order processing module performs the job of actually fetching the ordered items and placing them in the correct directories on the web or the FTP server, making them available for download. This module makes use of the helper functions defined in the giosystemcore.

Order processing consists in:

  • retrieving the ordered items and options
  • fetching the products from the giosystem host network
  • apply some post-processing routines
  • placing the ordered items in their web accessible destination

Note

Update this documentation to reflect the current status of the ordering server.

In order to handle order requests that specify delivery as HTTP GET requests, the django-giosystem-downloads app includes a get_ordered_file view that will fetch the files from disk. In order for it to work correctly, you must install the xsendfile apache module.

sudo apt-get install libapache2-mod-xsendfile

Debugging failed orders

Orders can fail if the user placing the order is not actually registered in the PDF portal. In order to verify that a given user is present in VITO’s LDAP database, one can use the ldapsearch command. For example, to see if the user demouser exists in VITO’s LDAP server, issue the command:

ldapsearch -v -x -D <bind_dn> -w <password> -s sub -b <search_base> -P 3 -H <ldap_uri> uid=demouser

Note

The variables bind_dn, password search_base and ldap_uri are sensitive information. They’re values are stored at IPMA’s internal network. Look for them in the passwords section of the collaborative platform.