giosystemcore¶

giosystemcore is a package that facilitates working with files and algorithms used in the IPMA GIO-GL processing lines. It is usable by itself, but the real advantage of this package is when it is used together with the django-giosystem-settings web application.

It is a set of python modules that abstract the details of product generation, making it easy to generate IPMA GIO-GL products.

For example, if we want to fetch all the inputs to the ‘preprocess_goes_lrit’ package into a working directory on our local filesystem:

import giosystemcore.packages.packagefactory as factory
from giosystemcore.settings import get_settings

# only works inside IPMA's network
get_settings('http://gio-gl.meteo.pt/giosystem/settings/api/v1/')
package = factory.get_package('preprocess_goes_lrit', '201402011000')
fetched = package.fetch_inputs()
print(fetched)

Installation¶

Detailed installation instructions are available at giosystemcore’s source code repository at http://bitbucket.org/ipmagio/giosystem-core

Core modules¶

settings¶

The giosystemcore.settings module interfaces with the django-giosystem-settings web application fetching the settings needed to configure the various objects.

When using giosystemcore, the first step will typically be the calling of giosystemcore.settings.get_settings() function, in order to establish a connection to the settings that we want to use.

from giosystemcore.settings import get_settings
get_settings('http://gio-gl.meteo.pt/giosystem/settings/api/v1/')
# now we are ready to rock!

files¶

A giosystemcore.files.GioFile represents a file used in the processing lines. Files are used by packages as inputs and outputs. They are present in hosts. A file has a search_pattern and a search_path which can be used to look for it in the defined hosts. Files can be configured in the giosystem-settings-django administration backend. The operational settings are available at

http://gio-gl.meteo.pt/giosystem/settings/admin/coresettings/files

The giosystemcore.files module provides an abstraction to work with the different file types used and generated in the giosystem.

It defines the giosystemcore.files.GioFile class, that has properties and methods useful for finding, fetching and sending files to/from multiple hosts.

Creating instances of GioFile should be done through the giosystemcore.files.get_file() factory function.

import giosystemcore.files
f = giosystemcore.files.get_file('', '201402071300')

packages¶

Giosystem relies on a set of specialized code routines for generating GIO products. These routines are defined as giosystem packages.

Packages represent execution units in the processing lines. They are used to generate products. Packages usually have inputs and outputs, which are GioFile instances. Packages have several methods for fetching their inputs, executing the product generation code and moving the outputs to designated hosts.

Packages should be instantiated by using the giosystemcore.packages.packagefactory.get_package() function. It will create the package using the settings defined in the giosystem-settings-django application.

The following example demonstrates creating a package for processing the process_goes_lst package for the 2014-03-15 09:00:00 timeslot:

import giosystemcore.settings
import giosystemcore.packages.packagefactory as pf

settings_url = 'http://gio-gl.meteo.pt/giosystem/coresettings/api/v1/'
giosystemcore.settings.get_settings(settings_url)
p = pf.get_package('process_goes_lst', '201403150900')
p.use_io_buffer_for_searching = False
p.use_archive_for_searching = False
p.copy_outputs_to_io_buffer = False
p.copy_outputs_to_archive = False
result, details = p.run()

hosts¶

A giosystemcore.hosts.hosts represents a host machine used in the giosystem processing lines. Hosts have methods for interacting with their respective filesystems, including ways to fetch files.

host_roles¶

Some hosts have special roles in the context of the giosystem processing lines. There is an archive host that holds the archive of IPMA’s GIO-GL products. There is also a web_server host that has the public giosystem webservices installed.

The giosystemcore.hosts package provides several host classes. These are an abstraction of the real host machines in use by the giosystem and provide a number of useful methods for dealing with file copy and editing.