Data loader

 Loading data in your SaadaDB is 2 folds

• Declaration of the configuration
• Product ingestion.

 Defining a configuration The configuration gives Saada some rules indicating how to load a file :

• Which file matches the configuration
• In which collection they must be stored
• In which class they must be mapped
• Which attributes must be reported at collection level

The image below shows a panel set up for a product configuration. This is the simplest panel for the plot category. Other product categories (images, tables and spectra) have specific inputs explained in further articles.

Product declaration

• Product name of this configuration : Set the configuration name. It must be unique in the category.

• In collection : List of all collections available.

• Identification of the files matching ... : There are three modes to identify files matching the configuration. All files with FITS, fits or fits.gz (and xml for table) as suffix are considered as datafiles.
  • By directory : All datafiles found into the directory will be considered as matching the configuration.
  • By filename : This input requires a java regular expression used to select files. Java REGEXP must match the whole filename (without suffix).
    • The expression « spectra.* » matches all filenames begining with « spectra » (but « spectra » not)
  • By Keyword/Value:The dataloader will read all datafiles and select products with a header containing the couple KW/Value set. This method is quite slow.
• Coordinate System : Select the coordinate system of the products matching the configuration. We assume that all products of a given configuration have the same system of coordinates.

• option Mapping : In this panel are selected keyword values which will be reported to the collection level.
  • Keyword used as instance name : This field can take multiple keywords which will be merged to set the Saada name of the instance. Saada record can always be selected by name.
  • Ignored keywords : The keywords listed here are not loaded
  • Extended attributes(see here) : The collection attributes defined by the operator are listed here. Each one will be set with the value of the keyword set in the input field.

• Class Mapping strategy (see here).

• Save : This button saves the new configuration in a file (in SAADA_DB_HOME/config), nothing else. There is no way to remove or modify a saved configuration except by editing this file.

 Data Loading the loading procedure is the same for all product categories.

• Configurations available : Select the configuration you want to load. The configuration selected is summarized in the text area below.

• Select an option : This selector allows (or not) the dataloader to modify data classes during the loading process. Attributes can be added (but never removed) or attribute type can be changed.

• Select a directory ; Set the root directory from where products matching the configuration will be searched.
  • If product are selected by directory, the dataloader will look for a directory having the right name and will take in consideration all datafiles it will found.
  • If products are selected by filename, the dataloader will search in this directory all datafiles matching the pattern defined in the configuration.
  • If products are selected by KW values, the dataloader will consider all datafiles found in this directory.

• Load : This button starts the loading process logged in the console. In the current Saada release, the console slows down dramatically the loading process.