Data import

Parse/Read: submitted files are parsed with oitools to produce a structure of data from the file.

Analyze/extract data: given the data structure above, it splits the data into granules and outputs relevant metadata that are formatted as an XML fragment for each granule.

XML2SQL: transform the XML fragments describing a granule into SQL insertion for the database.

Target resolution: try to identify the target of an observation given incomplete or approximate inputs from the file data.

Instrument mode resolution: try to identify the instrumental mode used during an observation given a few elements from the file.

DB: relational database containing observations as support for the web portal and tap service with subtables for instruments and targets. A subset of the database (view) is exported through TAP

Backoffice: a panel for administrator to drive the import process.

Data analysis

It add a few identifier to the OIFits structure as derived columns. These values can be queried to extract metadata granules.

OI_TARGET: add column(s) for target identifier - TARGET_UID - derived from OI_TARGET.TARGET, RAEP0, DECEP0, EQUINOX

OI_WAVELENGTH: add column instrument mode - INSMODE (string) - derived from OI_WAVELENGTH.EFF_WAVE_min/max, count(OI_WAVELENGTH.EFF_WAVE)

OI_DATA: add column for night identifier - NIGHT_ID - derived from OI_DATA.MJD or OI_DATA.DATE-OBS+TIME and OI_ARRAY(OI_DATA).ARRAYX, ARRAYY, ARRAYZ

Scenarios

• OIFits file without OI_ARRAY (optional)

Impact

• jmcs: convert coords to given epoch
• oitools: column definitions
• oiexplorer-core: analysis of target id, instrument mode, night identifiers / set column values

Target resolution

Purpose: define identifier for target

Input: TARGET_NAME (string), coords: RA(deg), DEC(deg) J2000)
Output: TARGET_UID (pair of strings with ra as HMS and dec as DMS ?)

It builds an identifier by interacting with other services (like Simbad and previous resolution from OiDB itself). It is passed data like values extracted from the rows of an OI_TARGET (TARGET, RAEP0, DECEP0, EQUINOX) of an OIFits file. So doing it also help validating the input data and assigning quality flag to input data.

When a target can not be resolved, the identifier is built using input data and a flag is added specifying the reason of the failure (service unavailable, unknown target, no match between name and coords).

Scenarios

• name resolution does not return a result (unknown target) -> try with coords
• name resolution return a single result
  • resolved coords equals to passed coords
  • resolved coords significantly different from passed coords
• name resolution return more than one result

• coord reverse resolution return no result
• coord reverse resolution return a single result
• coord reverse resolution return several result

Impact

• oiexplorer-core: resolutions, error reporting

Notes

Instrument mode resolution

Purpose: define identifier for instrument mode

Input: INSNAME (string), MIN_WL (double), MAX_WL (double), NB_CHANNEL (double), knowledge database for instrument modes
Output: INSMOD_UID (string)

Given the fingerprint of an instrument mode for, identify the instrument mode using a knowlegde database of instruments mode. This knowledge database may be the configuration file of ASPRO2 or an extract of OiDB and build along data imports.

Scenarios

• instrument names embedding the instrument mode
• fingerprints with slightly different values (wavelengths min and max)

Impact

• oiexplorer-core: fingerprint comparason, instrument matching

Notes

• How to handle data for unknown instrument?
• footprint definition (single string, fields concatenation...)?
• ambiguity of instrument modes (MIDI, MATISSE)

Group by nights

Purpose: associate a night identifier to observations

Input: MJD (double), ARRAY_NAME (string) Output: NIGHT_ID

Find which night corresponds to the given MJD at the array location.

Scenarios

Impact

• jmal: night boundary at location,date
• oiexplorer-core: compute NIGHT_ID

General

• do we have to refresh/compare data retrieved by simbad