OiDbTechnicalSpecs < Jmmc/Software

This topic aims to describe the specification from a technical point of view.

%TOC%

TODO: define the instrument mode (can we get same general concept across various instruments : resolution (LOW MEDIUM HIGH), Lambda min/max etc...?  )

---++ Data import

<img src="%ATTACHURLPATH%/import.png"/>

Parse/Read: submitted files are parsed with oitools to produce a structure of data from the file.

Analyze/extract data: given the data structure above, it splits the data into granules and outputs relevant metadata that are formatted as an XML fragment for each granule.

!XML2SQL: transform the XML fragments describing a granule into SQL insertion for the database.

Target resolution: a service that try to identify the target of an observation given incomplete or approximate inputs from the file data.

Instrument mode resolution: a service that try to identify the instrumental mode used during an observation given a few elements from the file.

DB: relational database containing observations as support for the web portal and tap service with subtables for instruments and targets. A subset of the database (view) is exported through TAP

Backoffice: a panel for administrator to supervise the import process.


---+++ Data analysis

It add a few identifier to the OIFits structure as derived columns. These values can be queried to extract metadata granules.

OI_TARGET: add column(s) for target identifier - TARGET_UID - derived from OI_TARGET.TARGET, RAEP0, DECEP0, EQUINOX (converted to coords RA[deg], DEC[deg] J2000)

OI_WAVELENGTH: add keyword instrument mode - INSMODE (string) - derived from OI_WAVENLENGTH.INSNAME, OI_WAVELENGTH.EFF_WAVE_min/max, count(OI_WAVELENGTH.EFF_WAVE)

OI_DATA: add column for night identifier - NIGHT_ID - derived from OI_DATA.MJD or OI_DATA.DATE-OBS+TIME (converted to MJD) and OI_ARRAY(OI_DATA).ARRAYX, ARRAYY, ARRAYZ 


---++++ Impact

   * jmcs: convert coords to given epoch
   * oitools: column definitions
   * oiexplorer-core: analysis of target id, instrument mode, night identifiers / set column values
      * API oiexplorer-core : TODO oifits.getOiTableOfInsMode(insmodeId), oidata.getInsModeId() oidata.getDistinctNightId(), oidata.getDistinctTargetUid()



---+++ Target resolution

Purpose: define identifier for target

Input: TARGET_NAME (string), coords (RA[deg], DEC[deg] J2000) %BR%
Output: TARGET_UID (pair of strings with ra as HMS and dec as DMS ?)

A service that builds an identifier based on previous resolutions and interaction with other services (like Simbad).
So doing it also help validating the input data and assigning validation flag to input data.

When a target can not be resolved, the identifier is built using input data and a flag is added specifying the reason of the failure (service unavailable, unknown target, no match between name and coords).

---++++ Scenarios

   * name resolution does not return a result (unknown target) -> try with coords
   * name resolution returns a single result
      * resolved coords equals to passed coords
      * resolved coords significantly different from passed coords
   * name resolution returns more than one result
   
   * coord reverse resolution returns no result
   * coord reverse resolution returns a single result
   * coord reverse resolution returns several results

---++++ Impact

   * oiexplorer-core: resolutions, error reporting

---++++ Notes

   * As new aliases are assigned to targets, new definition of names/aliases, insertions into Simbad, updates of coordinates happen, the service should also be routinely re-evaluated: identifying target in the database for which resolution return a result different from the one previously saved (main name, coords, identifier). A report would display the differences for validation/modification by an administrator.

---+++ Instrument mode resolution

Purpose: define identifier for instrument mode

Input: INSNAME (string), MIN_WL (double), MAX_WL (double), NB_CHANNEL (double), knowledge database for instrument modes %BR%
Output: INSMODE (string)

A service that build a fingerprint/hash to describe an instrument mode using a knowledge database of instrument modes.
This knowledge database may be the configuration file of ASPRO2 or an extract of OiDB and build along data imports.

---++++ Scenarii

   * fingerprints with slightly different INSNAME values (e.g. wavelengths min and max as in PIONIER_Pup(1.5994200/1.8035799) )
   * instrument names with constant values (e.g. VEGA )
   * unknown instrument mode
   * unknown instrument

---++++ Impact

   * oiexplorer-core: build a fingerprint/hash
   * oidb: match fingerprint/hash with instrument mode

---++++ Notes

   * footprint definition (single string, fields concatenation...)?
   * how to solve instrument modes ambiguity (e.g. for MIDI and MATISSE how to choose between SCI_PHOT_LM_LOW or HIGH_SENS_LM_LOW  )
   * identify interferometer (parse ARRAY_NAME, check coords XYZ against known locations -> inventory of interferometers)
   * detect unknown interferometers, flag for backoffice, update inventory
   * may also save extra data in inventory for suggestion on next submit
   * build knowledge database along sumbits

---+++ Group by nights

Purpose: associate a night identifier to observations

Input: MJD (double[]), optionnal ARRAY_NAME (string) %BR%
Output: NIGHT_ID[], distinct_NIGHT_ID[]

Find which night corresponds to the given MJD at the array location.

---++++ Scenarios

   * single night
   * multiple nights (contiguous or not)
   * OIFits file without OI_ARRAY (optional)
      * Location can be found with instrument name when OI_ARRAY is missing otherwise take GMT.
      * or treat as invalid (flag in database)

---++++ Impact

   * jmal: night boundary at location,date
   * oiexplorer-core: compute NIGHT_ID

---++ General

   * do we have to refresh/compare data retrieved by simbad
   * number of telescope: the more telescope, the better for image reconstruction -> interesting to identify
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
dia	import.dia	r1	manage	2.8 K	2014-04-10 - 07:15	PatrickBernaud	Source file for the import diagram (version 1)
dia	import2.dia	r1	manage	3.0 K	2014-04-10 - 07:15	PatrickBernaud	Source file for the import diagram (version 2)
This topic: Jmmc/Software > OiDbTechnicalSpecs
Topic revision: r11 - 2014-04-15 - PatrickBernaud