JmmcObsPortalDataAnalysis < Jmmc/Software

Tags: view all tags
---+ JMMC/ObsPortal: Data analysis

%TOC%

---+++ Astroquery
<pre>- https://astroquery.readthedocs.io/en/latest/<br />- https://astroquery.readthedocs.io/en/latest/eso/eso.html</pre>

It seems that there is no API to query the ESO archive. Astroquery uses the library <a href="https://pypi.org/project/beautifulsoup4/" title="BeautifulSoup">BeautifulSoup</a> to parse HTML pages of ESO archive website, extract content and aggregate data into a <a href="https://docs.astropy.org/en/stable/table/" title="'tabular' structure">"tabular" structure</a>. <a href="https://pypi.org/project/beautifulsoup4/" title="BeautifulSoup"></a>The search seems to fill the forms. So, it is possible to filter results on some properties (instrument name, target name, category (science, calibrator, ...), observation ID, program ID, ...). The headers related to a query result can be retrieved but it is necessary to use a list of 'dp_id'

---+++ AsproX / ESO archive

It is relatively easy to generate an Asprox file from one line extracted from the ESO archive (one OBS_ID for a defined instrument), but some informations must be "translated" to fill the related XML tags. However, the reversed way (from Aspro to ESO Arch.) is another work... : the Aspro UI lacks of fields for 'program ID' or 'observation ID' and we have to define how to identify the ESO data from the parameters entered in the UI.

---++++ Notes from meeting with Gilles (2019-11-19)
   * Main goal of Aspro: to help astronomers/scientists to prepare their proposals.
   * Observation = what (targets, calibrators), when, where/how (interferometer, instrument parameters)
      * planned / to do
      * done (extracted from ESO, OIDB, ...)
   * From one ProgID, it may be possible to find the calibrator associated to a target because a standard observation follow the sequence : Calibrator. Science, Calibrator
      * Need to check also if the information is given by a keyword in a header file.

---+++ Data model / Mapping

This gathers the unified data model from !OiDB, ASPRO2 and ESO logs:

---++++ Questions:
   * Granularity: 1 OB corresponds to 1 data point in ESO log, but ASPRO2 or OBxml can represent more data points, like the OIFITS format ...

   * Two solutions:
      * 1-1 mapping (flat model like OiDB): 1 OB (1 data point) is stored all in 1 table (all meta data)
      * relational mapping: object / interferometer / instrument / OB (time, dates) are stored in several tables with relations that allows grouping / projections ??

---++++ Eso Archive Query filter:

General filtering options (HIERARCH ESO DPR CATG / DPR TYPE keywords or DP_CAT / DP_TYPE form fields) in ESO archive:
   * GRAVITY:
      * CAT = 'SCIENCE' or 'CALIB'
      * TYPE in ('OBJECT,SINGLE', 'OBJECT,DUAL', 'STD,SINGLE', 'STD,DUAL')
   * MATISSE:
      * CAT = 'SCIENCE' and TYPE = 'OBJECT' (not 'OBJECT,RMNREC')
      * CAT = 'CALIB' and TYPE = 'STD'
   * PIONIER:
      * CAT = 'SCIENCE' and TYPE = 'OBJECT' (2015) or 'OBJECT,FRINGE' (2019)
      * CAT = 'CALIB' and TYPE = 'OBJECT'

Sample ESO keywords:
<verbatim>
HIERARCH ESO DPR CATG = 'SCIENCE' / DPR category.
HIERARCH ESO DPR TECH = 'INTERFEROMETRY,DIRECT' / DPR tech.
HIERARCH ESO DPR TYPE = 'OBJECT,SINGLE' / DPR type.</verbatim>

---++++ Identifiers

Many identifiers are existing in the ESO archive, none in either ASPRO2 / OBxml formats yet:
   * Program id:
      * obxml: TODO: add new field observingBlockDefinition/programId
      * oidb: prog_id
      * eso: keyword 'HIERARCH ESO OBS PROG ID' like '0103.C-0347(C)' (starts with ESO period in first 4 digits)
   * observation id (unique):
      * obxml: TODO: add new field observingBlockDefinition/observationId
      * oidb: obs_id
      * eso: keyword 'HIERARCH ESO OBS ID' like '2438871' (Observation block ID)
   * archive file (unique):
      * eso: keyword 'ARCFILE' like 'GRAVI.2017-09-28T08:14:15.939.fits' (Archive File Name)
      * TODO: decide if such field is useful in obs db ?
   * container id (p2): it refers to the OB container (P2PP / p2) that gathers individual OBs (CAL - SCI - CAL sequence)
      * eso: keyword 'HIERARCH ESO OBS CONTAINER ID' like '2438869' (Scheduling container ID)
   * PI, proposal id: out of scope

Former OiDB logs stored dataId (DP.ID) into obs_id (arcfile like), not real obs_id (missing in B/eso) !
ARCFILE seems more meaningful than obs_id (eso internal), so it could be a better identifier for the observation id.

TODO: check container_id ? but it is not reliable for older observations (AMBER or PIONIER observations)

Sample ESO keywords:
<verbatim>
HIERARCH ESO OBS PROG ID = '60.A-9295(A)' / ESO program identification

HIERARCH ESO OBS CONTAINER ID = 1833674 / Scheduling container ID
HIERARCH ESO OBS ID = 200436943 / Observation block ID

ARCFILE = 'GRAVI.2017-09-28T08:14:15.939.fits' / Archive File Name</verbatim>

---++++ Object
   * Name: object name (text)
      * asprox: target/name
      * obxml: target/name
      * oidb: target_name
      * eso: keyword 'HIERARCH ESO OBS TARG NAME', not 'OBJECT' (less reliable)
   * type: SCIENCE/CALIBRATOR
      * asprox: based on targetUserInfos/calibrators group
      * obxml: observationConfiguration/type = 'SCIENCE' or 'CALIBRATION'
      * oidb: Missing
      * eso: keyword 'HIERARCH ESO DPR CATG' = 'SCIENCE' or 'CALIB'
   * coordinates (in FK5 J2000)
      * asprox: target/RA target/DEC (sexagesimal): 02:36:37.9182097570 +12:26:51.459132329
      * obxml: target/RA target/DEC (sexagesimal)
      * oidb: s_ra / s_dec (deg) displayed in sexagesimal format
      * eso: keywords 'RA' / 'DEC' (deg), check EQUINOX = 2000 and RADECSYS='FK5'

Sample ESO keywords:
<verbatim>
OBJECT  = 'HD16234'        / Original target.
HIERARCH ESO OBS TARG NAME = 'HD16234' / OB target name

RA      =        39.156144 / [deg] 02:36:37.4 RA (J2000) pointing
DEC     =         12.44750 / [deg] 12:26:50.9 DEC (J2000) pointing

EQUINOX =             2000 / Standard FK5
RADECSYS= 'FK5'            / Coordinate system</verbatim>

---++++ Interferometer setup

This describes the interferometer parameters: facility (VLTI / CHARA ...), baselines (+ pops, beams), AO (naomi ...)
   * Observatory:
      * asprox: interferometerConfiguration/name (start) like 'VLTI Period 105'
      * obxml: interferometerConfiguration/name
      * oidb: facility_name
      * eso: 'VLTI' (hard-coded)
   * Period (eso like):
      * asprox: interferometerConfiguration/name (end) like 'VLTI Period 105'
      * obxml: interferometerConfiguration/version
      * oidb: Undefined
      * eso: keyword 'HIERARCH ESO OBS PROG ID' (start) like '0103.C-0347(C)' corresponds to Period 103
   * Configuration: 2T, 3T, 4T (VLTI) and 5T, 6T (CHARA)
      * asprox: instrumentConfiguration/stations + variant/stations like 'A0 G1 J2 K0' (based on internal dictionary // eso offered configurations)
      * obxml: interferometerConfiguration/stations (same format) + A2P2 maps given values into P2 (small, medium, large configurations from P104)
      * oidb: telescope_configuration (never filled in obsCore extension)
      * eso: keyword 'HIERARCH ESO OBS BASELINE' like 'A0-G1-J2-J3' (reliable ?)

TODO: describe AO settings (type, mode) in asprox / obxml

Extra parameters:
   * obxml: !PoPs (CHARA) / channels (beams) in interferometerConfiguration/pops (PoP1...5) and interferometerConfiguration/pops (V1...V6) (based on internal dictionary)

Sample ESO keywords:
<verbatim>
HIERARCH ESO OBS BASELINE = 'A0-G1-J2-K0' / User selected VLTI stations</verbatim>

---++++ Instrument setup

This describes the instrument parameters: the instrument mode is a label pointing to the instrument setup in ASPRO2's configuration where every mode describes the wavelength table (min / max, spectral resolution ...).

2 different points of view are present in ASPRO2 and OiDB:
ASPRO2 uses modes (referring to its internal configuration), whereas OiDB describes the instrument backend (wave_min/wave_max, resolution) + the instrument name encodes some meaning of the observing technique.

How to map such information ?
Idea: use specific rules (ESO) per instrument, i.e. keep important ESO keywords (specific) to be able to determine the corresponding ASPRO2 mode.

   * instrument (name):
      * asprox: instrumentConfiguration/name (ID in configuration)
      * obxml: interferometerConfiguration/name
      * oidb: instrument_name (may contains extra values)
      * eso: keyword 'INSTRUME' like 'GRAVITY'

Note: MATISSE is currently described as 2 instruments in ASPRO2: MATISSE_LM, MATISSE_N (like detectors).

   * instrument mode (spectral coverage): see rules

Sample ESO keywords:
<verbatim>
INSTRUME= 'GRAVITY'        / Instrument used.</verbatim>

---++++ Observation time:

Each OB has a start and exposure time that forms a time range. ASPRO 2 can deal with MJD or LST ranges, but LST is then computed at the observatory location. It seems better to also have the LST values given by the ESO archive.

Sample ESO keywords:
<verbatim>   
EXPTIME   316.0000000   Integration time
MJD-OBS   58627.07295558   Obs start
DATE-OBS   2019-05-24T01:45:12   Observing date
UTC   6301.000   [s] 01:45:00.000 UTC
LST   47333.932   [s] 13:08:53.932 LST</verbatim>

Proposed solution:
   * exposure time (s): exp_time = EXPTIME
   * mjd_min = MJD-OBS
   * mjd_max = MJD-OBS + EXPTIME
   * lst_start = LST
Other values like UTC or timestamps can be derived from MJD.

Sample OBxml observability ranges:
<verbatim>   
       <observationConstraints>
            <HAinterval>-3.40/3.40</HAinterval>
            <LSTinterval>20:53/03:41</LSTinterval>
        </observationConstraints></verbatim>

2020.02.06: EXPTIME values seem incorrect (~ dit, not the total time):

Ex PIONIER:

HIERARCH ESO DET NAME = 'RAPID' / Name of detector system
HIERARCH ESO DET DIT = 0.000750 / Integration Time [s]
HIERARCH ESO DET NDIT = 51200 / Number of integrations

EXPTIME=38.4 s = NDIT * DIT = real time in measurements (without overheads)

Note: Instrument or ISS keywords may give more accurate MJD_END = MJD_START + total exposure duration


---++++ UV coverage:
   * UV points are given by baseline in ESO headers

TODO explanations

<verbatim>
HIERARCH ESO OBS BASELINE = 'A0-G1-J2-K0' / User selected VLTI stations

HIERARCH ESO ISS CONF NTEL = 4 / Number of telescopes used.
HIERARCH ESO ISS CONF STATION1 = 'A0' / Station of telescope arm 1.
HIERARCH ESO ISS CONF STATION2 = 'G1' / Station of telescope arm 2.
HIERARCH ESO ISS CONF STATION3 = 'J2' / Station of telescope arm 3.
HIERARCH ESO ISS CONF STATION4 = 'K0' / Station of telescope arm 4.

HIERARCH ESO ISS PBL12 END = 74.435 / Projected baseline T1 T2 at end [m].
HIERARCH ESO ISS PBL12 START = 74.477 / Projected baseline T1 T2 at start [m].
HIERARCH ESO ISS PBL13 END = 119.562 / Projected baseline T1 T3 at end [m].
HIERARCH ESO ISS PBL13 START = 119.604 / Projected baseline T1 T3 at start [m].
HIERARCH ESO ISS PBL14 END = 126.627 / Projected baseline T1 T4 at end [m].
HIERARCH ESO ISS PBL14 START = 126.641 / Projected baseline T1 T4 at start [m].
HIERARCH ESO ISS PBL23 END = 58.178 / Projected baseline T2 T3 at end [m].
HIERARCH ESO ISS PBL23 START = 58.176 / Projected baseline T2 T3 at start [m].
HIERARCH ESO ISS PBL24 END = 84.283 / Projected baseline T2 T4 at end [m].
HIERARCH ESO ISS PBL24 START = 84.266 / Projected baseline T2 T4 at start [m].
HIERARCH ESO ISS PBL34 END = 37.910 / Projected baseline T3 T4 at end [m].
HIERARCH ESO ISS PBL34 START = 37.910 / Projected baseline T3 T4 at start [m].

HIERARCH ESO ISS PBLA12 END = 110.1 / Projected baseline orientation angle T1 T2
HIERARCH ESO ISS PBLA12 START = 110.1 / Projected baseline orientation angle T1
HIERARCH ESO ISS PBLA13 END = 87.6 / Projected baseline orientation angle T1 T3
HIERARCH ESO ISS PBLA13 START = 87.6 / Projected baseline orientation angle T1 T
HIERARCH ESO ISS PBLA14 END = 70.218 / Projected baseline orientation angle T1 T
HIERARCH ESO ISS PBLA14 START = 70.232 / Projected baseline orientation angle T1
HIERARCH ESO ISS PBLA23 END = 58.4 / Projected baseline orientation angle T2 T3
HIERARCH ESO ISS PBLA23 START = 58.4 / Projected baseline orientation angle T2 T
HIERARCH ESO ISS PBLA24 END = 35.744 / Projected baseline orientation angle T2 T
HIERARCH ESO ISS PBLA24 START = 35.739 / Projected baseline orientation angle T2
HIERARCH ESO ISS PBLA34 END = 359.531 / Projected baseline orientation angle T3
HIERARCH ESO ISS PBLA34 START = 359.503 / Projected baseline orientation angle T</verbatim>

4 stations: A0 G1 J2 K0
6 baselines: A0-G1 A0-J2 A0-K0 G1-J2 G1-K0 J2-K0

=&gt; UV tracks: PBL (radius in meters), PBLA (angle in degrees).

Note: as the exposure time is small, just use the middle point = (START + END) / 2

For example:
uv [ 'A0-G1': { length = 74.5 m, angle = 110 deg },
     'A0-J2': { length = ... m, angle = ... deg } ]

---++++ Specific ESO Instrument Mapping rules:

---+++++ GRAVITY
<verbatim>
INSTRUME= 'GRAVITY'        / Instrument used.

HIERARCH ESO INS SPEC RES = 'HIGH' / Science camera spectral resolution
HIERARCH ESO INS POLA MODE = 'COMBINED' / Polarization mode</verbatim>

These 2 keywords easily maps into ASPRO2 modes '[spectral resolution]-[polarization mode]':
<verbatim>
        <name>LOW-COMBINED</name>
        <name>LOW-SPLIT</name>
        <name>MEDIUM-COMBINED</name>
        <name>MEDIUM-SPLIT</name>
        <name>HIGH-COMBINED</name>
        <name>HIGH-SPLIT</name></verbatim>

   * Fringe Tracker

If 'HIERARCH ESO DPR TYPE' ends with 'DUAL', a second acquisition must be recorded in database from the same header wiith these keywords :
<verbatim>
HIERARCH ESO FT ROBJ NAME = 'WDS_J00063-4905B' / Fringe tracking object name
HIERARCH ESO FT ROBJ ALPHA = '000619.1443' / Fringe tracking object RA 
HIERARCH ESO FT ROBJ DELTA = '-490434.4608' / Fringe tracking object DEC

HIERARCH ESO FT POLA MODE = 'COMBINED' / Fringe tracker polarization mode</verbatim>

Spectral resolution : LOW
Instrument : GRAVITY_FT

Rule: The science target and the FT target should be the same if DPR TYPE' ends with 'SINGLE' but should be different if DPR TYPE' ends with 'DUAL'

---+++++ MATISSE
   * sample LM:
<verbatim>
INSTRUME= 'MATISSE'        / Instrument name

HIERARCH ESO INS MODE = 'HYBRID' / Instrument mode used.
HIERARCH ESO DET NAME = 'MATISSE-LM' / Name of detector system

HIERARCH ESO INS DIL ID = 'HIGH' / L&M DIL  unique id.
HIERARCH ESO INS DIL NAME = 'HIGH' / L&M DIL name.

HIERARCH ESO INS DIN ID = 'LOW' / N DIL unique id.
HIERARCH ESO INS DIN NAME = 'LOW' / N DIL name.

HIERARCH ESO INS FIL ID = 'L' / L&M FIL unique id.
HIERARCH ESO INS FIL NAME = 'L' / L&M FIL name.

HIERARCH ESO INS FIN ID = 'OPEN' / N FIN unique id.
HIERARCH ESO INS FIN NAME = 'OPEN' / N FIN name.

HIERARCH ESO INS PIL ID = 'PHOTO' / PIL unique ID.
HIERARCH ESO INS PIL NAME = 'PHOTO' / PIL name.

HIERARCH ESO INS PIN ID = 'INTER' / PIN unique ID.
HIERARCH ESO INS PIN NAME = 'INTER' / PIN name.</verbatim>
   * sample N:
<verbatim>
INSTRUME= 'MATISSE'        / Instrument used.

HIERARCH ESO INS MODE = 'HYBRID' / Instrument mode used.
HIERARCH ESO DET NAME = 'MATISSE-N' / Name of detector system

HIERARCH ESO INS DIL ID = 'LOW' / L&M DIL  unique id.
HIERARCH ESO INS DIL NAME = 'LOW' / L&M DIL name.

HIERARCH ESO INS DIN ID = 'LOW' / N DIL unique id.
HIERARCH ESO INS DIN NAME = 'LOW' / N DIL name.

HIERARCH ESO INS FIL ID = 'LM' / L&M FIL unique id.
HIERARCH ESO INS FIL NAME = 'LM' / L&M FIL name.

HIERARCH ESO INS FIN ID = 'OPEN' / N FIN unique id.
HIERARCH ESO INS FIN NAME = 'OPEN' / N FIN name.

HIERARCH ESO INS PIL ID = 'PHOTO' / PIL unique ID.
HIERARCH ESO INS PIL NAME = 'PHOTO' / PIL name.

HIERARCH ESO INS PIN ID = 'INTER' / PIN unique ID.
HIERARCH ESO INS PIN NAME = 'INTER' / PIN name.</verbatim>

DIL/DIN, FIN/FIL and PIL/PIN corresponds to LM / N detectors, but 1 OB corresponds to a single detector (DET NAME).
PIL/PIN indicates Photometry measurement i.e. SI_PHOT mode ?

Current ASPRO2 MATISSE modes (2019.11):
   * Instrument MATISSE_LM:
<verbatim>
<name>SI_PHOT_L_LOW</name>
<name>SI_PHOT_LM_LOW</name>
<name>SI_PHOT_M_LOW</name>
<name>SI_PHOT_L_MEDIUM</name>
<name>SI_PHOT_M_MEDIUM</name>
<name>SI_PHOT_L_HIGH</name>
<name>SI_PHOT_L_VERY_HIGH</name> (disabled)
<name>SI_PHOT_M_VERY_HIGH</name> (disabled)</verbatim>
   * Instrument MATISSE_N:
<verbatim>
<name>HIGH_SENS_N_LOW</name>
<name>HIGH_SENS_N_HIGH</name></verbatim>

Fringe tracker used:
   * If 'HIERARCH ESO DPR TYPE' ends with 'RMNREC'
---+++++ PIONIER

Headers have 2 formats for instrument modes (changed in 2015-08-29):
   * 2013:
<verbatim>
INSTRUME= 'PIONIER'        / Instrument used.

HIERARCH ESO INS FILT1 ID = 'H' / Filter unique id.
HIERARCH ESO INS FILT1 NAME = 'H' / Filter name.

HIERARCH ESO INS FILT2 ID = 'FREE' / Filter unique id.
HIERARCH ESO INS FILT2 NAME = 'FREE' / Filter name.

HIERARCH ESO INS OPTI2 ID = 'H' / OPTIi unique ID.
HIERARCH ESO INS OPTI2 NAME = 'H' / OPTIi name.

HIERARCH ESO INS OPTI3 ID = 'SMALL' / OPTIi unique ID.
HIERARCH ESO INS OPTI3 NAME = 'SMALL' / OPTIi name.</verbatim>
   * 2019:
<verbatim>
INSTRUME= 'PIONIER'        / Instrument used.

HIERARCH ESO INS FILT1 NAME = 'H' / Filter name.

HIERARCH ESO INS OPTI2 ID = 'GRISM' / OPTIi unique ID.
HIERARCH ESO INS OPTI2 NAME = 'GRISM' / OPTIi name.

HIERARCH ESO INS OPTI3 ID = 'REF' / OPTIi unique ID.
HIERARCH ESO INS OPTI3 NAME = 'REF' / OPTIi name.</verbatim>

ASPRO2 modes:
   * PIONIER (old):
<verbatim>
BROAD-H
SMALL-H
LARGE-H
BROAD-K
LARGE-K</verbatim>
   * PIONIER_2 (new):
<verbatim>
GRISM-H
FREE-H</verbatim>

---+++++ AMBER

TODO

<verbatim>
HIERARCH ESO OCS OBS MODE = '3T' / Observation mode
HIERARCH ESO OCS OBS SPECCONF = 'Medium_K_1_2.1' / Spectral configuration</verbatim>

ASPRO2 modes:
<verbatim>
mode 	resolution 	wlen min 	wlen max
Low_JHK: 	70 	1.469 	2.54
Medium_H_1_1.65: 	1500 	1.54 	1.82
Medium_K_1_2.1: 	1500 	1.926 	2.275
Medium_K_1_2.3: 	1500 	2.126 	2.474
High_K_1_1.979: 	12000 	1.955 	2.003
High_K_1_2.018: 	12000 	1.994 	2.042
High_K_1_2.056: 	12000 	2.032 	2.079
High_K_1_2.095: 	12000 	2.071 	2.118
High_K_1_2.133: 	12000 	2.110 	2.156
High_K_1_2.172: 	12000 	2.149 	2.195
High_K_1_2.211: 	12000 	2.187 	2.235
High_K_1_2.249: 	12000 	2.226 	2.272
High_K_1_2.288: 	12000 	2.265 	2.311
High_K_1_2.326: 	12000 	2.303 	2.348
High_K_1_2.365: 	12000 	2.342 	2.387
High_K_1_2.403: 	12000 	2.381 	2.425
High_K_1_2.442: 	12000 	2.420 	2.464
High_K_1_2.481: 	12000 	2.459 	2.503</verbatim>

---+++++ MIDI

TODO
Topic revision: r13 - 2020-02-06 - LaurentBourges