(r6) JmmcObsPortalDataAnalysis < Jmmc/Software

Tags: view all tags
---+ JMMC/ObsPortal: Data analysis

%TOC%

---+++ Astroquery
<pre>- https://astroquery.readthedocs.io/en/latest/<br />- https://astroquery.readthedocs.io/en/latest/eso/eso.html</pre>

It seems that there is no API to query the ESO archive. Astroquery uses the library <a href="https://pypi.org/project/beautifulsoup4/" title="BeautifulSoup">BeautifulSoup</a> to parse HTML pages of ESO archive website, extract content and aggregate data into a <a href="https://docs.astropy.org/en/stable/table/" title="'tabular' structure">"tabular" structure</a>. <a href="https://pypi.org/project/beautifulsoup4/" title="BeautifulSoup"></a>The search seems to fill the forms. So, it is possible to filter results on some properties (instrument name, target name, category (science, calibrator, ...), observation ID, program ID, ...). The headers related to a query result can be retrieved but it is necessary to use a list of 'dp_id'

---+++ AsproX / ESO archive

It is relatively easy to generate an Asprox file from one line extracted from the ESO archive (one OBS_ID for a defined instrument), but some informations must be "translated" to fill the related XML tags. However, the reversed way (from Aspro to ESO Arch.) is another work... : the Aspro UI lacks of fields for 'program ID' or 'observation ID' and we have to define how to identify the ESO data from the parameters entered in the UI.

---++++ Notes from meeting with Gilles (2019-11-19)
   * Main goal of Aspro: to help astronomers/scientists to prepare their proposals.
   * Observation = what (targets, calibrators), when, where/how (interferometer, instrument parameters)
      * planned / to do
      * done (extracted from ESO, OIDB, ...)
   * From one ProgID, it may be possible to find the calibrator associated to a target because a standard observation follow the sequence : Calibrator. Science, Calibrator
      * Need to check also if the information is given by a keyword in a header file.

---+++ Data model / Mapping

This gathers the unified data model from !OiDB, ASPRO2 and ESO logs:

---++++ Questions:
   * Granularity: 1 OB corresponds to 1 data point in ESO log, but ASPRO2 or OBxml can represent more data points, like the OIFITS format ...


---++++ Identifiers

Many identifiers are existing in the ESO archive, none in either ASPRO2 / OBxml formats yet:
   * Program id:
      * obxml: add new field programId
      * oidb: prog_id
      * eso: keyword 'HIERARCH ESO OBS PROG ID' like '0103.C-0347(C)' (starts with ESO period in first 4 digits)
   * observation id (unique):
      * eso: keyword 'HIERARCH ESO OBS ID' like '2438871' (Observation block ID)
   * archive file (unique):
      * eso: keyword 'ARCFILE' like 'GRAVI.2017-09-28T08:14:15.939.fits' (Archive File Name)
      * TODO: decide if such field is useful in obs db ?
   * container id (p2): it refers to the OB container (P2PP / p2) that gathers individual OBs (CAL - SCI - CAL sequence)
      * eso: keyword 'HIERARCH ESO OBS CONTAINER ID' like '2438869' (Scheduling container ID)
   * PI, proposal id: out of scope


---++++ Object
   * Name: object name (text)
      * asprox: target/name
      * obxml: target/name
      * oidb: target_name
      * eso: keyword 'HIERARCH ESO OBS TARG NAME', not 'OBJECT' (less reliable)
   * type: SCIENCE/CALIBRATOR
      * asprox: based on targetUserInfos/calibrators group
      * obxml: observationConfiguration/type = 'SCIENCE' or 'CALIBRATION'
      * oidb: Missing
      * eso: keyword 'HIERARCH ESO DPR CATG' = 'SCIENCE' or 'CALIB'
   * coordinates (in FK5 J2000)
      * asprox: target/RA target/DEC (sexagesimal): 02:36:37.9182097570 +12:26:51.459132329
      * obxml: target/RA target/DEC (sexagesimal)
      * oidb: s_ra / s_dec (deg) displayed in sexagesimal format
      * eso: keywords 'RA' / 'DEC' (deg), check EQUINOX = 2000 and RADECSYS='FK5'

---++++ Interferometer setup

This describes the interferometer parameters: facility (VLTI / CHARA ...), baselines (+ pops, beams), AO (naomi ...)
   * Observatory:
      * asprox: interferometerConfiguration/name (start) like 'VLTI Period 105'
      * obxml: interferometerConfiguration/name
      * oidb: facility_name
      * eso: 'VLTI' (hard-coded)
   * Period (eso like):
      * asprox: interferometerConfiguration/name (end) like 'VLTI Period 105'
      * obxml: interferometerConfiguration/version
      * oidb: Undefined
      * eso: keyword 'HIERARCH ESO OBS PROG ID' (start) like '0103.C-0347(C)' corresponds to Period 103
   * Configuration: 2T, 3T, 4T (VLTI) and 5T, 6T (CHARA)
      * asprox: instrumentConfiguration/stations + variant/stations like 'A0 G1 J2 K0' (based on internal dictionary // eso offered configurations)
      * obxml: interferometerConfiguration/stations (same format) + A2P2 maps given values into P2 (small, medium, large configurations from P104)
      * oidb: telescope_configuration (never filled in obsCore extension)
      * eso: keyword 'HIERARCH ESO OBS BASELINE' like 'A0-G1-J2-J3' (reliable ?)

TODO: describe AO settings (type, mode) in asprox / obxml

Extra parameters:
   * obxml: !PoPs (CHARA) / channels (beams) in interferometerConfiguration/pops (PoP1...5) and interferometerConfiguration/pops (V1...V6) (based on internal dictionary)

---++++ Instrument setup

This describes the instrument parameters:
   * mode (spectral coverage)
<verbatim>    
    <instrumentConfiguration>
        <name>GRAVITY</name>
        <pops></pops>
        <instrumentMode>HIGH-COMBINED</instrumentMode>
    </instrumentConfiguration>
</verbatim>

---++++ observation time:
   * time values

TODO:
<pre>
EXPTIME   316.0000000   Integration time
MJD-OBS   58627.07295558   Obs start
DATE-OBS   2019-05-24T01:45:12   Observing date
UTC   6301.000   [s] 01:45:00.000 UTC
LST   47333.932   [s] 13:08:53.932 LST
</pre>

---++++ UV coverage:
   * UV point data

TODO:
<pre>
HIERARCH ESO OBS BASELINE = 'A0-G1-J2-K0' / User selected VLTI stations

HIERARCH ESO ISS CONF NTEL = 4 / Number of telescopes used.
HIERARCH ESO ISS CONF STATION1 = 'A0' / Station of telescope arm 1.
HIERARCH ESO ISS CONF STATION2 = 'G1' / Station of telescope arm 2.
HIERARCH ESO ISS CONF STATION3 = 'J2' / Station of telescope arm 3.
HIERARCH ESO ISS CONF STATION4 = 'K0' / Station of telescope arm 4.

HIERARCH ESO ISS PBL12 END = 74.435 / Projected baseline T1 T2 at end [m].
HIERARCH ESO ISS PBL12 START = 74.477 / Projected baseline T1 T2 at start [m].
HIERARCH ESO ISS PBL13 END = 119.562 / Projected baseline T1 T3 at end [m].
HIERARCH ESO ISS PBL13 START = 119.604 / Projected baseline T1 T3 at start [m].
HIERARCH ESO ISS PBL14 END = 126.627 / Projected baseline T1 T4 at end [m].
HIERARCH ESO ISS PBL14 START = 126.641 / Projected baseline T1 T4 at start [m].
HIERARCH ESO ISS PBL23 END = 58.178 / Projected baseline T2 T3 at end [m].
HIERARCH ESO ISS PBL23 START = 58.176 / Projected baseline T2 T3 at start [m].
HIERARCH ESO ISS PBL24 END = 84.283 / Projected baseline T2 T4 at end [m].
HIERARCH ESO ISS PBL24 START = 84.266 / Projected baseline T2 T4 at start [m].
HIERARCH ESO ISS PBL34 END = 37.910 / Projected baseline T3 T4 at end [m].
HIERARCH ESO ISS PBL34 START = 37.910 / Projected baseline T3 T4 at start [m].

HIERARCH ESO ISS PBLA12 END = 110.1 / Projected baseline orientation angle T1 T2
HIERARCH ESO ISS PBLA12 START = 110.1 / Projected baseline orientation angle T1
HIERARCH ESO ISS PBLA13 END = 87.6 / Projected baseline orientation angle T1 T3
HIERARCH ESO ISS PBLA13 START = 87.6 / Projected baseline orientation angle T1 T
HIERARCH ESO ISS PBLA14 END = 70.218 / Projected baseline orientation angle T1 T
HIERARCH ESO ISS PBLA14 START = 70.232 / Projected baseline orientation angle T1
HIERARCH ESO ISS PBLA23 END = 58.4 / Projected baseline orientation angle T2 T3
HIERARCH ESO ISS PBLA23 START = 58.4 / Projected baseline orientation angle T2 T
HIERARCH ESO ISS PBLA24 END = 35.744 / Projected baseline orientation angle T2 T
HIERARCH ESO ISS PBLA24 START = 35.739 / Projected baseline orientation angle T2
HIERARCH ESO ISS PBLA34 END = 359.531 / Projected baseline orientation angle T3
HIERARCH ESO ISS PBLA34 START = 359.503 / Projected baseline orientation angle T

4 stations: A0 G1 J2 K0
6 baselines: A0-G1 A0-J2 A0-K0 G1-J2 G1-K0 J2-K0

=> UV tracks: PBL (length in meters), PBLA (angle in degrees)

uv [ 'A0-G1': { length = 74.5 m, angle = 110 deg },
     'A0-J2': { length = ... m, angle = ... deg } ]

</pre>

---++++ Specific ESO Instrument Mapping rules:
---+++++ GRAVITY

First priority

TODO: define filtering options (DP_CAT / DP_TYPE values) + extra FIELDS ?
Topic revision: r6 - 2019-11-25 - LaurentBourges