JmmcObsPortal < Jmmc/Software

Tags: view all tags
---+ Observation Portal (Share and track observations)

This page gathers initial ideas & inputs related to improve observers deal with their programs (many targets / OB) and perform (real-time) tracking, scheduling ...

%TOC%

---++ Context

General scope: VLTI and CHARA observations offered to any observer: end-user, large programs, instrument teams (GTO?) like GRAVITY, MATISSE, VEGA teams.

This sketch illustrates the existing tools:

<img align="middle" alt="JmmcUserPortal.png" height="657" src="%ATTACHURL%/JmmcUserPortal.png" title="JmmcUserPortal.png" width="738" />

- JMMC *Aspro2* (http://www.jmmc.fr/aspro):
   * See documentation: http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcAspro2
   * See xml schemas: http://apps.jmmc.fr/~swmgr/AsproOIConfigurations/
This Java GUI to help astronomers prepare their observations.

The session is stored in asprox files (xml based):
   * Portfolio: name, general log (text)
   * Target list + calibrators + ancillary stars (AO / FT) with their models + target notes + specific constraints (HA ranges...) + user groups (backup, priority ...)
   * Interferometer setup: observatory + period (eso) + array setup: configurations (baselines) + PoPs / beams (chara only) + AO setup (VLTI:
   * Instrument setup: instrument mode (resolution + wavelength range) + sampling (periodicity + obs. time)
   * Constraints: elevation, date ?
<img alt="" src="%PUBURL%/Jmmc/Software/JmmcAspro2/Aspro2-main.png" />

ASPRO2 computes observability (target can be observed with current setup during the ... range), UV coverage (coordinates in the fourier plane) that illustrates the target sampling that the user can compare with previous observations using the JMMC OIFits Explorer tool (OIFITS standard files contains targets & uv coordinates already).

ASPRO2 can export Observing blocks (O.B.) to ESO p2 thanks to the A2P2 tool: SAMP integration + specific OBxml exchange format (including observable ranges expressed as HA and LST ranges).

For history ASPRO2 / PIVOT (deprecated) integration was a powerful combination to prepare & manage VEGA programs (Denis Mourard - Instrument PI). PIVOT GUI can import ASPRO2 OBs in its database (proposal) and send global schedule to ASPRO2 (targets with the same interferometer / instrument setup) for the Run PI to finalize the night preparation i.e. the proper night schedule.

- JMMC *A2P2* (https://github.com/JMMC-OpenDev/a2p2):
   * See tutorial: https://github.com/JMMC-OpenDev/a2p2/wiki/ASPRO2-A2P2-Tutorial
A2P2 is a python tool (CLI or GUI) that acts as a SAMP bridge between ASPRO2 and ESO p2 / CHARA backends. This is an open source project maintained by JMMC and ESO instrument scientists (GRAVITY, PIONIER, MATISSE OB constraints). It handles login to ESO portal (user account) and convert ASPRO2 obxml to fill ESO OB templates (generic time ranges) and submit OBs into ESO p2 portal.

For CHARA, it only logs OBs as text log (ASCII art) but in the future A2P2 could submit OBs into CHARA backend (cosmic debris).

- JMMC OiDB (http://oidb-beta.jmmc.fr/index.html):

This JMMC web portal contains both (VLTI / CHARA) observation logs (L0) and OIFITS data (L2 & L3). It is based on a TAP server (taplib + postgres + pgsphere extension) and will soon support private collections (restricted access to data files, meta-data are always public).

Information is stored according to the ObsCore IVOA schema (see http://www.ivoa.net/documents/ObsCore/) (data PI, program Id, target, facility, instrument, time & wavelength range) + minor extensions. Several integrations are in place to perform authentication (JMMC custom user db), query SIMBAD (target validation), ESO logs (cds B/ESO catalog) and OIFits Explorer to provide quick plots.

Metadata description: http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/OiDbColumns

The database could evolve to provide the array setup (configuration) and/or baselines (UV coordinates ?) for observation data...

Examples:
   * 1 GRAVITY OB log: http://oidb-beta.jmmc.fr/show.html?id=729093
   * ESO logs: http://oidb-beta.jmmc.fr/collection.html?id=eso_vlti_import
   * CHARA logs:
      * CLIMB/CLASSIC: http://oidb-beta.jmmc.fr/collection.html?id=chara_import
      * TEST: http://oidb-beta.jmmc.fr/collection.html?id=6c0fe427-7cf8-44e3-a405-fddb75bba235
   * VEGA logs: http://oidb-beta.jmmc.fr/collection.html?id=vegaobs_import

- JMMC OITools / OIFits Explorer:

The OITools open-source library () deals with OIFITS file handling (read / write + processing) and OIFits Explorer () provides plots ...

These tools provides a command line interface and could be helpful to scan reduced OIFITS files (local or in OiDB) & extract detailed observation logs (target + configuration + instrument + UV coordinates)...

- ESO *p2* (http://www.eso.org/p2): Phase-2 preparation tool (once Observation proposals are accepted)
   * see https://www.eso.org/sci/observing/phase2/p2intro.html
P2 is the Science user portal that manages user OBs (corresponding to an accepted proposal) before running observations (service or visitor modes).

Such OB contains detailed setup (instrument, AO specific fields) ...

- ESO Archives:

ESO<b> Obs schedule</b> (http://archive.eso.org/wdb/wdb/eso/sched_rep_arc/form) : this gives the schedule of the observatory (done or planned observations with prog_id / abstract). For example: this query lists recent program ids accepted on the MATISSE instrument: http://archive.eso.org/wdb/wdb/eso/sched_rep_arc/query?wdbo=html%2fdisplay&max_rows_returned=100&tab_period=on&period=&tab_obs_mode=on&tab_run_type=on&run_type=%25&start=&tab_nights=on&tel=%25&tab_instrument=on&instrument=MATISSE%25&instrument_user=&progid=&tab_pi_coi=on&pi_coi=&pi_coi_name=PI_only&title=&tab_dp_id=on&tab_publications=on&order=PERIOD_PROGID&

The ESO *raw archive* gathers observation logs associated to a run PI or program Id. For example:

http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?prog_id=60.A-9134(A)&max_rows_returned=1000

It gives access to detailed information (instrument modes, configuration + uv coordinates from raw file headers).

It also provides specific search forms for VLTI instruments: http://archive.eso.org/cms/eso-data/instrument-specific-query-forms.html (explaining some interesting keywords)

Note: astroquery provides a python module to query the ESO archive: see https://astroquery.readthedocs.io/en/latest/eso/eso.html

note: ESO archive provides a TAP access point: http://archive.eso.org/programmatic/#TAP

It contains also raw information in dbo_raw table (but seems very slow).

---++ Preliminary ideas / Features

---++++ JMMC Observation portal / ObsDB
   * user login + team (group) management to share observation preparation (confidential); super user & instrument PI can bypass visibility restrictions at certain stages (submitted proposals)
   * Adopt TAP interface (VO) ? to allow cone-search queries ? See ObsLocTAP (http://www.ivoa.net/documents/ObsLocTAP/index.html) and OIDB schema (L0)
   * Custom REST API to upload data (observation proposals...)
   * Integrate and process OiDB meta-data to present a simple view on all observation log (VLTI / CHARA L0 to L3). The tricky point consists in gathering individual obs logs per nights, get array configuration (baselines) and UV tracks (coordinates ?) but also deal with duplicates ...
   * Database design: observation proposals + logs (relations / links ?)
   * Basic workflow ? proposals -&gt; accepted -&gt; planned -&gt; done / cancelled
   * Simple Web interface ?
A2P2 enhancements to be the Obs portal client and share information with ASPRO2 (xml + samp)
   * user login
   * new SAMP messages + ObsPortal REST client / ObsTAP client ?
   * CHARA integration (OB submission)

---++++ Support observation tracking in ASPRO2
   * First improve asprox & obxml DM to incorporate observation logs (PI, program ... + obs ranges / configuration / UV tracks); see new ESO UV plane optimiser tool (python)
   * Enter manually logs
   * Scan local OIFITS files =&gt; ingest observation logs

   * Update Observability / UV Coverage plots + adapt portfolio setup (program...)
   * Enhance A2P2 integration to deal with Obs Portal

---++++ Share observation preparation (ASPRO2 into ObsPortal)

Share up-to-date information about observations (log, progress, status, quality information) from ASPRO2 into ObsPortal
   * sharing asprox files could help (as a first step) ? versioning ?
   * To mimic PIVOT integration, ASPRO2 can load a VOTable (target + setup) to dynamically import observations from the ObsPortal ?

For now Asprox files acts as a local storage that users can share with colleagues (send document by email): it is generic.

Note: One science target can correspond to mutiple OBs on several nights (different setups) !

---++++ Other ideas:
   * Night scheduling ?
Observation proposal =&gt; night scheduling : complicated task in ASPRO2
   * Boards for Instrument scientists / runs ? merge OBs into nights ?

---++ Priority / Actions

---+++ Observation tracking

Objective:
   1 get observations logs from (ESO) archives to indicate what observations have been done in ASPRO2
   1 show the Calibrator OBs observed corresponding to the Science targets (CAL - SCI - CAL sequence): it supposes to interpret the OB list along the night to retrieve the SCI-CAL relationships.

Search criteria:
   * Per object (cross match by position) : TAP / cone search is needed as target name are not enough
   * others ? program id, instrument, date ranges ?
   * the user will be able to filter in ASPRO2 which OBs are displayed (i.e. enabled) by configuration (same baselines or not), instrument, instrument mode (spectral resolution)

ASPRO2 changes:
   * show observed LST ranges in Observability plot : every OB is represented by one range (exposure) and the OB details can be displayed in a tooltip
   * show the OB list in a new table widget with filtering options (to be refined)
   * store OBs in asprox files to keep this information in offline mode
   * handle OB list updates (regularly or manually)
   * show the concrete UV points on the UV Coverage panel

Open issues:
   * quality flags: how to get ESO QC flags or night logs ? seeing values could give a rough indication
   * granularity: should OB be grouped or not by instrument setup or night ? for now, every OB corresponds to 1 measurement (few minutes) but headers give LST / UV ranges per baseline =&gt; time / UV coverages must be expressed as value ranges
   * instrument mode:
      * the mapping between ASPRO2 modes and ESO OB keywords is instrument-specific. TODO: define rules per instruments
   * configurations:
      * ESO OB can have been observed at stations that are not present in ASPRO2 offered configuration: it depends on the period or it can be a telescope relocation issue
      * mapping rules are needed to interpret these cases and find 'equivalent' configurations like small, medium, large configurations (new in ESO Period 104)
   * note: Target groups could be used to group targets by program id, instrument setups or nights in the future.
<span style="color: crimson;">WARNING a cone search on ESO's headers for many interferometric stars is bound to fail as the COORDINATES in the HEADER are NOT precise enough. Similarly the OBJECT NAME is not well standardized and sometimes ABSENT. It is thus not possible to search for a star unsing its SIMBAD coodinates or its SIMBAD name.</span>

<span style="color: crimson;">Proposed strategy: find the headers in the database (observed objects) that are not found when looking for the coordinates, and/or the various SIMBAD names, of the objects listed in the headers. Report this list to ESO as invalid headers. Eventually, propose to help them working on a solution.</span>

Data model: see JmmcObsPortalDataAnalysis

Use cases:
   1 GRAVITY / AMBER (Gilles Duvert)
      * YSO large program ids:
         * 099.C-0667(A,B,C) ---&gt; seul B a des données.
         * 0103.C-0347(A,B,C,D,E)
         * 100.C-0278(A...L)
         * 0102.C-0408(A..D)
   1 MATISSE (Alexis Matter)
   1 PIONIER (JBLB)

---++ TODO
   * Study data models (asprox, obxml, p2, eso cfp, archive ...) to extract missing information in every component (oidb, obsPortal, aspro2) and define properly shared information
   * Data life cycle & Software Architecture

   * Build a tiger team ...
   * SPICA requirements ?

---++ Other links of interest
   * https://svn.jmmc.fr/jmmc-web/trunk/exist/oidb/ (auth required)
   * https://svn.jmmc.fr/jmmc-sw/ (auth required)
   * http://vegaobs-ws.oca.eu/

---++ Corner cases in Obs logs:
   * Obs with multiple instruments (MIDI+AMBER) : http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?prog_id=080.C-0344(A)
   * Bad target coordinates: ZCMa on AMBER
      * oidb L0 at (0,0): : http://oidb-beta.jmmc.fr/search.html?conesearch=00%3A00%3A00%2000%3A00%3A00%2CJ2000%2C2%2Carcsec&perpage=100&order=^instrument_name&caliblevel=0
      * oidb Granule http://oidb-beta.jmmc.fr/show.html?id=747525
      * Vizier B/ESO: http://vizier.u-strasbg.fr/viz-bin/VizieR-5?-ref=VIZ5dcadd26a655&-out.add=.&-source=B/eso/eso_arc&recno=1566299
      * ESO archive: http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?dp_id=AMBER.2005-12-22T04:22:40.850
      * Conclusion: all logs are wrong: (0,0) is propagated everywhere but z CMA exists in SIMBAD: http://simbad.u-strasbg.fr/simbad/sim-id?Ident=z+cma&NbIdent=1&Radius=2&Radius.unit=arcmin&submit=submit+id
   * Duplicated obs (same data but different sub_date):
      * http://oidb-beta.jmmc.fr/show.html?id=730030
      * http://oidb-beta.jmmc.fr/show.html?id=970903
      * http://archive.eso.org/wdb/wdb/eso/eso_archive_main/query?dp_id=GRAVI.2017-12-09T03:39:56.824

---++ Data analysis
   * Software prototype on Gitlab@UGA: https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/JMMC/jmmc-obsportal-analysis
   * see JmmcObsPortalDataAnalysis
---+++ Software
---+++ Architecture and database design

   * Using the modeling process described on https://c4model.com
   * see [[JmmcObsPortalArchictecture][JmmcObsPortalArchitecture]]
---+++ Deployment
   * Database
      * PostgreSQL
      * Extension PgSphere : https://pgsphere.github.io/

   * Application
      * Repository : https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/JMMC/jmmc-obsportal
      * Source code uses the Git-Flow pattern :
         * The branch "develop" is the main integration branch merging all developments
         * The branch "master" is the production branch ready to deploy on a server
      * Instructions : see the "Readme.md" file at the root of the project to have up-to-date instructions

   * Kubernetes
      * Repository : https://gricad-gitlab.univ-grenoble-alpes.fr/OSUG/JMMC/jmmc-obsportal-kubernetes
      * Instructions : see the "Readme.md" file at the root of the project to have up-to-date instructions
---+++ Usage
   * Through Aspro2
      * ...
   * Through a Web browser
      * http://obs.jmmc.fr/
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
png	JmmcUserPortal.png	r1	manage	200.4 K	2019-07-02 - 13:39	LaurentBourges
Topic revision: r20 - 2020-02-07 - GillesDuvert