CharaJmmcJJVisitMay2018 < Jmmc

---+ !!Discussion notes during Jeremy Jones visit in Grenoble on 14-15th may with Laurent and Guillaume, then in Calern with Denis, Frédéric and Jean-Michel on 15-16th 

%TOC%





---+ Inline notes:
_Summary of actions and associated priorities/milestones listed at the end of the notes_
---++ Observing Planning:
   * Sharing array & instrument configuration is important : update baseline solutions, array, instrument specific (update MIRC-X setup) etc...
      * [ ] P0-C1 Setup access to the CHARA gitlab (https://gitlab.chara.gsu.edu/) for JMMC staff - request sent to Nils
      * JMMC maintains: http://apps.jmmc.fr/~swmgr/AsproOIConfigurations/
         * CHARA configuration should follow the beam order:
            * [ ] P1-C2 provide the new period (CHARA 2018A) with proper configurations (stations ordered by beams)
            * VEGA 2018A configuration lists the offered baselines + preferred !PoPs (+ beams) that best fits science use cases 
            * [ ] P1-C3 provide the correct (default) beams per instrument configuration (like VEGA does)
            * [ ] P1-J1 update Aspro2's configuration
         * New User preferences: interferometer / period / instrument to be used as default values
      * Denis: in some cases, there are discrepancies between ASPRO2 & CHARA plan observability (same config / !PoPs): to be studied in details on concrete cases (related to the reference cart ?)
         * [ ] P1-V1 : provide detailed concrete cases of discrepancies in Pops btw Aspro and CHARA Plan for check
   * Aspro2 is used in many different modes: preparation (with more or less constraints), scheduling programming, observing
      * Missing features compared to !CharaPlan (see http://www.chara.gsu.edu/tutorials/timing-information )
         * cart position plot against the time :  
            * helps to define POPS and observability range (1 cart is the reference, so 1 delay-line is fixed (at a specific position within 0 - 44m) for the CAL-SCI-CAL sequence (30mins at least ?)
            * How easy is it to solve cart positions ? How stable is the algorithm in terms of maintenance (inputs / parameters) ? 
            * In !CharaPlan the user can select the reference cart, but not in the Aspro2 GUI (VEGA considers always the first telescope of the configuration as the reference cart)
            * [ ] P3-J2 improve the GUI to define scope / beam / pop association in Aspro2 like in !CharaPlan (combo boxes)
            * An option could be to expose how the best pop computation is performed in Aspro2 / and adapt the best pop algorithm (interactive) to take into account other constraints (shorter !PoPs, reference cart...)
         * Alternative: use SAMP to better integrate ASPRO2 with !CharaPlan ? to quickly check the cart positions more fluently (avoid filling GUI twice) and avoid to port specific code into ASPRO
         * [ ] P1-J3 Change default Twilight used at night limit to -12 deg for CHARA or all interferometers ?
         * Offer default / preferred interferometer (or order) and then change some default values depending on the instrument (UTC instead of LST @CHARA eg.)
   * CHARA is operated in different modes (depending on the control knowledge of the observer)
   * Data exchange between Aspro2 and outside world (A2... python tool)
      * Coming support of AO / FT / Guide stars in ASPRO2: CHARA has the same need (Lab AO stars, CHECK stars)
      * currently support of exporting most information shown in the aspro user setting (limited to one target) to XML (A2P2 approach)
      * [ ] P3-J4 develop an extension to export more content of the asprox file ? first all targets (displayed) like star lists
         * add !PoPs info in best-!PoPs mode
         * add UTC next to LST Intervals in the xml file
         * convert to TXT file (CSV ?) to have an observing night plan. Jeremy sent a sample file for PAVO observations
         * later: export target notes + extra information (PI, program, groups)
   * Target identification: HD number => CHARA number (to gather all info)
   * No scheduler / queuing system in the control center: later (could use OB info)
   * Missing information: PI name, Run ID / Program ID ? 
      * use new specific fields or use the new group feature ?
      * possibly use the target notes or observation notebook (in ASPRO2) to store such extra information (instrument ETC info...)
   * Observation logs:
      * how to retrieve / define the observed targets (with baseline / instrument modes) to track progress on large programs ? 
         * new obs log table in ASPRO2 ? filled manually or completed by OIDB L0 queries
         * scan OIFits files ?
         * How to share such information ? using asprox files is cumbersome as it supposes to exchange files and possibly need to merge / keep in sync asprox files (changes + logs). Ideally a database is needed to store & retrieve up-to-date observation preparation & status
      * goal: mark / flag targets in ASPRO2 (observability / UV plan)
   * How to better deal with multiple configurations ?
      * Denis & Jeremy are asking again if ASPRO2 could support multiple configurations : multiple baselines, CLIMB+MIRC or CLIMB+VEGA instruments, different !PoPs setting per target... different instrument modes & settings (integration time) as it is difficult to work with multiple ASPRO2 instances (several asprox files not in sync, confusing): 
      * ideally having several tabs (1 per setup) and also correctly managing the interop of the different tabs with the external applications (SearchCal for exemple).
      * or having the possibility to define & switch easily between setups would be great as it would allow to work on a single ASPRO2 instance (same target list)
      * It is a GUI issue: how to present such different observation setups ? Internally ASPRO2 already supports observation variants (multiple baselines)
      * To be studied: define several observation setup => SETUP1, SETUP2 ... and use target groups to associate targets to this setups (not exclusive i.e. a target may be observed using several setups ...)
      * (Best) !PoPs issue: it depends on the all target list for now, but it should be set on target groups or on setups

---++ Data Archive:
   * CHARA get a file after 2015 with all logged metadata informations night per night
      * started to extract data from txt file into xml
         * The script is not working for all instruments, but soon will be.
         * Once complete, I will run the script on the archive from 2015-present, and then periodically update with current observing
         * Eventually, I will have the chara control software make these xml files and upload them automatically
         * Example script output:
<verbatim>
<granule>
  <target_name>HD_41040</target_name>
  <s_ra>6.05760169444</s_ra>
  <s_dec>19.6905616667</s_dec>
  <t_exptime>0</t_exptime>
  <t_min>58155.131794</t_min>
  <t_max>58155.131794</t_max>
  <facility_name>CHARA</facility_name>
  <instrument_name>CLIMB_2</instrument_name>
  <nb_vis2>3</nb_vis2>
  <nb_t3>1</nb_t3>
  <em_min>2.2</em_min>
  <em_max>2.2</em_max>
  <obs_creator_name>Jeremy Jones</obs_creator_name>
  <datapi>Sandquist</datapi>
  <calib_level>0</calib_level>
</granule>
</verbatim>
            * remarks:
               * [X] add - obs_id Done
               * why is t_exptime = 0 ? (and t_min should probably be different from t_max ) Only one time (t_max) is reported in the logs and in the data files (at least for Classic/CLIMB data)
               * could be added :  nb_channels Done for Classic/CLIMB/JouFLU, more complicated for others
               * think about other ones like : instrument_mode, quality_level
         * http://oidb.jmmc.fr/show.html?id=116012
      * backlog can be ignored for fine details: keep only minimal metadata to contact the PI about observation on a specific target/date
      * /!\ think about providing key value that can be used in OIFits Header data for latter association
   * A web page lists the program schedule with PI, Program Numbers etc... http://www.chara.gsu.edu/observers/observing-schedule
      * [ ] P1-C4 provide this data in a single file (xml e.g.)
      * Also, a webpage with Instrument PIs
   * Observing report email can contains non standardised ProgNb (YYYY/PgrId)
   * Data policy:
      * http://www.chara.gsu.edu/observers/data-policy-access
      * Is metadata confidential?
   * OiDB improvements list:
      * [ ] P1-J5 give a pointer to the observing-schedule web page on every programId fields
      * [ ] P1-J6 support multiple programId (CHARA has such kind of record) ( or just concatenate ? )
      * [ ] P1-J7 add a free input metadata field to complete std ones with comments/quality etc... then request new std entries for most used anduseful ones  
      * [ ] P1-J8 do not hide error report on upload failure by metadata upload
      * [ ] P2-J9 search by program ID


---++ General discussed points:
   *  store a backup of OiDB on CHARA machines (few tens of MBytes TBC)
      * start with meta data + uploaded OIFITS
      * how to perform backup from CHARA side ? (no direct access for now)
         * [ ] P3-J10 provide a jmmc-backup script which downloads a single tar file (should be less than few GB)
            * and provide rotating time scale backup operations strategy : last 7 days, last 4 weeks, last 12 months + 1 tar per year ?
         * [ ] P3-C5 run jmmc-backup script each day with archival rotation
      * how JMMC retrieve previous backups ?
         * Just send an email to Jeremy ( waiting for a direct access in the future )
   *  backup CHARA's data on Grenoble site ?
      *  raw data not under our scope at present time. Could be imagine with Grenoble University at cost.
      * talk later about reduced data : Spring/Fall 2019 &#8211; Reduced data accessible via the OIDB.
   *  A2P2 demo + ESO Observing portal ?
      * 2 options to start working in direction to CHARA interaction:
         * clone a2p2 to a2c...
         * handle another facility depending on the message sent by Aspro2 
         * 2nd one is the prefered one - (-> new name proposal : Aspro 2 PreP)
         * [ ] P0-J11 provide a first implementation support for CHARA in A2P2
         * https://github.com/JMMC-OpenDev/a2p2
   *  every JMMC services and tools tour
      * bad calibrators feedbacks ?
      * New getstar service provides an up-to-date stellar diameter estimation (2.5M stars): http://www.jmmc.fr/getstar
   * add current existing last data onto oidb-beta
      * draw the dataflow for future automatic update
   * service monitoring
      * of JMMC webservice availability from CHARA machines
         * [ ] P3-J12 send jmmc-monitoring script samples and URLs lists to Jeremy.
      * of any CHARA service from JMMC side 
         * [ ] P1-J13  monitoring CHARA's website availability 
   * News / general info :
      * CHARA wiki will move because WIKISPACES will shut down in next jully 31st
         * will move to dokuwiki
      * [ ] P0-J14 notify Gerard Van Belle about wikispace's shutdown for Olbin instance
   * Current CHARA's archive size : 20 TB used today + 20~30 TB / year
    
---++ VEGA specific points:
We discussed with Denis, Frédéric and Jean-Michel at Calern related to VEGA some new points:
   * Long-term plan of getting OBs on CHARA
   * How a2p2 works and possible future applications for CHARA
   * Upcoming groups feature on aspro
   * CHARA logs and my plans for getting them on OIDB
      * Short-term option:
         * [ ] P0-C6 CHARA produces std metadata for VEGA + addition details specific to VEGA in a separated file (instrument setup)
            * metadata are sent to oidb with a link to the complementary file(s)
      * detailed log files hosted on the CHARA web site or in the CHARA archive
   * Missing information in L0 (obs logs): array setup (baseline / config at least), instrument setup (mode ?)
   * [ ] P2-J15 OiDB could provide a way for datapi to edit their metadata, especially for tracking completion progress + indicate data quality & data processing status (DM can complete the list)

---++ Email exchanges:
   * encourage information exchanges on the  jmmc-chara@jmmc.fr mailing list
   * remind JMMC-OpenDev github repository : https://github.com/JMMC-OpenDev   

---+ Summary of actions and associated priorities/milestones:
*task code : [' ' TODO - / started - X done] P<0 top priority - 3 no priority><C-hara/J-mmc/V-ega team>*

P0 - End of july 2018
   * [/] P0-C1 Setup access to the CHARA gitlab (https://gitlab.chara.gsu.edu/) for JMMC staff - request sent to Nils
   * [ ] P0-C6 CHARA produces std metadata for VEGA + addition details specific to VEGA in a separated file (instrument setup)
   * [X] P0-J11 provide a first implementation support for CHARA in A2P2
      * first version may be tested on a dedicated branch:
<verbatim>
git clone https://github.com/JMMC-OpenDev/a2p2.git
cd a2p2 
git checkout origin/firstChara
pip install .
a2p2
</verbatim>
   * [X] P0-J14 notify Gerard Van Belle about wikispace's shutdown for Olbin instance
P1 - End of 2018
   * [ ] P1-C2 provide the new period (CHARA 2018A) with proper configurations (stations ordered by beams)
   * [ ] P1-C3 provide the correct (default) beams per instrument configuration (like VEGA does)
   * [ ] P1-C4 provide this data in a single file (xml e.g.)
   * [ ] P1-V1 : provide detailed concrete cases of discrepancies in Pops btw Aspro and CHARA Plan for check
   * [ ] P1-J1 update Aspro2 configuration
   * [ ] P1-J3 Change default Twilight used at night limit to -12 deg for CHARA or all interferometers ?
   * [ ] P1-J5 give a pointer to the observing-schedule web page on every programId fields
   * [ ] P1-J6 support multiple programId (CHARA has such kind of record) ( or just concatenate ? )
   * [ ] P1-J7 add a free input metadata field to complete std ones with comments/quality etc... then request new std entries for most used anduseful ones
   * [ ] P1-J8 do not hide error report on upload failure by metadata upload
   * [ ] P1-J13  monitoring CHARA's website availability
P2 - Spring/Fall 2019
   * [ ] P2-J15 OiDB could provide a way for datapi to edit their metadata, especially for tracking completion progress + indicate data quality & data processing status (DM can complete the list)
   * [ ] P2-J9 search by program ID
P3 unscheduled / require specifications or discussions
   * Not listed, see details in notes.
This topic: Jmmc > WebHome > CharaJmmcCollaboration > CharaJmmcJJVisitMay2018
Topic revision: r3 - 2018-07-27 - GuillaumeMella