OIFITSTwoProject < Jmmc

--+ OIFits version 2:
   * has been released on A&A on january 2017.
   * %RED%PUBLISHED%ENDCOLOR% version is in [[http://adsabs.harvard.edu/abs/2017A%26A...597A...8D][Astronomy & Astrophysics]] 
   * %RED%MAINTAINED%ENDCOLOR% version is on [[http://dx.doi.org/10.1051/0004-6361/201526405][ArXiv]]
   * a copy of the latest version is kept [[%ATTACHURL%/oifits_norm_v2.pdf][HERE]]

---+ This page was intended for discussions on the draft: Second Version of the _[[http://www.mrao.cam.ac.uk/research/OAS/oi_data/oifits.html][%RED% OI-FITS %ENDCOLOR% format]]_, please find it below.

-------------------------------------------

---+ %RED%The Document in its current state:%ENDCOLOR%
The document will be published in A&A. It is deposited on [[http://arxiv.org/abs/1510.04556][<b>ArXiv</b>]] and will be maintained there independently of the A&A publication, if need arises. 

---+ %RED% VERY LAST CHANGES%ENDCOLOR%
   * Replaced OI_SPECTRUM by OI_FLUX, this is the naming used by GRAVITY, and furthermore suggested by the A&A referee. MATISSE (Main.PhilippeBerio) knows about this change and will use it.
 
---+ The historical discussion on this format is here http://www.jmmc.fr/twiki/bin/view/Jmmc/JmmcOldPageOnOifitsTwoProject

---+ This page is free for your comments. 
   1 You may need to register to enter coments. 
   1 Do not forget to sign them using the signature button!


---++ Comment by !XavierHaubois
- In order to get a better description of data and better curation possibilities, could INSNAME include an identifier of the instrument spectral mode (on top of the polarization)  ? Or even better create a new keyword in the OI_WAVELENGTH table.

- Astrometric data products of GRAVITY will eventually consist in a series of positions (RA and DEC) and their associated errors as functions of wavelength. 
In order to fit the OIFITS2 format, these positions could be decomposed in differential phases and astromectic baselines (ucoord and vcoord) than can be stored in a OI_VIS like table.


 

---++ Comment by !MikeIreland --> see comments by -- Main.GillesDuvert - 30 Sep 2014

As discussed at the Interferometry Forum, covariance between visibility and closure-phase points as a function of wavelength is *essential*. This can be included 
with the following:

In OI_VIS2

   * !VIS2COV  D(NWAVE,NWAVE)  (optional) Covariance matrix of squared visibility.

In OI_T3

   * !T3AMPCOV D(NWAVE,NWAVE)  (optional) Covariance matrix of triple product amplitude
   * !T3PHICOV D(NWAVE,NWAVE)  (optional) Covariance matrix of triple product phase

Implementing the full wavelength and observation covariance matrix is possibly tricky, but would be required for me to e.g. put aperture masking data in oifits.
The most general way to implement this would be:

In OI_VIS2

   * !VIS2IX I(NWAVE) (optional) Index of the V2 point.

In OI_T3

   * !T3AMPIX I(NWAVE)
   * !T3PHIIX I(NWAVE)

In a separate table OI_EXTRACOV:

   *  ... firstly requires a title...
   *  INDICES I(NIX)     The indices from the various !VIS2IX, !T3AMPIX, !T3PHIIX tables that are used in this covariance matrix
   *  COV D(NIX,NIX) Full covariance matrix of the quantities indexed by !INDICES.

This would enable e.g. the covariance between OI_T3(!T3AMP) and OI_VIS2(!VIS2DATA) to be stored. By convention, I suggest that these covariances are added to 
any covariances already stored in OI_VIS2 and OI_T3. This would enable e.g. temporal covariances due to common calibrators to be easily recorded and could help
to avoid unwieldy oifits files that have a giant N_OBSERVABLES * N_OBSERVABLES matrix in them.

Note that the VISCOVRI as described in the OIVIS format should have D(NWAVE) not D(NWAVE,NWAVE), with each element being the off-diagonal element of a 2x2 covariance matrix between the real and imaginary parts.

---++ -- Main.GillesDuvert - 30 Sep 2014: The covariance requirements have been added, after discussion with parties, using a new OI_CORR table, see final document

---++ Comment by !AntoineMerand

   * Visibility and correlated flux: do I understand correctly that UNITS (table 1) apply to all data of the HDU? if so, can you handle one OI_VIS holding visibilities and the correlated fluxes? If I understand well, you cannot, so you need 2 distinct OI_VIS tables. In practice, it should be easy since I expect instruments providing correlated fluxes in OI_VIS, and visibilities in OI_VIS2 (no OI_VIS). Maybe would be worth to detail how correlated fluxes should be stored.

   * MP and MB: "Ratio of input over output apertures&#8221; and "time-varying component of the IFOV&#8221;. I agree with Gilles, MB should not be in the table, or only optional. Do we have existing or plan instruments with such feature?

   * concerning OI_WAVELENGTH: 
      * why should we have EFF_BAND set to NULL in the case monochromatic values? I think there should always be a EFF_BAND, I cannot think of a realistic monochromatic case&#8230;  
      * "These estimates should include the effect of the earth&#8217;s atmosphere, but not the spectrum of the target (the effect of the target spectrum should be taken into account as part of any model-fitting/mapping process, i.e. the target spectrum is part of the model).&#8221; what does that mean? do yo mean that if the target is red/blue shifted, the wavelength should not be in the rest referential of the target?

   * for differential quantities in OI_VIS: "VISREFMAP must be present if any of VISTYP or PHITYP is "differential". If VISREFMAP is absent, neither can be differential.&#8221;: isn&#8217;t this too strict? Same could be said in the case visibility (or vis2) are binned.

   * concerning OI_SPECTRUM: 
      * should be restricted to spectra derived with the interferometric instrument, not to third party instruments! otherwise, why not have an OI_PHOTOMETRY as well? Few interferometric instruments are suitable to extract photometry, but MIDI for instance, is pretty good at it&#8230; 
      * I do not understand the meaning of &#8220;CALIBRATED&#8221;: I think the relevant information should be: is it raw, normalized, photometrically calibrated? Another important information is wether or not the telluric lines were removed. 
      * I do not understand either the usefulness of the following: "If the boolean keyword header CALIBRATED is FALSE, the STA_INDEX column must be present, and the flux stored in FLUXDATA is the flux measured on the telescope indicated by STA_INDEX."

   * some things not covered by the document:
      * Astrometry: GRAVITY will not be able to save its reduce data in the OIFITS format. I am not sure it is really possible, but maybe there should be a foreword stating that astrometry has been excluded.
      * Error correlations due to common calibrators for OI_VIS2. Each calibrated observation should give the name of the 2 calibrators used for calibration, as well as their visibilities. See http://www.aanda.org/articles/aa/pdf/2003/12/aa2990.pdf for a description of how correlations can be used. 


---++ Comment by Main.PierreCruzalebes
Dear colleagues,
Since optical long-baseline interferometric data can be processed using the instantaneous spectral distribution of the complex triple product, as is it done for instance with the SPIDAST software, I suggest to add to the OI_T3 data table the real and imaginary parts of the triple product with their errors.
Staying at your disposal, with my best regards.

|!T3REAL    |        D(NWAVE)|     Real Part of Triple Product|
|!T3REALERR |    D(NWAVE)    | Error in Real Part of Triple Product|
|!T3IMAG    |       D(NWAVE) |    Imaginary Part of Triple Product|
|!T3IMAGERR  |  D(NWAVE)    | Error in Real Part of Triple Product|

---+ Response by Main.JohnYoung
@Pierre - Could you please clarify why the real/imaginary representation is better for the instantaneous triple product than amplitude/phase. I am not convinced about the extent to which we should complicate the format to accommodate instantaneous data, given that it is mostly used for time-averaged data.
---+ Final(!) comment by -- Main.GillesDuvert - 05 Nov 2014
OIFITS2 does not support actively 'raw = instantaneous' data. However instantenous complex values of correlated fluxes and errors are available in OIFITS2 via the !OI_VIS RVIS,RVISERR,IVIS,IVISERR columns. Computing an instantaneous triple product on these values is straightforward (and covariances can be computed too).

---++ Comments by Main.FlorentinMillour
Dear Gilles,

First of all, thanks a lot for coordinating this work. I think you have done a great job putting everything together in the draft you sent us.
Please see attached my comments on that first version. I summarize the most relevant to me below.


   * I've put a list of suggested keywords to put in the main header and suggestions whether they may be mandatory or optional

Comments by-- Main.FlorentinMillour - 15 Jul 2014: add the list of proposed keywords:
One can use standard FITS keywords (http://heasarc.gsfc.nasa.gov/docs/fcg/standard_dict.html) for most of the information needed:
      * AUTHOR e.g. 'A. Millour' mandatory
      * DATE e.g. '2012-03-25T00:00:03.32' mandatory
      * DATE-OBS e.g. '2012-03-25T00:00:03.32' mandatory
      * EQUINOX e.g. 2000.0 mandatory
      * INSTRUME e.g. 'AMBER' mandatory
      * OBJECT e.g. 'ETA Car' mandatory
      * ORIGIN e.g. 'ESO' mandatory

      * OBSERVER e.g. 'N. Nerolf' optional
      * REFERENC e.g. 'Duvert et al. 2014, A&A, XXX, YY' optional

Plus the "commonly used FITS keywords" (http://heasarc.gsfc.nasa.gov/docs/fcg/common_dict.html) below:
      * RA e.g. 102.3 mandatory
      * DEC e.g. -20.5 mandatory
      * RADECSYS e.g. 'FK5' mandatory
      * MJD-OBS e.g. 56243.45 mandatory
      * TELESCOP e.g. 'ESO-VLTI-U134' mandatory
      * EXPTIME e.g. 243 (DIT x NDIT x selection for AMBER) mandatory
      * ELAPTIME e.g. 500 (DIT x NDIT for AMBER) mandatory

      * FILTER e.g. 'JHK' optional
      * GRATING e.g. 'GMR' optional

Plus a few specific keywords:
      * WLEN_CEN e.g 2.2e-6 mandatory
      * WLEN_RNG e.g 0.2e-6 mandatory

      * UTC e.g. 223.7 optional
      * LST e.g. 34520.5 optional
      * PI-COI e.g. 'F. Millour' optional




   * I suggest adding wavelength to your equations 123, and to describe what are differential visibilities and differential phase (drafts of paragraphs are provided)

   * Please add the possibility of "linear" or whatever other keyword you like for putting (non-squared) visibilities in the OI_VIS table. These are provided already by MIDI and might be one day provided by MATISSE

   * There is still (from OIFITS rev 1) an uncertainty of what is exactly INT_TIME: detector exposure time, total integration time or observing time (including overheads)? I think these three different values should be defined in some way 

This one should be explicitely described: is it detector integration time (DIT), total exposure time (DIT x NDIT at ESO instruments), or observing time (i.e. end minus start of exposure)? It is important to know which one is used in the data, as e.g. the observing time is important for looking for baseline smearing effects but exposure time and DIT are important for sensitivities issues.

Comments by-- Main.FlorentinMillour - 15 Jul 2014: add the list of proposed keywords:
Maybe these three information could be described in the new format?
      * EXPTIME for detector frame integration
      * INT_TIME for total exposure time
      * ELAPTIME for observing time
or any combination of these?

   * You seem to have discarded the idea of putting "raw" or "calibrated" keywords, as was suggested in the wiki page?

   * I liked better the proposition of "OI_FLUX" instead of "OI_SPECTRUM" for the flux table

   * the OI_SPECTRUM table should stand if the spectrum is normalized or not (and how). The proposition I made was to put a "SPECREFMAP" or "FLUXREFMAP" matrix the same way as in OI_VIS table

Comments by-- Main.FlorentinMillour - 15 Jul 2014:
   * Is it foreseen to merge at some point the OIFITS and UVFITS formats? This might be mentionned in the conclusion of the paper?

   * Why not putting a complex "VISERR" array similar as "VISDATA"? I would suggest either put everything complex (VISDATA + VISERR) or everyting real (RVISDATA, IVISDATA, RVISERR, IVISERR)

   * One needs to know if the spectrum is normalized (and how). Getting flux units with the TUNITn keyword is not sufficient to describe the spectrum. Examples of obtained spectra are MIDI spectra provided in Jy, and AMBER spectra with the continuum normalized to 1. See discussion in the wiki page and propositions.

Best wishes,
Florentin

Comments by-- Main.GillesDuvert - 01 Jul 2014
Updated current version with some of Florentin's helpful comments (thanks Florentin!). Left above points open for discussion. The text of all comments by Florentin has been distributed by Mail and is available upon request.
---++ List of Persons interested in this discussion.
By adding your !WikiName here (see How to Register below) you express interest in participating to a future phone conference on this subject.

Main.GillesDuvert, Main.JohnYoung, Main.JuwePott, Main.ChristianHummel, Main.LaurentBourges, Main.JohnMonnier, Main.LeonardBurtscher, Main.FlorentinMillour, Main.PhilippeBerio, Main.MichelTallon, Main.AntonySchutz, Main.SylvestreLacour, Main.JeanBaptisteLeBouquin, Main.FabienBaron, Main.BrianKloppenborg, Main.EricThiebaut, Main.MikeIreland, Main.PaulBoley
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
pdf	oifits_norm_v2.pdf	r27 r26 r25 r24 r23	manage	170.1 K	2017-09-05 - 08:08	GillesDuvert	update as of 4 september 2017
This topic: Jmmc > WebHome > OIFITSTwoProject
Topic revision: r82 - 2017-09-05 - GillesDuvert