JmmcOiTools < Jmmc/Software

This page gathers the enhancements and current limitations on the JMMC OITools library focused on the OIFITS data handling. 

Its main features are:
   * read / write OIFITS files (V1 / V2 + IMAGE_OI extensions = OImaging)
   * read / write FITS image(s) and cubes (primary HDU, extra Image HDU)
   * data model description: meta-data for keywords, columns, tables
   * in-memory data structures to deal with tables (columns, keywords) and derived columns (hard-coded & expression support)
   * validation (command-line interface + web interface)
   * serialization in TSV / XML formats (used by oidb)

%TOC%


---+ Useful links

   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiFits][Technical information about OI-FITS format, OIFTIS validator and rules for OIFTIS DataModel version 1 and version 2]]
      * shift here
   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiTools/Doc][Project documentation (JavaDoc link)]]
   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiTools/Doc][The roadmap]]
   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiTools/ChangeLog][ChangeLog for OITools]]
      * ---+++ Actual TODO in code

   * Finish adding any missing rules
   * take stock of all TODO, point by point
   * TODO: Headercards type is string for Int/float/double too
   * TODO: imageHDU should be returned even if no image to hold keywords -> could rely on the firstOnly flag ?
   * Javadoc fix
   * sticky notes



---+ The roadmap

   * 20/10/2017:
      * Personal presentation: prepare the presentation of my work on OITools and the explanations of my involvement. A link to this presentation will be available after this one (9/11) 
      * Review the project documentation, API Usage Tutorial.
      * Functional addition on version 2 of OITOOLS.
      * Improve the OIFitsValidator test to compare the validation results.
      * Nom.tam.fits update.
      * Github.


---++ Roadmap Detail


---+++ Personal presentation


---+++ Project documentation

   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiTools/Doc][Project documentation]]


---+++ Functional addition on version 2 of OITOOLS

---++++ Units: 

   * 20/10/2017:
   * We are reviewing the declaration of units, currently in the form of an enumeration that does not allow us to create a permissive unit. The columns concerned by this unit are RVIS and IVIS of table OI_VIS and FLUXDATA of table OI_FLUX.
   * We will therefore make a choice:
      * either we realize a new class Units more permissive, in this case it will be necessary to fix the validation of the units (OIFitsLoader.parseColumn ()), choose where we store these 'special' units because this one will reserve us. Be careful because they are reused in the data display under OIFitsExplorer as well as retrieve them for writing files with WriteOifits.
      * or we proceed to a rewrite of the enumeration Units in class which would allow us to be directly more permissive and to directly manage the case of these units which can vary.


---++++ Checksum/Datasum


---++++ Management of dimensions (NWaves)


---++++ Extra table/column


---++++ OI_INSPOL rules



---+++ Improve the OIFitsValidator test to compare the validation results


---++++ Message: 

   * checker.sever delete and create new message: pointer to file, line number, severity, declare the rule bound... (error list with full info)

---++++ Filter message

---++++ Define error profiles 

   * (lenient, strict, software)

---++++ Xml

   * Output of the messages in xml to reuse them after, impact possible on other classes. (oival/oidb)

---++++ Viewer

   * Reflection on a possible graphical panel displaying these new messages. (Widget GUI) (litpro/oixp)



---+++ Nom.tam.fits update

   * We must succeed in saving our code while deleting this library


---+++ Github

   * You have to open a repository and send our project on GitHub




---+ Junit Tests

   * TODO: describe the JUnit tests + file structure
   * TODO: Test update ? (History order) / (Hierarch)
      *    see strict compare Nom.tam.fits



---+ External OIFITS resources

   * https://github.com/jsy1001/oifitslib provides some V2 test files



---+ TO SHIFT

---+++ Improve the OIFITS validator (V1 / V2) 

   * improve the validation results: 
      * extract the severity, origin (table/keyword or column) from the validation message
      * sort & filter validation results


---+++ TEST UPDATE NOM.TAM.FITS

---++++ Problem 1: We have multi min/max methodes
   * In several file we have methode for calculate min/max value
   * for exemple : oifitsFile.getMinWavelengthBound() and oiWavelength.getEffWaveMin()
   * We need just 1 method to compute all kind of bound on any kind of mesurements (see also xmlOutputVisitor.appendRecord() )


---++++ Problem 2: We need to change Tam Fits Test for scrict comparison
   * Currently we ignore some error for the strict test or we need to fix that
   * We need to chose the best chose fix or ignore error, depending on severity.

See: oitools/src/test/resources/ref/WriteOIFitsTest.log

---+++++ Ignore case:
   * ARRAYXYZ: 123.0 vs 123.
   * NAXIS1 = row size different	(taille des String)
   * TFORM = Data format
      * different value   of header card[TFORM2] '6A' <> '16A'
      * different value   of header card[TFORM3] 'I' <> '1I'


---+++++ Extra Tables

    Binary Tables that are not in the OIFits standards (V1 or V2) are totally skipped (OIFitsLoader? / OIFitsWriter? )

   * Consequence:
        Copying a FITS file may loose 'unknown' tables (load then write) !
   * Solution:
        When loading such a table, it seems possible to parse its header and create keyword / column descriptors on the fly then load it as usual (GenericTable extends Table)


---+++++ Extra Columns in table
   * Try to detect the displaced columns ... => avoid false positives

   1 file: PRODUCT_V838_Mon_1-copy.90-2.53micron_2013-04-15T01_49_24.8034--------0.00000------------inf--------1--------2--------3
   * keywords:
      * OI_VIS Missing header card TTYPE15 was = VISCOVRI
      * OI_VIS Missing header card TFORM15 was = 16D
   * Columns:
      * different number of columns 17 <> 14 in VISCOVRI

   1 file: GRAVITY-copy.2016-01-09T05-37-06_singlesci_calibrated
	
   * different number of columns 16 <> 14 in ASTROMETRIC_DELAY
   * different number of columns 3 <> 2 in POLAR

   * OI_VIS Missing header card TTYPE15 was = ASTROMETRIC_DELAY
   * OI_VIS Missing header card TTYPE15 was = VISCOVRI

   * OI_T3 different number of header card 50 <> 52


---+++++ Dimensions (TDIM)

   1 file: AMBER_070409-copy

   * OI_ARRAY Missing header card TDIM5 was = (3)
   * OI_VIS Missing header card TDIM7 was = (20)
   * different values for column[VISDATA]
   * different values for column[VISERR]

See OIFits V2 standard that mentions how to use the TDIM keyword


---++++ Problem 6: Setters API

To easily fill tables (complex keywords, columns), a simple setter API will be implemented:
<pre>
/**
* Define the column value for the given row (index)
*/
(OITable).setColumn(int row, (Type) value)

/**
* Define all column values for the given row (index)
*/
(OITable).setRow(int row, values...)
</pre>
This topic: Jmmc/Software > WebHome > JmmcOiFits > JmmcOiTools
Topic revision: r22 - 2017-10-19 - CharleenKemps