JmmcOiTools < Jmmc/Software

Tags: view all tags
This page gathers the enhancements and current limitations on the JMMC OITools library focused on the OIFITS data handling. 

Its main features are:
   * read / write OIFITS files (V1 / V2 + IMAGE_OI extensions = OImaging)
   * read / write FITS image(s) and cubes (primary HDU, extra Image HDU)
   * data model description: meta-data for keywords, columns, tables
   * in-memory data structures to deal with tables (columns, keywords) and derived columns (hard-coded & expression support)
   * validation (command-line interface + web interface)
   * CLI interface: validation & serialization in TSV / XML formats (used by OIDB)

%TOC%


---+ Useful links

   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiFits][Technical information about OI-FITS format, validator and validation rules defined for OI-FITS data models 1 & 2]]
   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiToolsDocs][Project documentation (JavaDoc link)]]
   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiToolsRoadmap][The roadmap]]
   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiToolsChangelog][ChangeLog for OITools]]
   * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiToolsValidationFeedBack][Validation feedback for OITools]]


---+ Feedback on the OIFits2 standard

   * Rule: Columns that are not in the standard must start with "NS_". What would be the severity of an unknown table column name not starting with "NS_"
      * Answer: Yes I agree. Not using "NS_" is harmless except when designing a later version of the standard.
   * Question: The types for the keywords of the primaryHDU are defined by %s, %d, %f. We should therefore redefine these types in the same way for tables (currently in the form A, I, D, ...).
      * Answer: The correction has been applied to the standard and the new version is available.
   * Question: For the new table OI_INSPOL we have columns MJD_OBS and MJD_END, do we really need the keyword DATE_OBS in addition?
      * Answer: The correction has been applied to the standard and the new version is available. The keyword DATE_OBS is removed from the OI_INSPOL table.
   * Question: For some keywords (AMPTYP, PHITYPE) we have in description no information but only the possible values.
      * Answer: The usefulness of these keywords is explained in the text, as well as the explanation of possible values.



---+ To restore

---++ Improve the OIFITS validator (V1 / V2) 

   * improve the validation results: 
      * extract the severity, origin (table/keyword or column) from the validation message
      * sort & filter validation results

---++ TEST UPDATE NOM.TAM.FITS

---+++ Problem 1: We have multi min/max methodes
   * In several file we have methode for calculate min/max value
   * for exemple : oifitsFile.getMinWavelengthBound() and oiWavelength.getEffWaveMin()
   * We need just 1 method to compute all kind of bound on any kind of mesurements (see also xmlOutputVisitor.appendRecord() )


---+++ Problem 2: We need to change Tam Fits Test for scrict comparison
   * Currently we ignore some error for the strict test or we need to fix that
   * We need to chose the best chose fix or ignore error, depending on severity.

See: oitools/src/test/resources/ref/WriteOIFitsTest.log

See an example of difference in header for raw comparison [[ %ATTACHURL%/diffHeader.res ][ in the attached file ]], describing differences in header usually seen with raw comparison option (Set .level=INFO in file logging.properties).<br>
These diff are displayed in log generated in RAW mode. Look for key string "list srcHeader", "list dstHeader", following lines give content of cards.

   Note: enhancement possible: put cards content in java collection and process automatically the comparison (show only differences), method !TamFitsTest.compareHeader(). 

---++++ Ignore case:
   * ARRAYXYZ: 123.0 vs 123.
   * NAXIS1 = row size different	(taille des String)
   * TFORM = Data format
      * different value   of header card[TFORM2] '6A' <> '16A'
      * different value   of header card[TFORM3] 'I' <> '1I'


---++++ Extra Tables

    Binary Tables that are not in the OIFits standards (V1 or V2) are totally skipped (OIFitsLoader? / OIFitsWriter? )

   * Consequence:
        Copying a FITS file may loose 'unknown' tables (load then write) !
   * Solution:
        When loading such a table, it seems possible to parse its header and create keyword / column descriptors on the fly then load it as usual (GenericTable extends Table)


---++++ Extra Columns in table
   * Try to detect the displaced columns ... => avoid false positives

   1 file: PRODUCT_V838_Mon_1-copy.90-2.53micron_2013-04-15T01_49_24.8034--------0.00000------------inf--------1--------2--------3
   * keywords:
      * OI_VIS Missing header card TTYPE15 was = VISCOVRI
      * OI_VIS Missing header card TFORM15 was = 16D
   * Columns:
      * different number of columns 17 <> 14 in VISCOVRI

   1 file: GRAVITY-copy.2016-01-09T05-37-06_singlesci_calibrated
	
   * different number of columns 16 <> 14 in ASTROMETRIC_DELAY
   * different number of columns 3 <> 2 in POLAR

   * OI_VIS Missing header card TTYPE15 was = ASTROMETRIC_DELAY
   * OI_VIS Missing header card TTYPE15 was = VISCOVRI

   * OI_T3 different number of header card 50 <> 52


---++++ Dimensions (TDIM)

   1 file: AMBER_070409-copy

   * OI_ARRAY Missing header card TDIM5 was = (3)
   * OI_VIS Missing header card TDIM7 was = (20)
   * different values for column[VISDATA]
   * different values for column[VISERR]

See OIFits V2 standard that mentions how to use the TDIM keyword


---+++ Problem 6: Setters API

To easily fill tables (complex keywords, columns), a simple setter API will be implemented:
<pre>
/**
* Define the column value for the given row (index)
*/
(OITable).setColumn(int row, (Type) value)

/**
* Define all column values for the given row (index)
*/
(OITable).setRow(int row, values...)
</pre>



---+ External OIFITS resources

   * https://github.com/jsy1001/oifitslib provides some V2 test files
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
res	diffHeader.res	r1	manage	40.4 K	2018-06-29 - 08:42	VincentJammet	difference in header for raw comparison
Topic revision: r32 - 2018-06-29 - VincentJammet