JmmcOiToolsRoadmap < Jmmc/Software

---+ OITools Roadmap

Updated: 18/12/2017


In progress:
   * Project documentation
   * !OIFitsValidator enhancements
      * Severity profil
      * Filter
      * Message Viewer

In Study:
   * New features for OITOOLS 2
   * Nom.tam.fits update
   * Github

Done:
   * Personal presentation
   * New features for OITOOLS 2 : Units


%TOC%



---+ *<u>Personal presentation</u>*

Prepare the presentation of my work on OITools and the explanations of my involvement. A link to this presentation will be available after this one (9/11) 



---+ *<u>Project documentation</u>*

   * Complete the changelog & documentation wiki pages
      * [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/JmmcOiToolsDocs][Project documentation]]
   * API Usage Tutorial ?
   * Javadoc
   * Complete / create new schema ?




---+ *<u>New features for OITOOLS</u> 2*

---++ Units: [DONE]

Description:
   * Units handling: In OIFITS V2, the physical units for few columns are 'user-defined' i.e. given in the column description (TUnit keyword). OITools already parse column keywords but does not store such units in the column meta data
   * Impacted columns are:
      * OI_VIS table: RVIS (RVIS_ERR), IVIS (IVIS_ERR) and VISDATA (VISERR)
      * OI_FLUX table: FLUXDATA

Study:
   * Currently units are implemented by an Enumeration that is not extendable ! 
   * Possible choices:
      * Realize an enum units more permissive, in this case it will be necessary to fix the validation of the units (!OIFitsLoader.parseColumn ()), choose where are store these 'special' units because this one will serve again. They are reused in the data display under !OIFitsExplorer as well as retrieve them for writing files with !WriteOIFits.
      * or proceed to a rewrite of the enumeration units in class which would allow us to be directly more permissive and to directly manage the case of these units which can vary. They will directly store and manage here.

Impacts:
   * unit parsing & validation : !OIFitsLoader
   * !OIFitsExplorer : Display of column units
   * writing or rewriting files

Feedback:
   * The choice that has been made is to change the operation of Units (creating subclass) and to treat the case of special units in the loading of the file and the validation. The units are recover directly from OIFits files.
   * !FitUnit reporting units for images, Merge Units and !FitsUnit ?

---++ Checksum/Datasum

TODO: description & study


---++ Management of dimensions (NWaves)

TODO: description & study


---++ Extra table/column

TODO: study


---++ OI_INSPOL rules

TODO: description & study




---+ *<u>OIFitsValidator enhancements: [IN PROGRESS]</u>*

Description:

   1 An error message more provided of information and more explicit. As well as a clearer display and more exploitable for users (filter).
   2 Need to determine severity with profiles
   3 Setting up a filter, which would allow us not to display redundant or not important messages (and to group them together)
   4 The ability to re-use error messages (xml)
   5 Update test to compare the validation results.

Study:

   * JAVA object creation for the message: Rules, Origine, Severity. These objects contain all the information useful for the construction of the message.
      * Creation of an error object, containing all the necessary information for the different uses of the error display. (GUI, XML, ..)
      * 

   * M.V.C. idea:
      * Experiment: nosql appreach (Object/SQL) => granule.
      * !DataSelection => visual. OIXP (colors / select).
      * JTable
         * Process validation messages as a FITS table: with keyword and column.
      * Export: CSV, votable, FITS export.
      * Import CSV => plot custom
      * TOMCAT (integral)


   * !FileRef and !HDURef: In order to retrieve informations (fileName, extName, extNb) between our classes to be able to use them in the construction of our error. The idea was to create objects listing these informations !FileRef and !HDURef.
      * !FileRef: !FileRef is an object that retrieves informations from the file: fileName and absoluteFilePath. These information was originally contained in !FitsImageFiles, which now uses !FileRef to retrieve these informations.
      * !HDURef: !HDURef is an object that retrieves informations from the file: extName and extNb. These informations was originally contained in !FitsHDU, which now uses !HDURef to retrieve these informations.
   * The creation of these two objects was intended to link all these informations and make them available throughout the structure. Indeed, the idea was that !HDURef itself calls !FileRef and !FitsHDU and all classes derived from it have this access.
   * The main problem is that !FitsHDU could not directly reference !FitsImageFile because it would have been too heavy (in the reuse of objects created later using !FitsHDU).
   * After consultation it turned out that the object !HDURef would be too restrictive to set up (work and call in the constructor of all chrild classes) and reference to the !FileRef complicated to set up (recovery from the loader). This object has been abandoned for the moment.
   * In the end for the link in the retrieval of all the information it was chosen to make a reference to !FileRef at the time of checking the file. !FileRef of the current file is retrieved and registered in the validation of the !OIFits structure, it is here that the error will be built.


   * Format for the !RuleFailure: Request for discussion when choosing the format of the error.
      * ruleFailed(!RuleFail, !DataLocation) : !RuleFail is somehow the key. This is the part that will be checked and compared in order to avoid duplication in creating the error. DataLocation is the part of datas, it is the information that may be redundant depending on the case and repeating itself several times.
      * ruleFailed(Rule, !FileRef, extName, extNb, member).addValueAt(datas). Datas contains here value, row, and colunm.
         * In this case, the most would be not to reduce for nothing the messages which have very different and variable values. (where there is almost never a duplicate)
      * ruleFailed(Rule, !FileRef, extName, extNb, member, value).addValueAt(datas). Datas contains here row, and colunm. And value is part of the key.
         * This case is very interesting for errors that contain many duplicate values (example the name of a star)
      * There is also the possibility of using one or the other depending on the case!
   * It was also important to define all the terms of the message and avoid the constants:
      * Each term given in parameter allows us to refine the display but also to define the severity!

   * Format !RuleFailed:
      * ruleFailed(rule); (7)
      * ruleFailed(rule, HDU); (5, contains 1 with datas)
      * ruleFailed(rule, HDU, member); (78, contains 67 with datas)
      * ruleFailed(rule, extName, extNb) (1)
         * Special case: no hdu information but extname and extNB possible

   * Format !DataLocation: (68)
      * addFixedValue(Value) (4)
         * It's more an informative error format
      * addKeywordValue(Value) (15)
      * addKeywordValue(Value, Expected) (13)
      * addValueAt(Value, Row) (15)
      * addValueAt(Value, Expected, Row) (2)
      * addValuesAt(Value, Limit, Row) (5)
      * addValueAtRows(Value, Row1, Row2) (4)
      * addColValueAt(Value, Row, Col) (7)
      * addColValueAt(Value, Expected, Row, Col) (1)
      * addColValueAt(Value, Row, Col1, Col2) (1)
      * addColValueAt(Value, Row, Col, Detail) (1)

   * Format Informative:
      * Severity => INFO

   * Inventory of the list of messages (String)
      * An inventory of messages (String) has been developed to bring together similar ones and to try to build a consortium.
      * The message inventory allowed the development of the format for !RuleFailed and !Datalocation

Impacts:

   * !OIFitsChecker 
      * in creating error messages and validation
      * to define error profiles 
   * !getCheckReport() is used in !OITools, !Aspro2, !LITpro, !OImaging, !OIFitsExplorer for the error message display
   * !OIFitsValidatorTest test save & compare validation results
   * !OIVal & OIDB to the re-use of xml error messages

Feedback:


---++ 1: Format messages (DONE)

   * checker.sever delete and create new message: pointer to file, row/column, table, value, severity, declare the rule bound... (error list with full info)
   * Creation of an error object with JAVA object: Rules, Origine (HDU, member), Severity and datas (row/colunm, values)
   * ERROR (!Rule, !FileRef, extName, extNb, member, datas)
   * Ongoing discussion about the format of the error and its contents
      * Reflection on the type of object to use for storing the error. Management of duplications.
      * Reflection on all future uses of this message in order to really have all the information that will be useful.
      * At the same time thinking about methods and message retrieval rules. getErrors(filtre, tri)
      * Datas would be a storage object for values, rows, and columns when this information is available. First, define an inventory of messages (String) to gather those who are alike. And define the most convenient storage object to use using these cases.
   * The question that also arises is the management of the message (String) itself. Usefulness of this message and if necessary the creation of this one, is questioned.
      * An inventory of messages (String) has been developed to bring together similar ones and to try to build a consortium.

      * Check: 
         * checker.severe("Invalid value for column '" + this.getName() + "' line " + rowNb + ", found '" + values[r] + "'  should be >= 0");  [position 472:33]	
         * V2:
         * rule.template = "Invalid value for column '{{MEMBER}}' line {{ROW}}, found '{{VALUE}}'  should be >= 0" (constante)

         * toString() {
            *     str = rule.template;

            *     str = str.replaceAll("{{MEMBER}}", member); 
            *     str = str.replaceAll("{{ROW}}", row); 
            *     str = str.replaceAll("{{VALUE}}", value); 

         *     return str;
         * }



Feedback: Format message DONE.
      * Creating an error message for each failure or each data (if present)
      * A pattern for the messages has been created for each Rule (Use {{}} for values can be changes)
      * Use replaceAll(), to clean all text contained in '{{}}' by the true values of failures


---++ 2: Define error profiles 
Ideas:

   * The profiles are lenient, strict, or software.
   * The severity (Off, Info, Warning, Error) should be defined using profiles. This profiles can also be modified at the request of the user.
   * Change in severity depending on the use case:
      * example: If we are on a file whose instrument is AMBER, then the severity changes from warning to error.
      * It would be wise to have a user feedback on this, to have their needs.
   * The user could also define his choice of severity, but this would be more than one filtering function (display, GUI).
   * User could also create his own profile (his own config)
   * The idea would be the creation of a config file in which is defined if a violation of this or that rule is severe or not (severed).


Proposal:

Define the logical rules to choose which config files to use which correctly define the severity profile in the given case:

   * Questions:
      * Software definition ?
      * OI_CORR to treat now or put to OFF?

   * Profile config:
      * We propose to define by default all rules in severity ERROR. Then a configuration file can be loaded and redefine the severity profile as appropriate.
      * A severity profile is a configuration file that can be loaded and analyzed to define the severity of the errors triggered in the code as appropriate.
      * Which fields will be present in this configuration file (what are our needs) ?
      * File format (ASCII, CSV, XML, ...) ?

   * Concrete proposals:
   1 1 JMMC profile:
   * Config File:

   2 3 profiles V1, V2, + software profile (to add in more than another):
   * Config File:
      1 Profile V1 RULE_ID = SEVERITY:
         * COL_UNKNOWN => OFF
         * 
      2 Profile V2 RULE_ID = SEVERITY:
         * COL_UNKNOWN => OFF
         * COL_MANDATORY => WARNING (When MEMBER = FOV /FOV_TYPE)
      3 Software Profile:
         * OI_CORR_* => OFF


   * Actions: Rules to review
      * GENERIC_DATE_OBS_STANDARD => Format (to be less strict)
      * bullet item



---++ 3: Report enhancements:

   * collect, gather & filter all validation failures
   * generate an "User" report


---++ 4: Xml serialization (DONE)

   * Update the xml Output to incorporate messages as xml elements to handle & filter them after in the OIVal & OIDB tools
   * Fix OIVal / OIDB message parsing


!FeedBack: 
      * Creation of an Xml output with display of all possible failures.
      * Display and processing of these outputs by OIVal and OIDB

---++ 5: Message Viewer

   * graphical panel displaying the messages (sort, filter, severity) = new Widget (GUI) (litpro/oixp)
   * User view ideas:
      * (Conteneur)
      * File: blabla.oifits
         * On HDU:
            * On member:
               * Errors list...
               * Errors list...
               * Errors list...

      * Or rule display
      * Rule 1:
         * Fail on File blabla.oifits, on HDU, on member at row ...
         * Fail on File blabla.oifits, on HDU, on member at row ...
      * Rule 2:
         * Fail on File blabla.oifits, on HDU, on member at row ...

   * The ideal would be to be able to go from one to the other.


---+ *<u>Nom.tam.fits update</u>*

TODO: description & study

   * The version of the library that we use at have evolved
      * retrieve and compare versions of !GitHub (the one we use and the last one)
      * feasibility study: see at which level the changes made will impact our code.
      * delete our library and add the latest version


---+ *<u>GitHub</u>*

TODO: description & study

   * You have to open a repository and send our project on !GitHub
   * Licence
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
txt	ErrorsClasser.txt	r1	manage	15.5 K	2017-12-01 - 16:25	CharleenKemps	Inventory of rules and progress
This topic: Jmmc/Software > WebHome > JmmcOiFits > JmmcOiTools > JmmcOiToolsRoadmap
Topic revision: r25 - 2018-01-25 - CharleenKemps