(r9) OiDbPionier < Jmmc/Software

Tags: view all tags
---+ Overview

The PIONIER L2 data comes from OIFITS files provided by Jean-Baptiste !LeBouquin on a regulary basis. They are stored on jmmc server behind a webserver for distribution. The files are processed by the !OIFitsViewer utility from the JMMC oitools package and the resulting metadata are submitted to the application as an XML file. One htaccess is generated (should be every day) to restrict access to files younger than 1 year.

---+ Data extraction

!OIFitsViewer is a tool that parses and analyzes the submitted files to dump an XML representation of their contents. It has been modified to also output the metadata relevant to this application, see the 'metadata' element in the output of every run.

It is run on a local machine.

Example command line using !OIFitsViewer for a set of OIFits files:
<verbatim>
$ (echo "<collection>" ; java -jar oitools.jar fr.jmmc.oitools.OIFitsViewer *.fits ; echo "</collection>") > collection.xml
</verbatim>


---+ Metadata postprocessing

The XML granule format that OiDB expects consists in a document containing one or more 'granule' element. The names of the children elements are column names and their texts are their respective values.

The metadata previously extracted from OIFITS with oitools have to be locally processed and completed before being formatted as XML granules. 

Not every piece of data expected by the application is available in the OIFits file. The dataset description, the curation info and the data access data have to be added after the extraction

Example file for a collection of XML granules:
<verbatim>
<collection>
  <!-- GRANULE 1 -->
  <granule>
    <target_name>DELTA_CAP</target_name>
    <s_ra>326.761676</s_ra>
    <s_dec>-16.12737</s_dec>
    <t_exptime>103234.55859375</t_exptime>
    <t_min>55496.08832679422</t_min>
    <t_max>55496.17958541475</t_max>
    <em_res_power>0.0</em_res_power>
    <facility_name>VLTI</facility_name>
    <instrument_name>PIONIER_Pdown(1.5994200/1.7627500)</instrument_name>
    <nb_vis>0</nb_vis>
    <nb_vis2>30</nb_vis2>
    <nb_t3>20</nb_t3>
    <nb_channels>5</nb_channels>
    <em_min>1.59942E-6</em_min>
    <em_max>1.76275E-6</em_max>
    <access_url>http://apps.jmmc.fr/oidata/pionier/jun2014/2010-10-26_SCI_DELTA_CAP_oiDataCalib.fits</access_url>
    <access_estsize>46</access_estsize>
    <access_format></access_format>
    <obs_collection>pionier_jun2014</obs_collection>
    <obs_creator_name>Jean-Baptiste Le Bouquin</obs_creator_name>
    <calib_level>2</calib_level>
    <data_rights>secure</data_rights>
    </granule>
  <granule>
  <!-- GRANULE 2 -->
  <granule>
    ...
  <granule>
  ...
</collection>
</verbatim>

A more sophisticated collection of scripts is in developpment. It goes throught following process steps:
   * synchronize whole oifits collection from the datareduction workstation
   * check md5sums
   * extract metadata (OIFitsViewer java class)
   * reduce to an oidb granule (provide progid )
   * obs_creator_name is filled on the server side using the ESO archive web portal/service (TO be done on the submission side)
 Then all computed granules are uploaded on the oidb server.

After a submission and on a regular time basis, the .htaccess is retrieved next to the oifts files.
   

---+ Data submission

The user can then submit the XML file with granules through the HTML form of the application ([[http://apps.jmmc.fr/exist/apps/oidb/submit.html]]) or from a command line or utility to the REST endpoint.

The granules are parsed and saved in the database. At the moment they are not extensively checked or validated.


---++ REST endpoint

Submission requires authentication. When using the raw REST endpoint (i.e. not the HTML form) to push new data, the user has to start a new session prior to sending the granules.

For example, on a command line with [[http://curl.haxx.se/][cURL]]:
<verbatim>
# authenticate and start session, change <username> and <password>
curl --request POST --cookie-jar cookies -duser=<username> -d password=<password> http://oidb.jmmc.fr/submit.html

# push data from granules.xml file (the preceding '@' is important)
curl --request POST --cookie cookies --data @granules.xml -H 'Content-type:application/xml' http://oidb.jmmc.fr/modules/upload-granules.xql

# closing the session
curl http://oidb.jmmc.fr/index.html?logout=true
rm cookies
</verbatim>
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
sh	genMeta.sh	r2 r1	manage	1.1 K	2014-07-28 - 07:16	PatrickBernaud	Shell script to process PIONIER OIFITS
Topic revision: r9 - 2014-09-24 - GuillaumeMella