(r13) OiDbPionier < Jmmc/Software

Tags: view all tags
%TOC%

---+ Overview

The PIONIER L2 data comes from OIFITS files provided by Jean-Baptiste !LeBouquin on a regular basis. They are stored on jmmc server behind a webserver for distribution. 

The files are processed by the !OIFitsViewer utility from the JMMC oitools package and the resulting metadata are submitted to the application as an XML file. 

One htaccess is generated (should be every day) to restrict access to files younger than 1 year.

---+ Data extraction

---++ Base metadata

!OIFitsViewer is a tool that parses and analyzes the submitted files to dump an XML representation of their contents. It has been modified to also output the metadata relevant to this application, see the 'metadata' element in the output of every run.

It is run on a local machine.

Example command line using !OIFitsViewer for a set of OIFits files:
<verbatim>
$ (echo "<collection>" ; java -jar oitools.jar fr.jmmc.oitools.OIFitsViewer *.fits ; echo "</collection>") > collection.xml
</verbatim>

---++ Metadata postprocessing

The metadata previously extracted from OIFITS with oitools have to be locally processed and completed to validate as XML granules.

Not every items of a granule is available in the OIFits file: the dataset description, the curation info and the access data have to be added after the extraction.

Example file for a set of XML granules:
<verbatim>
<granules>
  <!-- GRANULE 1 -->
  <granule>
    <target_name>DELTA_CAP</target_name>
    <s_ra>326.761676</s_ra>
    <s_dec>-16.12737</s_dec>
    <t_exptime>103234.55859375</t_exptime>
    <t_min>55496.08832679422</t_min>
    <t_max>55496.17958541475</t_max>
    <em_res_power>0.0</em_res_power>
    <facility_name>VLTI</facility_name>
    <instrument_name>PIONIER_Pdown(1.5994200/1.7627500)</instrument_name>
    <nb_vis>0</nb_vis>
    <nb_vis2>30</nb_vis2>
    <nb_t3>20</nb_t3>
    <nb_channels>5</nb_channels>
    <em_min>1.59942E-6</em_min>
    <em_max>1.76275E-6</em_max>
    <access_url>http://apps.jmmc.fr/oidata/pionier/jun2014/2010-10-26_SCI_DELTA_CAP_oiDataCalib.fits</access_url>
    <access_estsize>46</access_estsize>
    <access_format></access_format>
    <obs_collection>pionier_jun2014</obs_collection>
    <obs_creator_name>Jean-Baptiste Le Bouquin</obs_creator_name>
    <calib_level>2</calib_level>
    <data_rights>secure</data_rights>
    </granule>
  <granule>
  <!-- GRANULE 2 -->
  <granule>
    ...
  <granule>
  ...
</granules>
</verbatim>

Each update, a new timestamped collection is created (for example pionier_sep2014).

Example collection file:
<verbatim>
<collection>
    <name>PIONIER September 2014</name>
    <title>PIONIER L2 data synchronized from September 2014</title>
    <description>
        Replaces pionier June 2014 after cleanup in the reduction archive.
    </description>
</collection>
</verbatim>

---+ Data submission

The granules are parsed from a XML granule file and saved in the database. At the moment they are not extensively checked or validated.

---++ Interactive Web form

The user can submit the XML file with granules through the HTML form of the application ([[http://oidb.jmmc.fr/submit.html]]).

Note: this web form does not provide a way to register the collection to associate the granules to.

---++ REST endpoint

The XML granule file can be sent to the !OiDB REST endpoints for granules (POST on =http://oidb.jmmc.fr/restxq/oidb/granules=).

The collection can be registered through another OiDB REST endpoints (PUT on =http://oidb.jmmc.fr/restxq/oidb/collection/COLLECTIONID=).

Both operations require authentication (supports HTTP Basic Access Authentication).

For example, on a command line with [[http://curl.haxx.se/][cURL]]:
<verbatim>
# create a new collection (if not yet present)
curl --user <username>:<password> --upload-file collection.xml http://oidb.jmmc.fr/restxq/oidb/collection/<collectionId>

# push data from granules.xml file (the preceding '@' is important)
curl --user <username>:<password> --data @granules.xml -H 'Content-type:application/xml' http://oidb.jmmc.fr/restxq/oidb/granule
</verbatim>

---+ Usage

A more sophisticated collection of scripts is in developpment. It goes throught following process steps:
   * synchronize whole oifits collection from the datareduction workstation
   * compute md5sums
   * extract metadata (!OIFitsViewer java class)
   * reduce to an oidb granule ( with progid information )
   * datapi is filled on the server side using the ESO archive web portal/service (TO be done on the submission side)
 Then all computed granules are uploaded on the oidb server.

After a submission and on a regular time basis, the .htaccess is retrieved next to the oifts files.
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
sh	genMeta.sh	r2 r1	manage	1.1 K	2014-07-28 - 07:16	PatrickBernaud	Shell script to process PIONIER OIFITS
Topic revision: r13 - 2015-01-09 - GuillaumeMella