Overview

The PIONIER L2 data comes from OIFITS files provided by Jean-Baptiste LeBouquin on a regulary basis. They are stored on jmmc server behind a webserver for distribution. The files are processed by the OIFitsViewer utility from the JMMC oitools package and the resulting metadata are submitted to the application as an XML file. One htaccess is generated (should be every day) to restrict access to files younger than 1 year.

Data extraction

OIFitsViewer is a tool that parses and analyzes the submitted files to dump an XML representation of their contents. It has been modified to also output the metadata relevant to this application, see the 'metadata' element in the output of every run.

It is run on a local machine.

Example command line using OIFitsViewer for a set of OIFits files:

$ (echo "<collection>" ; java -jar oitools.jar fr.jmmc.oitools.OIFitsViewer *.fits ; echo "</collection>") > collection.xml

Metadata postprocessing

Not every piece of data expected by the application is available in the OIFits file. The dataset description, the curation info and the data access data have to be added after the extraction

Example modified collection XML file:

<collection>
  <!-- collection description, added by user -->
  <obs_collection>PIONIER</obs_collection>
  <obs_creator_name>JB</obs_creator_name>

  ...

  <metadata>
    <target>
      <!-- automatically extracted data from file -->
      <target_name>HD_61248</target_name>
      <s_ra>113.915662</s_ra>
      <s_dec>-52.53387</s_dec>
      <t_exptime>179.20000076293945</t_exptime>
      <t_min>55974.06113212319</t_min>
      <t_max>55974.06113212319</t_max>
      <em_res_power>0.0</em_res_power>
      <facility_name>VLTI</facility_name>
      <instrument_name>PIONIER_Pnat(1.5585800/1.8035799)_1</instrument_name>
      <nb_vis>0</nb_vis>
      <nb_vis2>7</nb_vis2>
      <nb_t3>7</nb_t3>
      <nb_channels>7</nb_channels>
      <em_min>1.55858E-6</em_min>
      <em_max>1.80358E-6</em_max>

      <!-- data not available in the OIFits file, added by the user -->
      <access_url>http://...</access_url>
      <calib_level>2</calib_level>
      <data_rights>secure</data_rights>
    </target>
    <!-- more targets... -->
    <target>
   ...
    </target>
  </metadata>
  ...
  <metadata>
    ...
  </metadata>

  ...
</collection>

A script can be written to extract, post-process and submit data automatically.

Data submission

The user can then submit its XML file with metadata through the HTML form of the application (http://apps.jmmc.fr/exist/apps/oidb/submit.html) or from a command line or utility to the REST endpoint.

The metadata are parsed and saved in the database. At the moment they are not extensively checked or validated.

REST endpoint

Submission requires authentication. When using the raw REST endpoint (i.e. not the HTML form) to push new data, the user has to start a new session prior to sending the metadata.

For example, on a command line with cURL:

# authenticate and start session, change <username> and <password>
curl --request POST --cookie-jar cookies -duser=<username> -d password=<password> http://oidb.jmmc.fr/submit.html

# push data from metadata.xml file (the preceding '@' is important)
curl --request POST --cookie cookies --form file=@metadata.xml http://oidb.jmmc.fr/modules/upload-meta.xql

# closing the session
curl http://oidb.jmmc.fr/index.html?logout=true
rm cookies