(r9) OiDbInlineDoc < Jmmc/Software

Tags: view all tags
<!-- This topic is the source of the main oidb web portal -->

---+ OiDb User Manual

Revision : 0.1

Date : June 2014

Authors : [[http://www.jmmc.fr/jmmc_groups.htm#oidbgroup][JMMC oidb Working Group]]

---++!! Table of Content

%TOC%


---++ Introduction

   * The Optical interferometry Database (OiDb) is a service developed by the JMMC to provide astronomers with a convenient solution to query calibrated optical interferometry data (OIFITS format) as well as regularly-updated observation logs obtained with a wide range of interferometric instruments. It relies on Virtual Observatory standards and tools to increase diffusion and operability. Scientific and technical background of OiDb are described in the paper "Haubois et al., 2014, Proceedings of the SPIE, (to be published)".

   * OiDb is currently in development and this version is a prototype. [[http://oidb.jmmc.fr/feedback.html][User&#8217;s feedback]] is very welcome.


---++ Content of the database 

   * The first function of this database is to diffuse OIFITS1.0 files [1]. Following the IVOA nomenclature [2] (see table below) of calibration levels, the first version of the database will disseminate L2/L3 (reduced and calibrated) data but we also consider offering L1 (uncalibrated) data for the future versions of the service. Astronomers interested in keeping up with recent observations of particular sources can also browse an archive of observation logs (L0 metadata). At the moment, the observation logs provided by the web portal are only available for the CHARA/VEGA instrument.

|Data reduction level | Content | What is stored in OiDb  | Access Policy |
|Level 0 (L0)| Raw   | Metadata only (meaning Observation logs)  | Public |
|Level 1 (L1)| Reduced uncalibrated | None (for the first version of the Database) |-|
|Level 2 (L2)| Reduced calibrated | Data and metadata | Embargo period possible|
|Level 3 (L3)| Published  | Data and metadata |Public |


   * [[http://oidb.jmmc.fr/index.html][The home page of OiDb]] shows an overview of the database content. Concepts of dataPIs and granules that we developed for this project are defined below. This page also allows to directly query data by target name or position or to go to the advanced search form. 

   * A collection is defined as a set of granules that are linked together for different reasons. They could be L0 data taken with the same instrument (e.g. VegaObs Import), L2 data (e.g. PIONIER oct2014) or L3 data from an observing campaign including particularly VIZIER catalogs (e.g. J/A+A/544/A91).


---++ dataPIs: how to diffuse data with OiDb (Submit new data)

   * The dataPI is the person in charge of the scientific program that has led the observations and the subsequent production of the calibrated data that will be diffused on the database. Typically, a dataPI is a data provider who logs in to the database to submit his data. For published data (L3 level), the dataPI is the first author of the publication. The dataPI is responsible for the accuracy and consistency of the data and metadata that will be diffused via the database service. For various reasons, dataPIs may want to temporarily restrict the access to their data. We therefore offer the possibility of an embargo period but at the condition that all metadata always remain public. 

   * Once the dataPI is logged in (click on "Register" for the first use), two main submission options are available: through a form on the web portal or with an XML document. The direct upload of OIFITS files from a local device is not offered for the moment. The online data submission process (still in progress) is designed to be in two steps. First, the dataPI submits URLs of OIFITS files that are read and their metadata are extracted. The dataPI also fills a form to add the metadata that are not included in the OIFITS keywords and that are important to describe the data (e.g. instrument mode, contact details, etc). Secondly, a submission report is displayed and tells the dataPI if any field is missing or incorrectly filled. She can modify them to complete the submission. OIFITS files may be alternatively locally processed for metadata extraction and converted to an XML document. This document can be refined with data missing from the source files prior to being uploaded to a special endpoint of the service. Currently this technique is employed for batch processing and uploading the PIONIER L2 dataset to the database. This way of submitting metadata is suitable for large collection of OIFITS files, restricted access files and also non-OIFITS files (provided a suitable parser/extraction tool is written by the dataPI).

   * Once a set of OIFITS files has been successfully submitted, metadata are stored in tables that follow the !ObsCore DM. OIFITS are read, they are not modified by our service so that they remain exactly as they have been submitted. The OIFITS is a convenient &#8220;container&#8221; format but it is not scientifically meaningful per se. For instance, one single file can contain as many observations of many targets at different epochs as one wants. This means the content of an OIFITS file has to be examined and split into relevant granules of information. The following definition of a granule has been adopted for our database:

 1 granule = 1 target / 1 night / 1 instrument mode / 1 OIFITS. The metadata of a granule will often represent a subset of an OIFITS metadata. This implies that several granules might correspond to the same OIFITS file.


   * Rules on the instrument mode: for any granule, the correspondance with the official liste of modes (ASPRO 2 pages) is mandatory for L1/L2 and optionnal for L3. Rules on the coordinates: for L3 data, coordinates are not critical if a bibcode is given but a warning is indicated if they are not valid. For L1,L2, coordinates MUST be valid. If coordinates are null, the oifits submission is rejected. The name of the object is informative 




---++ How the data are processed. Backoffice.



---++ Users: how to query data (Search menu)

   * On the [[http://oidb.jmmc.fr/search.html][search web portal]], the user can query a set of OIFITS files with different parameters: Cone Search[3], observation date, instrument, spectral band,  OIFITS collections name and dataPI name. The user can also select the data reduction levels she's interested in. These parameters are then converted into ADQL[4] queries and searches into the database server via an underlying TAP[5] interface.

   * The results of the search are displayed in a table. Each line of the table represents a granule corresponding to the query. The columns show a selection of the most relevant metadata of our data model. They are chosen so that the user can have a fair idea about the content of the data at one glance. The access_url column provides the user with direct links to download the OIFITS files of her choice. In future versions of the database, the user will be able to configure the columns to display. For more details about a given granule, the user can click on the gear icon located in the first column. The first option &#8220;Details&#8221; lists all the metadata of the data model (see Figure below).  The second option &#8220;View in SIMBAD&#8221; opens the CDS/SIMBAD page using target_name for the identifier query. Finally, the third option &#8220;Paper at ADS&#8221; also redirects to the NASA/ADS webpage using the bibcode. The way to display contact details of the dataPI who submitted a given set of OIFITS is being implemented with the creation of dataPI accounts.





---++ Users: how to operate data
   * Also accessible from the gear icon on the result panel, several interoperability options are enhanced using the SAMP[6] protocol. If no SAMP connection has been started, a notification &#8220;No SAMP connection&#8221; appears. When the user clicks on it, a window pops up and offers to install various applications to analyse the selected OIFITS files via !AppLauncher[7]. A typical use case would be to query and select some OIFITS on the database web portal, send them to the data-discovery application OIFITSExplorer (www.jmmc.fr/oifitsexplorer) and finally to the modelling application LITpro[8] (see figure below). When the user downloads an OIFITS file from the database, she will get one or several granules, including some that might be useless for her. This stems from our definition of granularity and not modifying submitted data. OIFITSExplorer will soon feature the possibility of operating the selected granules only.

* Results panels 1: <br />
     <img src="%ATTACHURLPATH%/results_panels1.png" alt="" width='560' height='228' />


   * For more tools, the JMMC/AppLauncher application conveniently allows the user to connect the web portal to many different applications (JMMC apps, TOPCAT, Cassis, Iris, VOPlot, see figure below). However for the moment, we do not guarantee that OIFITS are correctly supported by all these various VO applications.

* Results panels 2: <br />
     <img src="%ATTACHURLPATH%/results_panels_2.png" alt="" width='563' height='411' />


   * Finally, the user can perform operations on all the results returned by the query with the gear icon located on the table header. He can export all the metadata of the DM for all the granules in a VOTable, download all the OIFITS files at once, or send the whole collections of results for analysis to OIFITSExplorer. See figure below.

  * Results panel 3: <br />
     <img src="%ATTACHURLPATH%/res_panel_3.png" alt="res_panel_3.png" width='614' height='254' />


---+ IMPORTANT RULES: 

---++ Terms of use

All parties must agree to the following terms before consider using the database:

   * To the dataPI's knowledge, all data that are provided to the database are authentic and accurate in the sense that they really are the products of interferometric observations as described by their metadata.


   * The user has a commitment to use the data for public astronomical research only. The user has a commitment to get in touch with the dataPI before presenting (in article, conferences) any work including the exploitation of her data. 

   * The archive staff commits in providing help to users and dataPIs to answer questions and fixing issues related to the database project. In the case where archives are uploaded to the database (not url linked) the archive service commits in making sure the data and metadata remain as they were uploaded.


---++ Credits and citations

   * If OiDb is useful to your research, please cite it in your publications by adding the following sentence in the acknowledgement section: 

"This research has made use of the Jean-Marie Mariotti Center \texttt{OiDb} service \footnote{Available at http://oidb.jmmc.fr}."

   * All publication based on the data taken from the database must give credits and citations that are decided in agreement with the dataPI.


---+ Feedback

OiDb is currently in development and this version is a prototype. [[http://apps.jmmc.fr/exist/apps/oidb/feedback.html][User&#8217;s feedback]] is very welcome.

---+ References

[1]  Pauls, T.A., Young, J.S., Cotton, W.D., & Monnier, J.D., &#8220;A Data Exchange Standard for Optical (Visible/IR) Interferometry&#8221;, PASP, 117, 1255, (2005) 

[2] Louys, M., Bonnarel, F., Schade, D., et al., &#8220;IVOA Recommendation: Observation Data Model Core Components and its Implementation in the Table Access Protocol Version 1.0&#8221;, arXiv:1111.1758, (2011) 

[3] Williams, R., Hanisch, R., Szalay, A., & Plante, R., &#8220;IVOA Recommendation: Simple Cone Search Version 1.03&#8221;, arXiv:1110.0498, (2011) 

[4] Ortiz, I., Lusted, J., Dowler, P., et al., &#8220;IVOA Recommendation: IVOA Astronomical Data Query Language Version 2.00&#8221;, arXiv:1110.0503, (2011)

[5] Dowler, P., Rixon, G., & Tody, D., &#8220;IVOA Recommendation: Table Access Protocol Version 1.0&#8221;,  arXiv:1110.0497, (2011)

[6] Taylor, M.B., Boch, T., Fay, J., Fitzpatrick, M., & Paioro, L.,&#8220;SAMP: Application Messaging for Desktop and Web Applications&#8221;, Astronomical Data Analysis Software and Systems XXI, 461, 279, (2012)

[7] Lafrasse, S., Bourges, L., & Mella, G.,&#8221;SAMP App Launcher - An on-demand VO application starter by JMMC&#8221;, Astronomical Data Analysis Software and Systems XXI, 461, 379, (2012)

[8] Tallon-Bosc, I., Tallon, M., Thiébaut, E., et al,&#8221;LITpro: a model fitting software for optical interferometry&#8221;, Proceedings of the SPIE, 7013, (2008)
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
png	res_panel_3.png	r1	manage	44.5 K	2014-06-23 - 15:04	XavierHaubois
png	results_panels1.png	r1	manage	52.9 K	2014-06-23 - 13:57	XavierHaubois
png	results_panels_2.png	r1	manage	95.3 K	2014-06-23 - 13:57	XavierHaubois
Topic revision: r9 - 2014-11-23 - XavierHaubois