OiDbInlineDoc < Jmmc/Software

Tags: view all tags
<!-- This topic is the source of the main oidb web portal -->

---+ OiDB User Manual

Revision : 0.7

Date : 16 October 2022

Authors : [[http://www.jmmc.fr/jmmc_groups.htm#oidbgroup][JMMC oidb Working Group]]

---++!! Table of Contents

%TOC%

---++ 1. Introduction

   * The Optical interferometry !DataBase (OiDB) is a service developed by the JMMC to provide astronomers with a convenient solution to query calibrated optical interferometry data (OIFITS format) as well as regularly-updated observation logs obtained with a wide range of interferometric instruments. It relies on Virtual Observatory standards and tools to increase diffusion and operability. Scientific and technical background of OiDB are described in the [[http://spie.org/Publications/Proceedings/Paper/10.1117/12.2056977][2014 SPIE proceedings]] [1] and a first year report was published in the [[https://ui.adsabs.harvard.edu/abs/2016SPIE.9907E..3VH/abstract][2016 SPIE proceesdings.]]

   * OiDB is in continuous development. [[http://oidb.jmmc.fr/feedback.html][User’s feedback]] is very welcome.

---++ 2. !DataPIs

   * The dataPI is the person who legitimately diffuses data and metadata via OiDB and who is responsible for their accuracy and consistency.
   * For non-published data (calibration levels L0 and L2, see section 4), the dataPI is typically the person in charge of the scientific program that has led to the observations and the subsequent production of data and metadata.
   * If data have been reduced in another way and/or by another person and that they are worthy to be diffused on OiDB, the dataPI of these re-reduced data is the person who did this new data reduction.
   * For published data (calibration level L3, see section 4), the dataPI is the first author of the publication.

---++ 3. Terms of use: USERS MUST READ

All parties must agree and comply with the following terms of use <u>before using the database</u>. The cookie at the very bottom of the page is meant to officially accept these terms and also to provide the OiDB team with statistics to improve the quality of the service.

   * To the dataPI's knowledge, all data that are submitted to the database are authentic and accurate in the sense that they really are the products of interferometric observations as described by their metadata.

   * The user has a commitment to use data and OiDB services for public astronomical research only.

   * For any unpublished data (i.e. calibration level L2, see next section), the user has an imperative commitment to get in touch with the corresponding dataPI before presenting (in publications or at conferences) any work making use of the data and agree with him/her on the citation/acknowledgment/collaboration policy regarding these data.

   * For any publication or presentation that makes use of previously published data (i.e. calibration level L3, see next section), the user has an imperative and minimum commitment to thank the dataPI and to cite the corresponding original publication.

   * The OiDB staff at JMMC commits in providing help to users and dataPIs to answer questions and fixing issues related to the OiDB project. In the case where archives are uploaded to the database (not url linked) the archive service commits in making sure the data and metadata remain as they were uploaded.

   * If OiDB is useful to your research, please cite it in your publications by adding the following sentence in the acknowledgement section:
<verbatim>
"This research has made use of the Jean-Marie Mariotti Center \texttt{OiDB} service \footnote{Available at http://oidb.jmmc.fr }."</verbatim>

---++ 4. Contents of the database

   * The first function of this database is to diffuse OIFITS 1.0 files [2]. Following the IVOA nomenclature [3] (see table below) of calibration levels. Astronomers interested in keeping up with recent observations of particular sources can also browse an archive of observation logs (L0 metadata). In OiDB 1.0, the observation logs provided by the web portal are available for the VEGA, CLIMB and CLASSIC instruments and VLTI's ones since OiDB 2.0 . The table below presents a summary of the content of the database:

| Data reduction level  | Content | What is stored in OiDB  | Access Policy |
| Level 0 (L0) | Raw  | Metadata only (meaning Observation logs)  | Public |
| Level 1 (L1) | Reduced uncalibrated | Metadata and/or dataone (since OiDB 2.0) | Public, Embargo period possible* |
| Level 2 (L2) | Reduced calibrated | Metadata and/or data | Public, Embargo period possible* |
| Level 3 (L3) | Published  | Metadata and/or data | Public |
*PIONIER L2 data are regularly uploaded to OiDB in an automatic way. Following the conditions set by the PIONIER team, they become publicly available for download one year after the observations took place. For any enquiry, please contact Dr. Jean-Baptiste !LeBouquin ( Jean-Baptiste.Lebouquin AT univ-grenoble-alpes.fr). Private collections can get an additional [[#embargoed_collection][embargo period in respect to the data policy]] .

   * The CDS tool !VizieR has made available online data from published articles for a long time. In order not to short-cut this well-established system, it is preferable that OIFITS corresponding to a recent publication (L3 data) should be made available first on !VizieR.

   * [[http://oidb-beta.jmmc.fr/index.html][The home page of OiDB]] shows an overview of the database content. Concepts of dataPIs and granules that were developed specifically for this project are defined in sections 2. and 5. of this document. This home page also allows users to directly query data by target name/position or to go to the advanced search form.

---++ 5. Granularity and Data Model

   * The OIFITS format is a convenient “container” format but it is not scientifically meaningful per se. For instance, one single OIFITS can contain observations of different targets at different epochs. In order to display conveniently its scientific content, an OIFITS file is examined and split into *elementary pieces of observation, the so-called "granules"*. The following definition for a granule has been adopted:

<div style="text-align: center;">
*1 granule = 1 target / 1 night / 1 instrument mode / 1 OIFITS*
</div>

The metadata of a granule will often represent a subset of an OIFITS metadata. <u>This implies that several granules might correspond to the same OIFITS file.</u>

   * For each granule, a set of metadata is built from the content of the OIFITS file and the submission form. These metadata are characteristic of our data model ( [[http://www.jmmc.fr/twiki/bin/view/Jmmc/Software/OiDbColumns][see metadata table]]) that is based on the IVOA standard !ObsCore[3] and that was extended to better describe optical interferometry data.

   * A *collection* is defined as a set of granules that are linked together for different reasons. They could be L0 data taken with the same instrument (e.g. !VegaObs Import), L2 data (e.g. PIONIER import) or L3 data from an observing campaign (including !VizieR catalogs, e.g. J/A+A/544/A91). Collections are very convenient to sort and easily retrieve observations. The list of collections is directly available from the Search panel. 3 types of collections exist: public, simulation and SUV. Simulation collections correspond to simulations of observations that have been used, for example, for the SPIE Interferometric Imaging Contests. SUV collections are private collections of data that have been reduced using the [[https://www.jmmc.fr/english/the-jmmc/suv/][SUV service]].

---++ 6. Sharing data with OiDB ("Submit new data" menu)

New dataPIs have to click on "Register" to provide JMMC with a name and an email address. WARNING: that email address will be given to any user interested in the dataPI's data. It is therefore important to choose this address carefully. Some people/groups in charge of an instrument might want to use a generic address here (e.g. instrument@facility.country) rather than their own personal email address. On the search result panel, the dataPI's email address is hidden behind a "Mail-to" icon in order to avoid spamming.

Once the dataPI is logged in four submission options are available:

     1) From OIFITS files: Uncalibrated (L1), Calibrated (L2) and Published (L3) OIFITS

A fast 3-step process allows dataPIs to add OIFITS to the database.
   * Step 1: *Upload the files* with 2 possible options: *locally or from URL links.*
First, a syntactic test is performed on the files to check whether they are valid OIFITS files or not. If the test is positive, the message "Successfully uploaded" appears in green after clicking the "Add" button. Otherwise, a useful error message is printed. Zipped files with extension ".zip" are supported (all files must be at the root level). Regarding the name of the file, note that that the following ASCII signs ' #%:?[]` will be transformed into "%XX" (e.g "Test:MIRC.oifits" will be recorded as "Test%3FMIRC.oifits" into the database). Non-ASCII will be also replaced by "%XX%XX". So they better be avoided. The sign "/" is forbidden.

Secondly, next to the name of the file, a blue "information" button lists the syntaxic information, warnings and errors that occurred during the upload. The gear icon allows dataPIs to examine the OIFITS with an application such as OIFITSExplorer via the SAMP protocol (see section 8 of this page). If the previous steps were successful, metadata are extracted and a set of granules (one per line) is created. The dataPI then checks and completes a pre-filled short form to validate and add the metadata that are not included in the OIFITS file but which are fundamental to describe the data (namely the instrument mode, the program ID of the observations and a quality flag).

Upload information: <br />
     <img alt="Information_button.png" height="450" src="%ATTACHURLPATH%/Information_button.png" width="800" />

   * The *"Program ID"* field corresponds to any identifying number of the observing program that led to the data the dataPI submits. It is given by the observatory where the observations were executed (e.g. "94.D-0074" or "94.D-0074(A)" for ESO and "CLASSIC 10" for CHARA). This information is worth-indicating to describe the data (data curation) and eventually provide some external links (e.g. ESO Observation Schedule page for a given run).

   * The *"Target"* column displays the Target name and coordinates taken from OI_Target. Based on the coordinates, close objects found with SIMBAD are suggested in case the Target name and coordinates were approximated.

   * *"Coordinates"*: close objects found by SIMBAD are suggested in the scrolling menu in case the coordinates were approximated. In future versions, for the L2 level, coordinates will have to be valid. If coordinates are null, the OIFITS submission will be rejected (SEVERE error). The name of the object is informative (in the sense that it is not checked).

   * For the *"Instrument"* and *"Instrument mode"* fields, suggestions are presented in scrolling menus. These were made to have a coherent sorting of data in the whole database. Please use preferably the suggestions listed under "Facilities and instruments" for the "Instrument" field. If you do so, the field box will turn green and a list of registered instrument modes will be suggested in the "Instrument mode" field next to it. Select the one that corresponds to the given granule.

   * *Quality flag*: since there is no magical formula to estimate the quality and scientific usefulness of a granule, a quality flag allows the dataPI to mark the overall quality of a granule using the following grades:

<div style="text-align: center;">
       Unknown (default)   

       Trash  

       To be reduced again

       Quick look (risky to publish)

       Science ready

       Outstanding quality
</div>

Quality flags are associated to the =quality_level= metadata with next code : 0:Unknown  1:Trash  2:To be reduced again  3:Quick look (risky to publish)  4:Science ready  5:Outstanding quality

   * Step 2: Create or append the OIFITS to a collection. Collections are very convenient to sort and easily retrieve observations. Here you have the possibility to create a collection, by giving it a short name, title, thorough description and keywords, or append your files to an existing collection. It is highly recommended to use the "Description" field to clearly describe the data and how they were obtained (e.g. information on the reduction software version, reduction parameters, etc). Anything that is relevant to understand the data will increase the probability of re-use by someone else. For L3 data, it is mandatory to give a bibcode (bibliographic code, e.g. 2004A&A...426..279P, can be retrieved on NASA/ADS) as a reference to the publication. In OIDB 1.0, only one bibcode is allowed per collection of files. It is also not possible to append L2 data to a L3 data collection and vice versa.

   * Step 3: just click on "Save" to finish the data submission!

     2) *From a collection of OIFITS files on !VizieR:* interferometric data reported in a publication may be submitted as a !VizieR catalog. Once it is done, the data can be automatically imported to OiDB by giving the name of the catalog. Similarly to the previous submission mode, a form that summarizes the main metadata allows the user to modify Target, Instrument and Instrument mode before validation.

     3) *From a metadata file:* OIFITS files may be alternatively locally processed for metadata extraction and converted to an XML document. This document can be refined with data missing from the source files prior to being uploaded to a special endpoint of the service. Currently this technique is employed for batch processing and uploading the PIONIER L2 dataset to the database. This way of submitting metadata is suitable for large collection of OIFITS files, restricted access files and also non-OIFITS files (provided a suitable parser/extraction tool is written by the dataPI).

     4) *From an external database/service (for administrators):* this submission mode is used by administrators to upload L0 data.

Once a set of OIFITS files has been successfully submitted, metadata are stored in tables that follow the !ObsCore Data Model. OIFITS are only read and are not modified by our service so that *they remain exactly as they have been submitted*.

<a id="embargoed_collection"/>
---+++ Embargoed / protected collections

The "embargoed/protected" collection service is meant to *share data amongst a team* during an embargo period prior to publication. Such data will be made public once the first article using them is published (need manual operation) or upon expiry of the embargo period, whichever comes first. The embargo period of private collections is fixed to 1 year extension after the associated raw data's release_date, after which the PI must either make the collection public or remove it from OiDB. An extension of the embargo period is possible in duly justified cases. OiDB does apply these rules for any new record but respects given release_date if provided on submission.

If you reduced your dataset with the *help of !SUV*, your OIFITS will also get extended release date according to the embargoed/protected collection following the [[http://www.jmmc.fr/suv/][SUV Data protection policy]].

In all cases, *Metadata* associated to your files will *stay public* but you will need to provide your jmmc credentials to get your data until the release_date.

Common proprietary periods for raw data are:
   * [[http://archive.eso.org/cms/eso-data-access-policy.html][1 year for ESO data]]
   * [[http://www.chara.gsu.edu/observers/data-policy-access][18 months for CHARA data]]

<a id="delegated_accesses"/>
Delegations are available on demand to allow a given datapi to share data-access of protected data with another jmmc account. Read access are granted per:
   * collection
   * progid
   * proposal_subid

Some interface will be added to setup such delegation from the OiDB portal. In the meantime, any datapi who wants to share some data [[https://www.jmmc.fr/feedback/?component=OiDB][just has to fill a request]] with proper information.

---++ 7. Querying data (Search)

   * On the [[http://oidb.jmmc.fr/search.html][search web portal]], the user can query a set of OIFITS files with different parameters: Cone Search[4] using target name or coordinates, observation date, instrument, wavelength range, OIFITS collections name and dataPI name. The user can also select the data reduction levels (s)he's interested in. These URL parameters are then converted into ADQL[5] queries (shown in red above the result panel, this query is editable) and the search is executed into the database server via an underlying TAP[6] interface. The results of this query are returned and displayed in a table.

   * The results of the search are displayed in a table. Each line of the table represents a granule corresponding to the query. The columns show a selection of the most relevant metadata of our data model. They are chosen so that the user can have a fair idea about the content of the data at one glance: Target name, access url to directly download the OIFITS (extension can sometimes appear as ".fits" even if they are OIFITS), Time at the beginning of the observations, instrument name, minimum observed wavelength, maximum observed wavelength, number of spectral channels and !DataPI name. Next to the !DataPI name, a "Mail-to" icon links to the dataPI's email address in the user's default mailbox if the dataPIs has a registered JMMC account (otherwise an invitation to do so appears). In future versions of the database, the user will be able to set the columns to be displayed. For debug purpose, the user can manually add a HTTP parameter 'all=y' to the URL to force the display of all columns in the search page. For more details about a given granule, the user can click on the gear icon located in the first column. The first option "Details" lists all the metadata of the data model (see Figure below) and offers the possibility to the user to write a comment on a given granule (about the quality of the granule for example). The second option "View in SIMBAD" opens the CDS/SIMBAD page using target_name for the identifier query. Finally, the third option "Paper at ADS" also redirects to the NASA/ADS webpage using the bibcode (for L3 data only).

---++ 8. Interoperability (Virtual Observatory tools and protocols)
   * Also accessible from the gear icon on the result panel, several interoperability options are enhanced using the SAMP[7] protocol. If no SAMP connection has been started, a notification "No SAMP connection" appears. When the user clicks on it, a window pops up and offers to install various applications to analyse the selected OIFITS files via [[http://www.jmmc.fr/applauncher_page.htm][!AppLauncher&#91;8&#93;]]. A typical use case would be to query and select some OIFITS on the database web portal, send them to the data-discovery application [[http://www.jmmc.fr/oifitsexplorer][OIFITSExplorer]] and finally to the modeling application LITpro[9] (see figure below). When the user downloads an OIFITS file from the database, (s)he will get one or several granules, including some that might be useless for her. This stems from our definition of granularity and not modifying submitted data. OIFITSExplorer will soon feature the possibility of operating the selected granules only.

   * Results panels 1: <br />
     <img alt="" height="228" src="%ATTACHURLPATH%/results_panels1.png" width="560" />

   * For more tools, the [[http://www.jmmc.fr/applauncher_page.htm][JMMC/AppLauncher]] application conveniently allows the user to connect the web portal to many different applications (JMMC apps, TOPCAT, Cassis, Iris, VOPlot, see figure below). However for the moment, we do not guarantee that OIFITS are correctly supported by all these various VO applications.

   * Results panels 2: <br />
     <img alt="" height="411" src="%ATTACHURLPATH%/results_panels_2.png" width="563" />

   * Finally, the user can perform operations on all the results returned by the query with the gear icon located on the table header. He can export all the metadata of the DM for all the granules in a VOTable, download all the OIFITS files at once using curl, or send the whole collections of results for analysis to OIFITSExplorer. See figure below. Take care, credentials are case sensitive !

   * Results panel 3: <br />
     <img alt="res_panel_3.png" height="251" src="%ATTACHURLPATH%/res_panel_3.png" width="424" />

Notes:
   * You may have to enter several times your email/password because data can be hosted on several data storage servers.
   * Please use firefox if your browser does not succeed to connect to SAMP Hub ( OIFitsExplorer or AppLauncher must previously be launched )
   * OIFits downloaded files from OiDB 's generated collections will be cached in =~/OIFitsExplorer= or =~/Documents/OIFitsExplorer=

---++ 9. License

The portal and all its contents including all sorts of data are protected under the Creative Commons License "Attribution-NonCommercial-ShareAlike 4.0 International (CC BY-NC-SA 4.0)". Information can be found [[http://creativecommons.org/licenses/by-nc-sa/4.0/][here]].

---++ 10. References

[1] Haubois, X., Bernaud, P., Mella, G. et al., 2014, [[http://spie.org/Publications/Proceedings/Paper/10.1117/12.2056977][Proc. SPIE 9146]], Optical and Infrared Interferometry IV, 91460O (24 July 2014); doi: 10.1117/12.2056977

[2] Pauls, T.A., Young, J.S., Cotton, W.D., & Monnier, J.D., “A Data Exchange Standard for Optical (Visible/IR) Interferometry”, PASP, 117, 1255, (2005)

[3] Louys, M., Bonnarel, F., Schade, D., et al., “IVOA Recommendation: Observation Data Model Core Components and its Implementation in the Table Access Protocol Version 1.0”, arXiv:1111.1758, (2011)

[4] Williams, R., Hanisch, R., Szalay, A., & Plante, R., “IVOA Recommendation: Simple Cone Search Version 1.03”, arXiv:1110.0498, (2011)

[5] Ortiz, I., Lusted, J., Dowler, P., et al., “IVOA Recommendation: IVOA Astronomical Data Query Language Version 2.00”, arXiv:1110.0503, (2011)

[6] Dowler, P., Rixon, G., & Tody, D., “IVOA Recommendation: Table Access Protocol Version 1.0”,  arXiv:1110.0497, (2011)

[7] Taylor, M.B., Boch, T., Fay, J., Fitzpatrick, M., & Paioro, L.,“SAMP: Application Messaging for Desktop and Web Applications”, Astronomical Data Analysis Software and Systems XXI, 461, 279, (2012)

[8] Lafrasse, S., Bourges, L., & Mella, G.,”SAMP App Launcher - An on-demand VO application starter by JMMC”, Astronomical Data Analysis Software and Systems XXI, 461, 379, (2012)

[9] Tallon-Bosc, I., Tallon, M., Thiébaut, E., et al,”LITpro: a model fitting software for optical interferometry”, Proceedings of the SPIE, 7013, (2008)

[10] Haubois, X., Mella, G., Duvert, G., et al., 2016, Proc. SPIE 9907, 99073V. doi:10.1117/12.2233525
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who
png	Information_button.png	r1	manage	300.7 K	2015-05-21 - 22:03	XavierHaubois
png	res_panel_3.png	r2 r1	manage	39.3 K	2022-10-10 - 13:10	GuillaumeMella
png	results_panels1.png	r1	manage	52.9 K	2014-06-23 - 13:57	XavierHaubois
png	results_panels_2.png	r1	manage	95.3 K	2014-06-23 - 13:57	XavierHaubois
Topic revision: r34 - 2024-04-12 - GuillaumeMella