(r3) OiDbSpecs < Jmmc/Software

Tags: view all tags
%TOC%

---++ Data Model

The storage of the data will follow the IVOA's Observation Data Model Core ([[http://www.ivoa.net/documents/ObsCore/][ObsCore]]). Most of the domain specific data have been mapped to this data model (see [[http://www.jmmc.fr/twiki/pub/Jmmc/Software/OiDbReunion26June2013/definition_oidb.pdf][User requirements for an Optical interferometry database]]). Other fields still need a definition: =s_fov=, =s_region=, =s_resolution=, =t_resolution=.

Optional elements of the data model are used such as curation metadata for identification of collections and publications in relation with the data. It is also used in managing the availability of the data (public or private data and release of the data on a set date).

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/553][553]] | Conform to the !ObsCore Data Model for registration as a service in VO |  0 |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/554][554]] | Provide a unique identifier for each data |  0 |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/554][555]] | Define and document extensions to !ObsCore Data Model |  0 |

<!-- dataproduct_type: lowercase, enum, use _subtype for details -->

---+++ OIFits files

Specific fields are extracted from submitted OIFits files. The base model is extended with fields for counts of the number of measurements (VIS, VIS2 and T3) and the number of channels.

An observation from an OIFits file is defined as a row per target name (OI_TARGET row) and per instrument used (OI_WAVELENGTH table, INSNAME keyword): if measurements for a single target are made with two instruments, the result two rows with different instrument names.

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/558][558]] | Extract as much as possible from OIFits files |  0 |

---+++ Other data

Metadata from other sources may not have all fields filled, depending on the calibration level, data availability, and source database format.

<!-- access_url for VEGA data? -->

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/557][557]] | Provide system to upload from external databases |  0 |


---++ Data submission

Depending on the calibration level of the original data, submission process is different.

---+++ L0 data

Data submission for L0 data is based on submission of an XML file (format to be defined) by the user or on scripts regularly downloading observations from the instrument data base.

<!-- TODO: feed L0 from ESO database -->
<!-- VEGA data, requests on VegaObs DB, exctration of data -->

---+++ Individual L2/L3 data

OIFits files are analyzed and metadata extracted using =OIFitsViewer= from =oitools= (JMMC). Metadata missing from the OIFits file are requested through a Web form on the submission page: collection name, bibliographic reference, data rights...

Alternatively, the user can submit an XML file from on batch process on the client side. The metadata are extracted, formatted according to the file format (to be defined) and submitted for inclusion in the database.

---+++ Collection (L3 data)

OIFits files associated with a publication are automatically retrieved from the associated catalog ([[http://vizier.u-strasbg.fr/viz-bin/VizieR][VizieR Service]]). The data are associated with the specific publication and the author, optionnaly using other web services to access the information.

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/549][549]] | Permit upload of OIFits L3 collection attached to article |  0 |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/550][550]] | Upload of preprocessed metadata |  0 |


---++ Data validation

For consistency of the dataset, data has to be validated as a first level of quality assertion.

Batch submission can be validated using an XML Schema.

Submission based on OIFits file can be validated by the validator from =oitools= (see the =checkReport= element) followed by an XML Schema validation specific to OiDB.

Data failing the validation are rejected and the submitter is notified through the web page or the return value/message of the submit endpoint.

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/551][551]] | Validate submitted material |  0 |

---++ User authentification

User authentification is necessary for data submission. Access to the submission form and submission endpoints are controlled by the existing JMMC authentication service.

The name, email and affiliation can help pre-filling submission form of new data.

This authentication may also be used as control access (authorization/restriction) to specific curated data under the responsability of their dataPIs.

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/552][552]] | Authenticate user for submitting data |  0 |


---++ Data access

---+++ Web portal

The project provides a Web front end for data submission and data consultation.

---++++ Data submission

Forms are provided for injection of data depending on calibration level and data source. OIFits files can be uploaded with additionnal metadata that can not be derived from its contents.

Raw metadata can also be uploaded.

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/556][556]] | Accept private data and enforce access restriction to data |  0 |

---++++ Data browsing

The data from the application are displayed in tables with a default set of columns. User can select which columns to show.

Each observation is rendered as a row in the table and may be link to internal or external resource (see OiDbImplementationsNotes).

The page also proposes a set of filters to build custom requests on the dataset. These filters can be combined to refine the search.

The page also show the availability status of the data.

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/545][545]] | Add a search interface to create complex requests |  0 |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/546][546]] | Let user select displayed columns |  0 |

---++++ Statistics

Basic user interaction is collected as usage statistics : number of download for an observation or a collection.

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/548][548]] | Log activity of users in application |  0 |

---+++ Virtual Observatory

The project will be published as a service in the Virtual Observatory as a data model supported by a TAP service.

See [[http://wiki.ivoa.net/twiki/bin/view/IVOA/PublishingInTheVONew][Publishing Data into the VO]].


---++ Current architecture

<img src="%ATTACHURLPATH%/current_architecture.png"/>

The webapp accepts user requests from the form on the search page. Currently the search interface is limited to a set of pre-defined filters that are not combinable.

The Web portal makes requests to DSA/Catalog (a layer on top of existing database providing IVOA search capabilities) that serialize data to VOTable.

The webapp parses the response (XML VOTable) and formats data as HTML tables for rendering.

When submitting new data, the webapp directly builds and executes SQL queries on the database.

<!-- DSA/Catalog (Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.) -->


---++ Notes

   * [[http://soft.g-vo.org/dachs][DaCHS]]
   * IVOA SSO
   * IVOA !ObsProvDM

---++ Misc

| *Ticket#*                                       | *Description* | *% Done* |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/559][559]] | Check regularly the availability and contents of source data |  0 |
| [[http://trac.jmmc.fr/jmmc-sw/ticket/560][560]] | Build a repository of source data files |  0 |
Topic revision: r3 - 2014-01-22 - PatrickBernaud