Tags:
view all tags
%TOC% ---++ Data Model The storage of the data will follow the IVOA's Observation Data Model Core ([[http://www.ivoa.net/documents/ObsCore/][ObsCore]]). Most of the domain specific data have been mapped to this data model (see [[http://www.jmmc.fr/twiki/pub/Jmmc/Software/OiDbReunion26June2013/definition_oidb.pdf][User requirements for an Optical interferometry database]]). Other fields still need a definition: =s_fov=, =s_region=, =s_resolution=, =t_resolution=. Optional elements of the data model are used such as curation metadata for identification of collections and publications in relation with the data. It is also used in managing the availability of the data (public or private data and release of the data on a set date). | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/553][553]] | Conform to the !ObsCore Data Model for registration as a service in VO | 0 | | [[http://trac.jmmc.fr/jmmc-sw/ticket/554][554]] | Provide a unique identifier for each data | 0 | | [[http://trac.jmmc.fr/jmmc-sw/ticket/554][555]] | Define and document extensions to !ObsCore Data Model | 0 | <!-- dataproduct_type: lowercase, enum, use _subtype for details --> ---+++ OIFits files Specific fields are extracted from submitted OIFits files. The base model is extended with fields for counts of the number of measurements (VIS, VIS2 and T3) and the number of channels. An observation from an OIFits file is defined as a row per target name (OI_TARGET row) and per instrument used (OI_WAVELENGTH table, INSNAME keyword): if measurements for a single target are made with two instruments, the result two rows with different instrument names. | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/558][558]] | Extract as much as possible from OIFits files | 0 | ---+++ Other data Metadata from other sources may not have all fields filled, depending on the calibration level, data availability, and source database format. <!-- access_url for VEGA data? --> | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/557][557]] | Provide system to upload from external databases | 0 | ---++ Data submission Depending on the calibration level of the original data, submission process is different. ---+++ L0 data Data submission for L0 data is based on submission of an XML file (format to be defined) by the user or on scripts regularly downloading observations from the instrument data base. <!-- TODO: feed L0 from ESO database --> <!-- VEGA data, requests on VegaObs DB, exctration of data --> ---+++ Individual L2/L3 data OIFits files are analyzed and metadata extracted using =OIFitsViewer= from =oitools= (JMMC). Metadata missing from the OIFits file are requested through a Web form on the submission page: collection name, bibliographic reference, data rights... Alternatively, the user can submit an XML file from on batch process on the client side. The metadata are extracted, formatted according to the file format (to be defined) and submitted for inclusion in the database. ---+++ Collection (L3 data) OIFits files associated with a publication are automatically retrieved from the associated catalog ([[http://vizier.u-strasbg.fr/viz-bin/VizieR][VizieR Service]]). The data are associated with the specific publication and the author, optionnaly using other web services to access the information. | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/549][549]] | Permit upload of OIFits L3 collection attached to article | 0 | | [[http://trac.jmmc.fr/jmmc-sw/ticket/550][550]] | Upload of preprocessed metadata | 0 | ---++ Data validation For consistency of the dataset, data has to be validated as a first level of quality assertion. Batch submission can be validated using an XML Schema. Submission based on OIFits file can be validated by the validator from =oitools= (see the =checkReport= element) followed by an XML Schema validation specific to OiDB. Data failing the validation are rejected and the submitter is notified through the web page or the return value/message of the submit endpoint. | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/551][551]] | Validate submitted material | 0 | ---++ User authentification User authentification is necessary for data submission. Access to the submission form and submission endpoints are controlled by the existing JMMC authentication service. The name, email and affiliation can help pre-filling submission form of new data. This authentication may also be used as control access (authorization/restriction) to specific curated data under the responsability of their dataPIs. | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/552][552]] | Authenticate user for submitting data | 0 | ---++ Data access ---+++ Web portal The project provides a Web front end for data submission and data consultation. ---++++ Data submission Forms are provided for injection of data depending on calibration level and data source. OIFits files can be uploaded with additionnal metadata that can not be derived from its contents. Raw metadata can also be uploaded. | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/556][556]] | Accept private data and enforce access restriction to data | 0 | ---++++ Data browsing The data from the application are displayed in tables with a default set of columns. User can select which columns to show. Each observation is rendered as a row in the table and may be link to internal or external resource (see OiDbImplementationsNotes). The page also proposes a set of filters to build custom requests on the dataset. These filters can be combined to refine the search. The page also show the availability status of the data. | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/545][545]] | Add a search interface to create complex requests | 0 | | [[http://trac.jmmc.fr/jmmc-sw/ticket/546][546]] | Let user select displayed columns | 0 | ---++++ Statistics Basic user interaction is collected as usage statistics : number of download for an observation or a collection. | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/548][548]] | Log activity of users in application | 0 | ---+++ Virtual Observatory The project will be published as a service in the Virtual Observatory as a data model supported by a TAP service. See [[http://wiki.ivoa.net/twiki/bin/view/IVOA/PublishingInTheVONew][Publishing Data into the VO]]. ---++ Current architecture <img src="%ATTACHURLPATH%/current_architecture.png"/> The webapp accepts user requests from the form on the search page. Currently the search interface is limited to a set of pre-defined filters that are not combinable. The Web portal makes requests to DSA/Catalog (a layer on top of existing database providing IVOA search capabilities) that serialize data to VOTable. The webapp parses the response (XML VOTable) and formats data as HTML tables for rendering. When submitting new data, the webapp directly builds and executes SQL queries on the database. <!-- DSA/Catalog (Layer on top of existing database providing Cone Search, (proto) TAP, and other capabilities, including IVOA Single Sign On authentication. However, TAP implementation is out of date, and software is no longer maintained.) --> ---++ Notes * [[http://soft.g-vo.org/dachs][DaCHS]] * IVOA SSO * IVOA !ObsProvDM ---++ Misc | *Ticket#* | *Description* | *% Done* | | [[http://trac.jmmc.fr/jmmc-sw/ticket/559][559]] | Check regularly the availability and contents of source data | 0 | | [[http://trac.jmmc.fr/jmmc-sw/ticket/560][560]] | Build a repository of source data files | 0 |
Edit
|
Attach
|
Watch
|
P
rint version
|
H
istory
:
r22
|
r5
<
r4
<
r3
<
r2
|
B
acklinks
|
V
iew topic
|
Raw edit
|
More topic actions...
Topic revision: r3 - 2014-01-22
-
PatrickBernaud
Home
Site map
Jmmc web
Faq web
ProspectiveHRA2014 web
Software web
VltiSchool2010 web
VltiSchool2013 web
VltiSchool2015 web
VltiSchool2018 web
Main web
Sandbox web
DeuxiemePage web
TWiki web
Software Web
Create New Topic
Index
Search
Changes
Notifications
RSS Feed
Statistics
Preferences
P
View
Raw View
Print version
Find backlinks
History
More topic actions
Edit
Raw edit
Attach file or image
Edit topic preference settings
Set new parent
More topic actions
Account
Log In
Register User
Edit
Attach
Copyright © 2008-2025 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki?
Send feedback