(r3) OiDbTechDesign < Jmmc/Software

Tags: view all tags
This page gathers main information about the current design of the OiDB and any hint that can help its futur development. This content is not dedicated to final users but developpers and designers.

You can look on the other  [[OiDbDocs][documentation topics]].

<!--
History:

-- Main.GuillaumeMella - 21 Jan 2015
Creation

-->

%TOC%

---+ Components

---+ Data
Input: 
   * observation logs ( CSV / VOTable )
   * files ( OIFits )
   * metadata  ( xml )

Output:
   * Collections ( xml )
   * Granules ( xml / sql )

---++ Description
xml data do not have associated xsd. Followin hierarchical structures give a brief overview of the structure:
   * collections: 
<verbatim>
collection
collection/@id
collection/@created
collection/@updated
collection/source
collection/name
collection/title
collection/description
collection/last-modified
collection/article
collection/article/bibcode
collection/article/title
collection/article/author
collection/article/pubdate
collection/article/keyword
</verbatim>
   * granule:
<verbatim>
granules
granules/granule
granules/granule/access_estsize
granules/granule/access_format
granules/granule/access_url
granules/granule/calib_level
granules/granule/data_rights
granules/granule/em_max
granules/granule/em_min
granules/granule/em_res_power
granules/granule/facility_name
granules/granule/instrument_name
granules/granule/nb_channels
granules/granule/nb_t3
granules/granule/nb_vis
granules/granule/nb_vis2
granules/granule/obs_collection
granules/granule/obs_creator_name
granules/granule/s_dec
granules/granule/s_ra
granules/granule/target_name
granules/granule/t_exptime
granules/granule/t_max
granules/granule/t_min
</verbatim>

More data also are autodocumented onto TBD.


---+++ File access permissions
---++++ Hosted by OiDb
ExistDb stores oifits (add all non xml) on disk and manages associated permissions using its own permissions.

First protection will follow the unix like protection:
   * files are owned by the user who uploaded the data
   * addition to one collection requires to get write permission on the collection
   * TBD implement handling of embargo period : a scheduler will have to check and update permission to make it public automatically

Provision has been done to deliver these files directly by the apache frontend. This would require to update the apache web server configuration and existd would have to update near the data some .htaccess permission files replacing management by existdb and file access permissions.

---++++ Hosted on remote sites
Each data provider is in charge of file delivery protection.

---+++++ Public repository:
Nothing has to be done by the system.

Current public archive 
   * CDS

---+++++ Private repository
OiDB provides access list files but this requires coordination. 

   * Pionier data are hosted at OSUG and managed by the JMMC technical team under the responsability of Jean-Baptiste LeBouquin.

On cron runs every day to update the associated .htaccess

e.g. http://oidb.ujf-grenoble.fr/modules/htaccess.xql?obs_collection=pionier_jan2015

---++ User Interface / Web Portal

   * technologies / environment : existdb (xquery, html template) + bootstrap(html) + jquery
   * user authentication
      * admin group


---+ Interfaces

---++ External services
   * CDS/Simbad
   * CDS/Sesame
   * ESO/Archive


---++ Internal services


---+ Conventions / Tips / Hints

---++ debug
   * javascript uses console.log to help debug in the browser
   * console.log() may also be executed in the xquery and watched/analysed in the monex application

---++ tap sql query samples:
   * SELECT COUNT(*) as c, t.datapi FROM oidb as t GROUP BY t.datapi ORDER BY c DESC 

---++ schedulded jobs

---+++ setuid xquery
Some action are perfomed using setuid bit so that action get admin permissions. If this script fails, no error is thrown.

To help debug, curl can be used directly on the appropriate xql endpoint:
e.g.: 
<verbatim>curl -v -uguillaume.mella@obs.ujf-grenoble.fr -X POST --data-binary @CHARA-data-all.csv -H "content-type: application/csv" http://oidb.ujf-grenoble.fr:8080/exist/apps/oidb/modules/upload-chara.xql</verbatim>

one jobs which fails, stay in COMPLETE state but do not deasapear from the list. Please use scheduler:get-scheduled-jobs() and scheduler:delete-scheduled-job($jobsname) to release the situation.
Topic revision: r3 - 2015-01-23 - GuillaumeMella