This page gathers main information about the current design of the
OiDB and any hint that can help its future development. This content is not dedicated to final users but developpers and designers.
You can look on the other
documentation topics.
Components
oiexplorer
OIFitsViewer (from oitools) has been modified to extract granule metadata when serializing an OIFits file to XML.
The output includes one
metadata
element as a child of the
oifits
root element. Each identified granule is serialized as a child
target
element in
metadata
.
The application tries to match the content of the file with the columns of the database. The description of a granule consists of children elements of a
target
. The elements are named after the column of the data model and their contents are values for these columns.
Only a subset of all granule metadata can be extracted from an OIFits.
Example output:
$ java -cp oitools.jar fr.jmmc.oitools.OIFitsViewer -v http://cdsarc.u-strasbg.fr/vizier/ftp/cats/J/A+A/565/A71/fits/NGC1068_ats.oifits
<oifits>
<filename>http://cdsarc.u-strasbg.fr/vizier/ftp/cats/J/A+A/565/A71/fits/NGC1068_ats.oifits</filename>
[...]
<metadata>
<target>
<target_name>NGC 1068</target_name>
<s_ra>40.6697</s_ra>
<s_dec>-0.013370000000000002</s_dec>
<t_exptime>0.018</t_exptime>
<t_min>56195.18644676</t_min>
<t_max>56195.28700231</t_max>
[...]
</target>
<target>
<target_name>NGC 1068</target_name>
<s_ra>40.6697</s_ra>
<s_dec>-0.013370000000000002</s_dec>
<t_exptime>0.018</t_exptime>
<t_min>56190.32167824</t_min>
[...]
</target>
[...]
</metadata>
[...]
</oifits>
oiexplorer-existdb: OIFits Explorer Existdb Extension
oiexplorer-existdb is designed as a Java-based XQuery module for wrapping OIFitsViewer (from oitools/oiexplorer) under the namespace
http://exist.jmmc.fr/extension/oiexplorer.
It is used to check, parse and extract granules from (local or remote) OIFITS files. It provides two functions:
-
oi:to-xml
: Parse OIFits data and return XML serialization as document.
-
oi:check
: Validate data as OIFits data.
Both functions accept either an URL or an XQuery binary document as inputs.
See
Creating XQuery Modules.
Installation
- copy the jar file of the extension to the lib/user directory of eXist-db.
- restart eXist-db.
Usage
See
jmmc-resources's jmmc-oiexplorer module.
jmmc-realm: JMMC Realm for eXist-db
jmmc-realm is an extension to eXist-db for an authenticating realm on top of JMMC's account system.
It is based on the source code of the LDAP realm (
extensions/security/ldap/src/org/exist/security/realm/ldap/
) bundled with eXist-db.
It creates eXist-db users for any JMMC account login in the application.
The JMMC account service is a PHP service on HTTPS at
https://apps.jmmc.fr/account/.
Any time a user tries to authenticate, the extension makes a HTTP POST request to the JMMC authentication service (
checkPassword
action).
When enquiring about user information, the extension makes a HTTP POST request to the JMMC authentication service (
getInfo
action).
All JMMC authenticated user are added to a eXist-db
jmmc
group.
Users with special credentials are also added to groups named after the credentials they own. These users are not added as database administrators (dba) for more security. The application is responsible of elevating specific user priviledge to dba.
Applications can use these groups as control access to resources.
See
Authentication Realms.
Installation
- rename the jar file to
exist-security-jmmc.jar
- copy the jar file to the
lib/extensions
directory of eXist-db
- activate the realm in the security manager in
/db/system/security/config.xml
, setting the path to the undelying authentication service
Notes
Writing and enabling a custom realm for eXist-db is not documented/supported (as of version 2.2).
The created user are not exactly regular eXist-db users: even though they are recognized as valid user, they does not appear in the user lists, nor as member of groups.
According to the documentation, the realms can be chained to provide more than a single authentication authority (JMMC + LDAP for example).
oidb-tap: TAP servlet for OiDB
oidb-tap is an servlet implementing a TAP service based on the
taplib.
It initially was an adapted version of the
TAP Libary v1.0 tutorial. It has since been updated to support the yet unreleased taplib version 2.0.
It can be run in the same servlet engine as eXist-db and relies on an underlying PostgreSQL database. The database contains the description of the tables and columns included in the service (
TAP_SCHEMA).
Installation
See oidb-tap's README for installation and
OiDbImplementationNotes#oidb_tap
jmmc-resources: Shared resources for JMMC's eXist-db apps
jmmc-resources is a XAR package with modules dedicated to interaction with online services (
VizieR,
Simbad,
oival,
ADS) and utilities to be shared by eXist-db applications.
It embeds
oiexplorer-existdb (module
jmmc-oiexplorer
) and JMMC'd jmal library (module
jmmc-astro
).
Installation
Deploy the XAR file from the eXist-db dashboard's package manager.
It is necessary to set the
enable-java-binding
option to
true
in eXist-db configuration file,
conf.xml
(see
Calling Java Methods from XQuery).
oidb-data
oidb-data is a repository for all data of the
oidb application (no code in this collection).
Its content must be backed up.
On deployment a script (see
post-install.xql
) takes care of setting permissions on the documents and sub-collections. Permissions on these are important as they are used to allow and deny operations on resources by the application. For examples:
- the resource dedicated to comments can be read by any one but require authentification with a JMMC account to post new comment.
- the schedule job
schedule-job.xql
is setuid to elevate privileges to dba role of some operations
Contents
collections/
This collection contains description of collections of granules. Each collection is serialized as a document. The owner of the collection is the owner of the document and by default the only person who can edit the collection and add/modify granules of the collection.
Collections are identified by an id that is set on any granule of the collection in its 'obs_collection' column.
Collections have at least a title, description. One or more article can be attached to a collection. Keywords can also be added.
Example collection (attached to a VizieR catalog):
<collection id="J/A+A/545/A130" created="2015-01-21T16:35:51.956+01:00">
<source>http://cdsarc.u-strasbg.fr/viz-bin/Cat?cat=J%2FA%2BA%2F545%2FA130</source>
<name>J/A+A/545/A130</name>
<title>Interferometry of {alpha} Eri (Domiciano de Souza+, 2012)</title>
<description>We used [...]</description>
<last-modified>06-Sep-2012</last-modified>
<article>
<bibcode>2012A&A...545A.130D</bibcode>
<title>Beyond the diffraction limit of optical/IR interferometers. I. Angular diameter and rotation parameters of Achernar from differential phases</title>
<author>Domiciano de Souza, A.</author>
[...]
<pubdate>2012-09-01</pubdate>
<keyword>stars: rotation</keyword>
[...]
</article>
</collection>
Example collection (user collection):
<collection id="afd8efc6-e62e-46ef-bc13-081f70434f64" created="2015-01-21T16:25:53.551+01:00">
<name>mycollection</name>
<title>My Collection</title>
<description>Description of my collection [...]</description>
<keywords>stars: rotation</keywords>
[...]
</collection>
The directory initially has special collections for CHARA (
chara_import.xml
) and VEGA (
vega_import.xml
) L0 imports. These collections have special rights to only allow modification by application admins.
See the
collection.xqm
module of
oidb.
comments/
This collection contains a single resource,
comments.xml
, for user comments on granules. Any JMMC authenticated user read and write this file (to add comments).
Example of comment:
<comment granule-id="25748">
<author>user1@example.com</author>
<date>2015-01-28T09:55:08.27+01:00</date>
<text>Comment</text>
</comment>
The files contains comments on all granules. The comments are threaded and uniquely identified.
Example comments.xml contents
<comments>
<comment id="9d18fb49-025e-4ea0-af8c-2930d61d2e77" granule-id="25748">
<author>user1@example.com</author>
<date>2015-01-28T09:55:08.27+01:00</date>
<text>Comment</text>
<comment id="e3e426cf-1fe5-437c-81f7-86a91f4100c4" granule-id="25748">
<author>user2@example.com</author>
<date>2015-01-28T09:55:15.278+01:00</date>
<text>Reply 1</text>
</comment>
</comment>
[...]
</comments>
See the
rest/comments.xqm
module of the
oidb package.
instruments/
This collection contains models for instruments and instrument modes as a checkout of ASPRO2 configuration files (downloaded when the package is built).
keywords/
This collection contains a single resource,
journals.xml
. This is a preprocessed list of keywords extracted from A&A to be used as possible values for keywords added to granule collections.
log/
This collection contains traces from the application as various XML documents:
- downloads of files attached to granule (
downloads.xml
)
- search queries (
searches.xml
)
- reports of granule uploads (
submits.xml
)
- page requests with errors (
visits.xml
)
See the
log.xqm
module of the
oidb package.
oifits/
This collection is a repository for user uploaded OIFITS files.
When a JMMC authenticated user pushes a set of individual OIFITS files to the service, a sub-collection is created in the
staging/
collection and files are saved into this sub-collection. The user is marked as the owner of the sub collection and the uploaded files.
The permissions on the files control who can access and/or modify the files.
tmp/
This collection is a repository for temporary documents (for example currently used by the CHARA uploader for storing temporarily the source CSV file prior to parsing).
Installation
Deploy the XAR file from the eXist-db dashboard's package manager.
Note: when deploying a XAR, the package manager destroy any existing collection of the same name.
Care must be taken not to overwrite the oidb-data
collection without first saving its content.
This component is the main Web application as a XAR package for eXist-db for the Optical Interferometry DataBase.
It indexes observations logs and OIFITS files as granules.
The application implements the Model-View-Controller pattern. It is built on top of eXist-db's
shared-resources, importing
jQuery for JavaScript and
Bootstrap framework for page layout and controls.
It also depends on the subprojects above.
Datamodel
The datamodel is base on
ObsCore version 1.0 extended with specific columns.
See
OiDbColumns.
The granules and observation logs are stored in a PostgreSQL database. Any other piece of data (collections, comments, ...) are instead saved to the XML database.
Authentication
The
JMMC Realm provide for user authentication against the JMMC's user database. The application makes use of eXist-db's persistent login (
/extensions/modules/src/org/exist/xquery/modules/persistentlogin
): it generates a cookie when the user successfully login to be used with the next request. The application will not request a new authentication if it recognizes the cookie. See
controller.xql
for tests prior to routing the request.
Users with the 'oidb' credential are considered application admins.
Templating
eXist-db provides in the
shared-resources a basic templating system using a map as model. See
templates-helpers.xql
.
The stock system has been extended with:
- new helpers and fixes for upstream functions
- composite models: the stock templating system use model based on map. The new templating function can use an extended model where the values can be maps, XML fragments or sequences
- hierarchical keys: a dotted notation of keys for traversing a composite model
- partial rendering: repeat the import of sub templates to each element of a sequence as value to a model key. Lightweight/limited adaptation of Ruby on Rails'
render
directive.
Note: this module should be moved to
jmmc-resources
for mutualization with other applications.
ADQL
The application runs ADQL queries on an underlying TAP server. The application then displays the results of the query.
An
HTTP query string is translated (
modules/adql.xqm
) to an ADQL query using pre-defined filters and clauses. The filters (
modules/filters.xqm
) are used to build the WHERE clause of the query.
The TAP server returns granule as
VOTable that is parsed and rendered.
Main search form
The user can search the database with a form similar to
trac form for querying tickets: the user enables filters that are combined into a query, the results of which are displayed under the search form. Currently all the filters are combined within a AND operation. Provision in the code have been made for enabling more complex combination (à la trac).
It makes a HTTP POST request to the service with a serialization of the values from the HTML form controls into a query string. That query string is then transformed into a new query string with parameters for each filter (
modules/app.xql
). The browser is then redirected to the same page with the new query string (following the
Redirect after Post design pattern).
Note: TAP does not support fine selection of rows like SQL: there is no equivalent to OFFSET. As a result when paginating results it is necessary to retrieve the rows of the current page plus all the preceding rows. The overhead is minimized by selecting and templating only the rows of interest.
Note: A TAP server may set a limit to the maximum number of rows it sends in response to a query (see
2.7.4 Overflows). If it happens the application will display an overflow message on the current and next pages. It neither affect neither the count of results nor the count of pages.
SAMP
Backoffice
The application provides an administration panel with:
- display of application logs (submits, visits, searches, comments)
- basic usage statistics
- triggers for maintenance operation
The access to this page is restricted to authenticated user with special credential (
oidb
).
Background jobs
Long running, unsupervised operations are ran as background jobs.
eXist-db provides a scheduler for which jobs can be written in XQuery and periodically ran.
Registering new jobs is only possible for users within the 'dba' role. As a consequence, it is necessary to temporarily elevate the privileges of an
OiDB administrator through the evaluation (
modules/backoffice.xql
) of a a setuid job launcher (
modules/backoffice-job.xql
). The execution of this script is limited to
OiDB administrators.
At the moment, it is possible to asynchronously:
- update of the inline documentation from the source Wiki page (
modules/update-doc.xql
)
- upload a CSV of CHARA observation logs (
modules/upload-chara.xql
)
- update the VEGA collection from VegaObs (
modules/upload-vega.xql
)
Additionnaly, the scripts for the jobs above have non-default permissions to prevent direct execution by a regular user. Still, for single shot, one-off execution, it is possible to request the immediate execution of the scripts through an HTTP request. For example:
$ curl -u user@example.com:password -X POST http://oidb.jmmc.fr/modules/upload-vega.xql
and
$ curl -v -u user@example.com:password -X POST --data-binary @data.csv -H 'Content-Type: application/csv' http://oidb.jmmc.fr/modules/upload-chara.xql
REST API
The application provides a REST API for submitting and manipulating objects (granules, oifits, collections, comments). All resource functions use the HTTP response status code with optional message as XML document in body. They are located in modules under the
rest
collection.
The REST API is build on top of a lower-level functions in charge of authorization and data validation (see
modules/collection.xqm
,
modules/granules.xqm
).
It is used by the user interface through AJAX calls and for batch processing, see the
PIONIER L2 update mechanism for example.
It is based on the
restxq.xql
module from
dashboard. As a result any REST request will go through the application controller. The module has been improved for better serialization of resource function outputs.
Note: the
RESTXQ trigger of eXist-db has been evaluated (with version 2.1 and 2.2) but discarded as it does not support the persistent login (no access to request object from within the resource function, mandates HTTP Basic authentication instead) and because of issue with support of binary data in requests.
Granule editor
An HTML form using JavaScript and the REST services of the application let users submit new data from OIFITS files:
- upload of individual OIFITS files
- extraction of granule from OIFITS files
- submission of granules after user review
The communication is based on the exchange of XML fragment between client and server with AJAX on the REST endpoints.
Collection creator
The user can prepare a new collection as a container for the new granules.
Collections for L2 date are not attached to an article. The user can specify keywords.
Collections for L3 data require the bibcode of an article.
A variant of the form allows for submission of special L3 collections for VizieR catalog where the collection content is extracted from a VizieR catalog description file (ReadMe). This file follows the rules of the
Standard Description for Astronomical Catalogues. See
vizier.xql
and
jmmc-resources jmmc-vizier.xql
.
Upload of OIFITS files
Authenticated user can upload local OIFITS files to a
staging directory. The form is making use of JavaScript's
File API and the REST endpoint for OIFITS (
rest/oifits.xqm
). Only files recognized as OIFITS by
oitools are saved.
It also supports the upload of ZIP files: OIFITS files are extracted and saved to the staging area, preserving the directory structure in the archive.
The user is set as the owner of the files.
Alternatively the user can provide URLs of publically accessible files. At the moment the content of these files is not preserved in the repository.
Metadata extraction from OIFITS files
Granules are extracted from uploaded OIFITS files and URL of OIFITS files using
oiexplorer (see
oifits:granules()
in module
oifits.xql
).
While processing the OIFITS files the application validates the contents of the file and the granules according to rules from
bug 607 (Work in progress, very basic test only at the moment).
The granules are then displayed as HTML form elements in a table for user validation.
Collection and granule submission
On submission of the form, the collection is created if necessary using REST operation for registering collections (see module
rest/oifits.xqm
, resource function for the POST or PUT verb). On success the granules are attached to the new collection.
The granule are then serialized as an XML fragment and sent to the REST endpoint saving new granules (see module
rest/granule.xqm
, resource function for the POST verb).
Each granule successfully saved is given an identifier by the application and its display is updated in the table as link to the page with granule details.
An error message is returned for failed granules and the user can fix the granule using the form controls and give another try at saving these granules.
Selectors
For the purpose of user validation, the granule form is using special form controls for groups of related columns in the granule definition.
These selectors are jQuery plugins displaying a dropdown list of possible values as an HTML select element depending and then modifying linked hidden input elements (
resources/scripts/selectors.js
).
The base class is then specialized to provide :
- a target selector that link together the
target_name
, s_ra
and s_dec
columns;
- an instrument selector that links together the
facility_name
and instrument_name
;
- an instrument mode selector that links the
instrument_mode
with the instrument selection.
Data
Input:
- observation logs ( CSV / VOTable )
- files ( OIFits )
- metadata ( xml )
Output:
- Collections ( xml )
- Granules ( xml / sql )
Description
xml data do not have associated xsd. Followin hierarchical structures give a brief overview of the structure:
collection
collection/@id
collection/@created
collection/@updated
collection/source
collection/name
collection/title
collection/description
collection/last-modified
collection/article
collection/article/bibcode
collection/article/title
collection/article/author
collection/article/pubdate
collection/article/keyword
granules
granules/granule
granules/granule/access_estsize
granules/granule/access_format
granules/granule/access_url
granules/granule/calib_level
granules/granule/data_rights
granules/granule/em_max
granules/granule/em_min
granules/granule/em_res_power
granules/granule/facility_name
granules/granule/instrument_name
granules/granule/nb_channels
granules/granule/nb_t3
granules/granule/nb_vis
granules/granule/nb_vis2
granules/granule/obs_collection
granules/granule/obs_creator_name
granules/granule/s_dec
granules/granule/s_ra
granules/granule/target_name
granules/granule/t_exptime
granules/granule/t_max
granules/granule/t_min
More data also are autodocumented onto TBD.
File access permissions
Hosted by OiDb
ExistDb stores oifits (add all non xml) on disk and manages associated permissions using its own permissions.
First protection will follow the unix like protection:
- files are owned by the user who uploaded the data
- addition to one collection requires to get write permission on the collection
- TBD implement handling of embargo period : a scheduler will have to check and update permission to make it public automatically
Provision has been done to deliver these files directly by the apache frontend. This would require to update the apache web server configuration and existd would have to update near the data some .htaccess permission files replacing management by existdb and file access permissions.
Hosted on remote sites
Each data provider is in charge of file delivery protection.
Public repository:
Nothing has to be done by the system.
Current public archive
Private repository
OiDB provides access list files but this requires coordination.
- Pionier data are hosted at OSUG and managed by the JMMC technical team under the responsability of Jean-Baptiste LeBouquin.
On cron runs every day to update the associated .htaccess
e.g.
http://oidb.ujf-grenoble.fr/modules/htaccess.xql?obs_collection=pionier_jan2015
User Interface / Web Portal
- technologies / environment : existdb (xquery, html template) + bootstrap(html) + jquery
- user authentication
Interfaces
External services
- CDS/Simbad
- CDS/Sesame
- ESO/Archive
Internal services
Conventions / Tips / Hints
debug
- javascript uses console.log to help debug in the browser
- console.log() may also be executed in the xquery and watched/analysed in the monex application
tap sql query samples:
- SELECT COUNT(*) as c, t.datapi FROM oidb as t GROUP BY t.datapi ORDER BY c DESC
- SELECT COUNT(*) as c, t.instrument_name FROM oidb as t GROUP BY t.instrument_name ORDER BY c DESC
- SELECT COUNT(*) as c, t.facility_name FROM oidb as t GROUP BY t.facility_name ORDER BY c DESC
schedulded jobs
setuid xquery
Some action are perfomed using setuid bit so that action get admin permissions. If this script fails, no error is thrown.
To help debug, curl can be used directly on the appropriate xql endpoint:
e.g.:
curl -v -uguillaume.mella@obs.ujf-grenoble.fr -X POST --data-binary @CHARA-data-all.csv -H "content-type: application/csv" http://oidb.ujf-grenoble.fr:8080/exist/apps/oidb/modules/upload-chara.xql
one jobs which fails, stay in COMPLETE state but do not deasapear from the list. Please use scheduler:get-scheduled-jobs() and scheduler:delete-scheduled-job($jobsname) to release the situation.