This page gathers main information about the current design of the OiDB and any hint that can help its future development. This content is not dedicated to final users but developpers and designers.

You can look on the other documentation topics.

Components

oiexplorer

OIFitsViewer (from oitools) has been modified to extract granule metadata when serializing an OIFits file to XML.

The output includes one metadata element as a child of the oifits root element. Each identified granule is serialized as a child target element in metadata.

The application tries to match the content of the file with the columns of the database. The description of a granule consists of children elements of a target. The elements are named after the column of the data model and their contents are values for these columns.

Only a subset of all granule metadata can be extracted from an OIFits.

Example output:

$ java -cp oitools.jar fr.jmmc.oitools.OIFitsViewer -v http://cdsarc.u-strasbg.fr/vizier/ftp/cats/J/A+A/565/A71/fits/NGC1068_ats.oifits
<oifits>
<filename>http://cdsarc.u-strasbg.fr/vizier/ftp/cats/J/A+A/565/A71/fits/NGC1068_ats.oifits</filename>
[...]
<metadata>
  <target>
    <target_name>NGC 1068</target_name>
    <s_ra>40.6697</s_ra>
    <s_dec>-0.013370000000000002</s_dec>
    <t_exptime>0.018</t_exptime>
    <t_min>56195.18644676</t_min>
    <t_max>56195.28700231</t_max>
    [...]
  </target>
  <target>
    <target_name>NGC 1068</target_name>
    <s_ra>40.6697</s_ra>
    <s_dec>-0.013370000000000002</s_dec>
    <t_exptime>0.018</t_exptime>
    <t_min>56190.32167824</t_min>
    [...]
  </target>
  [...]
</metadata>
[...]
</oifits>

oiexplorer-existdb: OIFits Explorer Existdb Extension

oiexplorer-existdb is designed as a Java-based XQuery module for wrapping OIFitsViewer (from oitools/oiexplorer) under the namespace http://exist.jmmc.fr/extension/oiexplorer.

It is used to check, parse and extract granules from (local or remote) OIFITS files. It provides two functions:

  • oi:to-xml: Parse OIFits data and return XML serialization as document.
  • oi:check: Validate data as OIFits data.

Both functions accept either an URL or an XQuery binary document as inputs.

See Creating XQuery Modules.

Installation

  • copy the jar file of the extension to the lib/user directory of eXist-db.
  • restart eXist-db.

Usage

See jmmc-resources's jmmc-oiexplorer module.

jmmc-realm: JMMC Realm for eXist-db

jmmc-realm is an extension to eXist-db for an authenticating realm on top of JMMC's account system.

It is based on the source code of the LDAP realm (extensions/security/ldap/src/org/exist/security/realm/ldap/) bundled with eXist-db. It creates eXist-db users for any JMMC account login in the application.

The JMMC account service is a PHP service on HTTPS at https://apps.jmmc.fr/account/.

Any time a user tries to authenticate, the extension makes a HTTP POST request to the JMMC authentication service (checkPassword action).

When enquiring about user information, the extension makes a HTTP POST request to the JMMC authentication service (getInfo action).

All JMMC authenticated user are added to a eXist-db jmmc group.

Users with special credentials are also added to groups named after the credentials they own. These users are not added as database administrators (dba) for more security. The application is responsible of elevating specific user priviledge to dba.

Applications can use these groups as control access to resources.

See Authentication Realms.

Installation

  • rename the jar file to exist-security-jmmc.jar
  • copy the jar file to the lib/extensions directory of eXist-db
  • activate the realm in the security manager in /db/system/security/config.xml, setting the path to the undelying authentication service

Notes

Writing and enabling a custom realm for eXist-db is not documented/supported (as of version 2.2).

The created user are not exactly regular eXist-db users: even though they are recognized as valid user, they does not appear in the user lists, nor as member of groups.

According to the documentation, the realms can be chained to provide more than a single authentication authority (JMMC + LDAP for example).

oidb-tap: TAP servlet for OiDB

oidb-tap is an servlet implementing a TAP service based on the taplib.

It initially was an adapted version of the TAP Libary v1.0 tutorial. It has since been updated to support the yet unreleased taplib version 2.0.

It can be run in the same servlet engine as eXist-db and relies on an underlying PostgreSQL database. The database contains the description of the tables and columns included in the service (TAP_SCHEMA).

Installation

See oidb-tap's README for installation and OiDbImplementationNotes#oidb_tap

jmmc-resources: Shared resources for JMMC's eXist-db apps

jmmc-resources is a XAR package with modules dedicated to interaction with online services (VizieR, Simbad, oival, ADS) and utilities to be shared by eXist-db applications.

It embeds oiexplorer-existdb (module jmmc-oiexplorer) and JMMC'd jmal library (module jmmc-astro).

Installation

Deploy the XAR file from the eXist-db dashboard's package manager.

It is necessary to set the enable-java-binding option to true in eXist-db configuration file, conf.xml (see Calling Java Methods from XQuery).

oidb-data

oidb-data is a repository for all data of the oidb application (no code in this collection). Its content must be backed up.

On deployment a script (see post-install.xql) takes care of setting permissions on the documents and sub-collections. Permissions on these are important as they are used to allow and deny operations on resources by the application. For examples:

  • the resource dedicated to comments can be read by any one but require authentification with a JMMC account to post new comment.
  • the schedule job schedule-job.xql is setuid to elevate privileges to dba role of some operations

Contents

collections/

This collection contains description of collections of granules. Each collection is serialized as a document. The owner of the collection is the owner of the document and by default the only person who can edit the collection and add/modify granules of the collection.

Collections are identified by an id that is set on any granule of the collection in its 'obs_collection' column.

Collections have at least a title, description. One or more article can be attached to a collection. Keywords can also be added.

Example collection (attached to a VizieR catalog):

<collection id="J/A+A/545/A130" created="2015-01-21T16:35:51.956+01:00">
    <source>http://cdsarc.u-strasbg.fr/viz-bin/Cat?cat=J%2FA%2BA%2F545%2FA130</source>
    <name>J/A+A/545/A130</name>
    <title>Interferometry of {alpha} Eri (Domiciano de Souza+, 2012)</title>
    <description>We used [...]</description>
    <last-modified>06-Sep-2012</last-modified>
    <article>
        <bibcode>2012A&amp;A...545A.130D</bibcode>
        <title>Beyond the diffraction limit of optical/IR interferometers. I. Angular diameter and rotation parameters of Achernar from differential phases</title>
        <author>Domiciano de Souza, A.</author>
   [...]
        <pubdate>2012-09-01</pubdate>
        <keyword>stars: rotation</keyword>
   [...]
    </article>
</collection>

Example collection (user collection):

<collection id="afd8efc6-e62e-46ef-bc13-081f70434f64" created="2015-01-21T16:25:53.551+01:00">
    <name>mycollection</name>
    <title>My Collection</title>
    <description>Description of my collection [...]</description>
    <keywords>stars: rotation</keywords>
    [...]
</collection>

The directory initially has special collections for CHARA (chara_import.xml) and VEGA (vega_import.xml) L0 imports. These collections have special rights to only allow modification by application admins.

See the collection.xqm module of oidb.

comments/

This collection contains a single resource, comments.xml, for user comments on granules. Any JMMC authenticated user read and write this file (to add comments).

Example of comment:

<comment granule-id="25748">
    <author>user1@example.com</author>
    <date>2015-01-28T09:55:08.27+01:00</date>
    <text>Comment</text>
</comment>

The files contains comments on all granules. The comments are threaded and uniquely identified.

Example comments.xml contents

<comments>
    <comment id="9d18fb49-025e-4ea0-af8c-2930d61d2e77" granule-id="25748">
        <author>user1@example.com</author>
        <date>2015-01-28T09:55:08.27+01:00</date>
        <text>Comment</text>
        <comment id="e3e426cf-1fe5-437c-81f7-86a91f4100c4" granule-id="25748">
            <author>user2@example.com</author>
            <date>2015-01-28T09:55:15.278+01:00</date>
            <text>Reply 1</text>
        </comment>
    </comment>
    [...]
</comments>

See the rest/comments.xqm module of the oidb package.

instruments/

This collection contains models for instruments and instrument modes as a checkout of ASPRO2 configuration files (downloaded when the package is built).

keywords/

This collection contains a single resource, journals.xml. This is a preprocessed list of keywords extracted from A&A to be used as possible values for keywords added to granule collections.

log/

This collection contains traces from the application as various XML documents:

  • downloads of files attached to granule (downloads.xml)
  • search queries (searches.xml)
  • reports of granule uploads (submits.xml)
  • page requests with errors (visits.xml)

See the log.xqm module of the oidb package.

oifits/

This collection is a repository for user uploaded OIFITS files.

When a JMMC authenticated user pushes a set of individual OIFITS files to the service, a sub-collection is created in the staging/ collection and files are saved into this sub-collection. The user is marked as the owner of the sub collection and the uploaded files.

The permissions on the files control who can access and/or modify the files.

tmp/

This collection is a repository for temporary documents (for example currently used by the CHARA uploader for storing temporarily the source CSV file prior to parsing).

Installation

Deploy the XAR file from the eXist-db dashboard's package manager.

Note: when deploying a XAR, the package manager destroy any existing collection of the same name. Care must be taken not to overwrite the oidb-data collection without first saving its content.

oidb: OiDB

This component is the main Web application as a XAR package for eXist-db for the Optical Interferometry DataBase.

It indexes observations logs and OIFITS files as granules.

The application implements the Model-View-Controller pattern. It is built on top of eXist-db's shared-resources, importing jQuery for JavaScript and Bootstrap framework for page layout and controls.

It also depends on the subprojects above.

Datamodel

The datamodel is base on ObsCore version 1.0 extended with specific columns.

See OiDbColumns.

The granules and observation logs are stored in a PostgreSQL database. Any other piece of data (collections, comments, ...) are instead saved to the XML database.

Authentication

The JMMC Realm provide for user authentication against the JMMC's user database. The application makes use of eXist-db's persistent login (/extensions/modules/src/org/exist/xquery/modules/persistentlogin): it generates a cookie when the user successfully login to be used with the next request. The application will not request a new authentication if it recognizes the cookie. See controller.xql for tests prior to routing the request.

Users with the 'oidb' credential are considered application admins.

Templating

eXist-db provides in the shared-resources a basic templating system using a map as model. See templates-helpers.xql.

The stock system has been extended with:

  • new helpers and fixes for upstream functions
  • composite models: the stock templating system use model based on map. The new templating function can use an extended model where the values can be maps, XML fragments or sequences
  • hierarchical keys: a dotted notation of keys for traversing a composite model
  • partial rendering: repeat the import of sub templates to each element of a sequence as value to a model key. Lightweight/limited adaptation of Ruby on Rails' render directive.

Note: this module should be moved to jmmc-resources for mutualization with other applications.

ADQL

The application runs ADQL queries on an underlying TAP server. The application then displays the results of the query.

An HTTP query string is translated (modules/adql.xqm) to an ADQL query using pre-defined filters and clauses. The filters (modules/filters.xqm) are used to build the WHERE clause of the query.

The TAP server returns granule as VOTable that is parsed and rendered.

Main search form

The user can search the database with a form similar to trac form for querying tickets: the user enables filters that are combined into a query, the results of which are displayed under the search form. Currently all the filters are combined within a AND operation. Provision in the code have been made for enabling more complex combination (à la trac).

It makes a HTTP POST request to the service with a serialization of the values from the HTML form controls into a query string. That query string is then transformed into a new query string with parameters for each filter (modules/app.xql). The browser is then redirected to the same page with the new query string (following the Redirect after Post design pattern).

Note: TAP does not support fine selection of rows like SQL: there is no equivalent to OFFSET. As a result when paginating results it is necessary to retrieve the rows of the current page plus all the preceding rows. The overhead is minimized by selecting and templating only the rows of interest.

Note: A TAP server may set a limit to the maximum number of rows it sends in response to a query (see 2.7.4 Overflows). If it happens the application will display an overflow message on the current and next pages. It neither affect neither the count of results nor the count of pages.

SAMP

Backoffice

The application provides an administration panel with:

  • display of application logs (submits, visits, searches, comments)
  • basic usage statistics
  • triggers for maintenance operation

The access to this page is restricted to authenticated user with special credential (oidb).

Background jobs

Long running, unsupervised operations are ran as background jobs.

eXist-db provides a scheduler for which jobs can be written in XQuery and periodically ran.

Registering new jobs is only possible for users within the 'dba' role. As a consequence, it is necessary to temporarily elevate the privileges of an OiDB administrator through the evaluation (modules/backoffice.xql) of a a setuid job launcher (modules/backoffice-job.xql). The execution of this script is limited to OiDB administrators.

At the moment, it is possible to asynchronously:

  • update of the inline documentation from the source Wiki page (modules/update-doc.xql)
  • upload a CSV of CHARA observation logs (modules/upload-chara.xql)
  • update the VEGA collection from VegaObs (modules/upload-vega.xql)

Additionnaly, the scripts for the jobs above have non-default permissions to prevent direct execution by a regular user. Still, for single shot, one-off execution, it is possible to request the immediate execution of the scripts through an HTTP request. For example:

$ curl -u user@example.com:password -X POST http://oidb.jmmc.fr/modules/upload-vega.xql
and
$ curl -v -u user@example.com:password -X POST --data-binary @data.csv -H 'Content-Type: application/csv' http://oidb.jmmc.fr/modules/upload-chara.xql

REST API

The application provides a REST API for submitting and manipulating objects (granules, oifits, collections, comments). All resource functions use the HTTP response status code with optional message as XML document in body. They are located in modules under the rest collection.

The REST API is build on top of a lower-level functions in charge of authorization and data validation (see modules/collection.xqm, modules/granules.xqm).

It is used by the user interface through AJAX calls and for batch processing, see the PIONIER L2 update mechanism for example.

It is based on the restxq.xql module from dashboard. As a result any REST request will go through the application controller. The module has been improved for better serialization of resource function outputs.

Note: the RESTXQ trigger of eXist-db has been evaluated (with version 2.1 and 2.2) but discarded as it does not support the persistent login (no access to request object from within the resource function, mandates HTTP Basic authentication instead) and because of issue with support of binary data in requests.

Granule editor

An HTML form using JavaScript and the REST services of the application let users submit new data from OIFITS files:

  • upload of individual OIFITS files
  • extraction of granule from OIFITS files
  • submission of granules after user review

The communication is based on the exchange of XML fragment between client and server with AJAX on the REST endpoints.

Collection creator

The user can prepare a new collection as a container for the new granules.

Collections for L2 date are not attached to an article. The user can specify keywords.

Collections for L3 data require the bibcode of an article.

A variant of the form allows for submission of special L3 collections for VizieR catalog where the collection content is extracted from a VizieR catalog description file (ReadMe). This file follows the rules of the Standard Description for Astronomical Catalogues. See vizier.xql and jmmc-resources jmmc-vizier.xql.

Upload of OIFITS files

Authenticated user can upload local OIFITS files to a staging directory. The form is making use of JavaScript's File API and the REST endpoint for OIFITS (rest/oifits.xqm). Only files recognized as OIFITS by oitools are saved.

It also supports the upload of ZIP files: OIFITS files are extracted and saved to the staging area, preserving the directory structure in the archive.

The user is set as the owner of the files.

Alternatively the user can provide URLs of publically accessible files. At the moment the content of these files is not preserved in the repository.

Metadata extraction from OIFITS files

Granules are extracted from uploaded OIFITS files and URL of OIFITS files using oiexplorer (see oifits:granules() in module oifits.xql).

While processing the OIFITS files the application validates the contents of the file and the granules according to rules from bug 607 (Work in progress, very basic test only at the moment).

The granules are then displayed as HTML form elements in a table for user validation.

Collection and granule submission

On submission of the form, the collection is created if necessary using REST operation for registering collections (see module rest/oifits.xqm, resource function for the POST or PUT verb). On success the granules are attached to the new collection.

The granule are then serialized as an XML fragment and sent to the REST endpoint saving new granules (see module rest/granule.xqm, resource function for the POST verb).

Each granule successfully saved is given an identifier by the application and its display is updated in the table as link to the page with granule details.

An error message is returned for failed granules and the user can fix the granule using the form controls and give another try at saving these granules.

Selectors

For the purpose of user validation, the granule form is using special form controls for groups of related columns in the granule definition.

These selectors are jQuery plugins displaying a dropdown list of possible values as an HTML select element depending and then modifying linked hidden input elements (resources/scripts/selectors.js).

The base class is then specialized to provide :

  • a target selector that link together the target_name, s_ra and s_dec columns;
  • an instrument selector that links together the facility_name and instrument_name;
  • an instrument mode selector that links the instrument_mode with the instrument selection.

Data

Input:
  • observation logs ( CSV / VOTable )
  • files ( OIFits )
  • metadata ( xml )

Output:

  • Collections ( xml )
  • Granules ( xml / sql )

Description

xml data do not have associated xsd. Followin hierarchical structures give a brief overview of the structure:
  • collections:
collection
collection/@id
collection/@created
collection/@updated
collection/source
collection/name
collection/title
collection/description
collection/last-modified
collection/article
collection/article/bibcode
collection/article/title
collection/article/author
collection/article/pubdate
collection/article/keyword
  • granule:
granules
granules/granule
granules/granule/access_estsize
granules/granule/access_format
granules/granule/access_url
granules/granule/calib_level
granules/granule/data_rights
granules/granule/em_max
granules/granule/em_min
granules/granule/em_res_power
granules/granule/facility_name
granules/granule/instrument_name
granules/granule/nb_channels
granules/granule/nb_t3
granules/granule/nb_vis
granules/granule/nb_vis2
granules/granule/obs_collection
granules/granule/obs_creator_name
granules/granule/s_dec
granules/granule/s_ra
granules/granule/target_name
granules/granule/t_exptime
granules/granule/t_max
granules/granule/t_min

More data also are autodocumented onto TBD.

File access permissions

Hosted by OiDb

ExistDb stores oifits (add all non xml) on disk and manages associated permissions using its own permissions.

First protection will follow the unix like protection:

  • files are owned by the user who uploaded the data
  • addition to one collection requires to get write permission on the collection
  • TBD implement handling of embargo period : a scheduler will have to check and update permission to make it public automatically

Provision has been done to deliver these files directly by the apache frontend. This would require to update the apache web server configuration and existd would have to update near the data some .htaccess permission files replacing management by existdb and file access permissions.

Hosted on remote sites

Each data provider is in charge of file delivery protection.

Public repository:
Nothing has to be done by the system.

Current public archive

  • CDS

Private repository
OiDB provides access list files but this requires coordination.

  • Pionier data are hosted at OSUG and managed by the JMMC technical team under the responsability of Jean-Baptiste LeBouquin.

On cron runs every day to update the associated .htaccess

e.g. http://oidb.ujf-grenoble.fr/modules/htaccess.xql?obs_collection=pionier_jan2015

User Interface / Web Portal

  • technologies / environment : existdb (xquery, html template) + bootstrap(html) + jquery
  • user authentication
    • admin group

Interfaces

External services

  • CDS/Simbad
  • CDS/Sesame
  • ESO/Archive

Internal services

Conventions / Tips / Hints

debug

  • javascript uses console.log to help debug in the browser
  • console.log() may also be executed in the xquery and watched/analysed in the monex application

tap sql query samples:

  • SELECT COUNT(*) as c, t.datapi FROM oidb as t GROUP BY t.datapi ORDER BY c DESC
  • SELECT COUNT(*) as c, t.instrument_name FROM oidb as t GROUP BY t.instrument_name ORDER BY c DESC
  • SELECT COUNT(*) as c, t.facility_name FROM oidb as t GROUP BY t.facility_name ORDER BY c DESC

schedulded jobs

setuid xquery

Some action are perfomed using setuid bit so that action get admin permissions. If this script fails, no error is thrown.

To help debug, curl can be used directly on the appropriate xql endpoint: e.g.:

curl -v -uguillaume.mella@obs.ujf-grenoble.fr -X POST --data-binary @CHARA-data-all.csv -H "content-type: application/csv" http://oidb.ujf-grenoble.fr:8080/exist/apps/oidb/modules/upload-chara.xql

one jobs which fails, stay in COMPLETE state but do not deasapear from the list. Please use scheduler:get-scheduled-jobs() and scheduler:delete-scheduled-job($jobsname) to release the situation.

Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r10 - 2015-12-15 - GuillaumeMella
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback