Tags: view all tags

This page is listing technical discussions and notes on the OiDB project. See the definition of the OiDB project in the document from the 2013-06-26 meeeting or in the wiki page.

Notes

Collection

Define a collection as a way for a submitter (dataPI) submitting data to group entries together.

Push/Pull for database insertion

Methods for adding data :

submit files/collection: the user make HTTP PUT request to an URL on the server for his OIFits files or collection of OIFits files. The portal displays a Web form for manually importing new data.

download files/collection: the application is in charge of automatically downlaading the files from the server specified by the user. TODO: define format: XML files with resources to analyze ? parsing HTML file listing ? The portal let the user specify the address of his server and the access conditions.

Data update

Files/collections may be modified by the owner from time to time. The application should regularly download and reprocess the distant repository to stay up to date.

Ask the user for reimporting the data (user resend data)? Automatic import (identify modifications on files: checksum)?

Is it possible to use the name of the file as observation ID?

Data display

To retrieve external data :

turn the portal into a proxy: at user request, the portal check the repository, download the file and return it to the user. The application knows for sure that the user has download the file.

the user perform a request on the application that return the URL for the requested resource. The user then make an HTTP GET request on this URL to download the document. The application can log the request (the user may have downloaded the file but not guaranteed).

make a redirection (similar to Google search results): slightly different from the previous proposition, the user performs a request on the portal that issue a redirect (URL forwarding, HTTP 303) to the external repository. Do all clients and libs support 303 redirections ?

Private data

Some data may not be downloadable until a given amount of time. Still should let people get in contact with the dataPI.

Changes to the current database schema

sort by collection.

link data and dataPI.

add fields for descriptions and comments on data (file, collection target).

make link between observation in the database and scientific publication (bibliographic reference)

keep versions from a same source.

Data model

This information is now in a dedicated topic

OIFITS

Define/extract metadata from submitted OIFits files.

manage datum with flag=true (OI_VIS, OI_VIS2, OI_T3).

take into account repeated wavelength (more than one OI_WAVELENGTH in a single OIFits with the same wavelength value).

measurements from more than one instrument in the same file (OI_WAVELENGTH tables with different INSNAME while observing the same target).

Publishing the the VO

See :

http://wiki.ivoa.net/twiki/bin/view/IVOA/PublishingInTheVONew

Web portal examples

Web Site	MultiCriterii	ADQL	TAP/ObsCore	Samp	Description/comments
http://portal-nvo.noao.edu/adql/query	X	X
http://voparis-srv.obspm.fr/portal/vo.php	X		?	X
http://saada.u-strasbg.fr/taphandle/		X		X
http://labs.adsabs.harvard.edu/adsabs/	X	NC	NC	NC	Not for observation data
http://polarbase.irap.omp.eu/	X		X
http://herschel.esac.esa.int/Science_Archive.shtml					javaWebStart interface see Johan screenshots on the mailinglist : 05/02/2014 14:54
http://telbib.eso.org/?boolany=or&boolaut=or&boolti=or&yearto=2014&boolins=or&telescope[]=%22ALMA%22&booltel=or&search=Search	X	?	?	?	base de données de publi qui renvoie vers l'archive alma
http://almascience.eso.org/aq/	X	?	?	?	Archive Eso. simple d'utilisation recuperation de tgz sur pages web+ script shell

Infrastructure documentation

Linux packages (as root)

Add gavo debian sources (http://soft.g-vo.org/repo)

cat > /etc/apt/sources.list.d/gavo.list << EOF 
# oidb requirements
# http://docs.g-vo.org/DaCHS/install.html#debian-systems
deb http://vo.ari.uni-heidelberg.de/debian stable main
deb-src http://vo.ari.uni-heidelberg.de/debian stable main
EOF 
wget -qO - http://docs.g-vo.org/archive-key.asc | apt-key add -
apt-get update

apt-get install -y postgresql postgresql-9.1-pgsphere libpostgresql-jdbc-java openjdk-7-jdk apache2 vim indent sudo

Accounts (as root)

create the existdb user account with bash account (can be done during install process)

Manual software package (as existdb user)

wget http://optimate.dl.sourceforge.net/project/exist/Stable/2.2/eXist-db-setup-2.2.jar
mkdir exist-db ; cd exist-db/
java -jar ~/eXist-db-setup-2.2.jar -console

existdb@oidb:~/exist-db$ java -jar ~/eXist-db-setup-2.2.jar -console
Select target path [/home/existdb/exist-db] 

press 1 to continue, 2 to quit, 3 to redisplay
1
Set Data Directory
Please select a directory where eXist-db will keep its data files. On Windows, this should be outside the 'Program Files' directory. Please make sure eXist can write to the directory it is installed in.
Data dir:  [webapp/WEB-INF/data] 
/home/existdb/data
press 1 to continue, 2 to quit, 3 to redisplay
1
Set Admin Password and Configure Memory
Enter password:  [] 
XXXX
Enter password:  [XXXX] 
XXXX
------------------------------------------

Maximum memory in mb: [1024] 
2048
Cache memory in mb: [128] 
512
press 1 to continue, 2 to quit, 3 to redisplay
1
[ Starting to unpack ]
...
--- Initialization complete. Shutdown embedded database instance ---
[ Console installation done ]

open dahsboard/package-manager and remove useless applications (demo/client...)

install exist-db wrapper (root)

   cd /etc/init.d
   ln -s ~existdb/exist-db/tools/wrapper/bin/exist.sh .
   update-rc.d exist.sh defaults

set memory options for wrapper:

root@oidb:/home/existdb/exist-db# grep 2048 /home/existdb/exist-db/tools/wrapper/bin/../conf/wrapper.conf
wrapper.java.initmemory=2048
wrapper.java.maxmemory=2048

apache setup

create the virtual host file for oidb:

<VirtualHost *:80>

   ServerName                   oidb.ujf-grenoble.fr

   DocumentRoot                 /var/www/
   AllowEncodedSlashes          on

   # http://exist-db.org/exist/apps/doc/production_web_proxying.xml#D2.2.3.5
   ProxyRequests                off

#   ProxyPass                    /exist/ http://localhost:8080/exist/ nocanon
#   ProxyPassReverse             /exist/ http://localhost:8080/exist/

   ProxyPass                    /server-status !
   
   ProxyPass                    / http://localhost:8080/exist/apps/oidb/ nocanon
   ProxyPassReverse             / http://localhost:8080/exist/apps/oidb/

   ProxyPassReverseCookieDomain localhost oidb.ujf-grenoble.fr
   ProxyPassReverseCookiePath   /exist /

   ErrorLog                     ${APACHE_LOG_DIR}/oidb-error.log
   LogFormat "%h %l %u %t \"%r\" %>s %b" common
   CustomLog            ${APACHE_LOG_DIR}/oidb-access.log common

</VirtualHost>

a2ensite oidb
a2dissite 000-default
a2enmod proxy proxy_http deflate

Set MaxRequestsPerChild to 10000 in /etc/apache2/apache2.conf

PostgreSQL configuration (as postgres)

Create database and database user

#    create PostgreSQL user
createuser --no-createdb --no-createrole --no-superuser --password --encrypted oidb

#    create PostgreSQL database
createdb --owner=oidb --encoding=UTF-8 oidb

#    init pgsphere on new database
psql -d oidb -f //usr/share/postgresql/9.1/contrib/pg_sphere.sql

# set pass

Setup database permissions

#    add as second rules in pg_hba.conf the following lines:
local   oidb        all                               trust

#    restart postgres
service postgres restart

Retrieve setup data (as existdb)

cd ~
read -s -p "enter password" p
wget --no-check-certificate --user=mella --password=$p https://svn.jmmc.fr/jmmc-web/trunk/exist/oidb/conf/oidb.sql

Create table

psql -U oidb -d oidb -f oidb.sql

Sources repository

Main oidb sources are located under the next repository:

https://svn.jmmc.fr/jmmc-sw/oiTools/trunk/
- oitools : OIFits reader / writer library
- oiexplorer-existdb : oitools wrapper (in future OIExplorer wrapper ...)

https://svn.jmmc.fr/jmmc-web/trunk/exist/
- jmmc-realm: An eXist-db realm for authentication with the JMMC user database.

It creates a new eXist-db account with the registration information from the JMMC service. Accounts are created on-demand. The user password is never stored in eXist-db, user identification is still performed by the underlying JMMC user database. All authenticated users are added to a 'jmmc' group. Users with specific credentials are also added to groups named after the credential.

- jmmc-resource: shared XQuery functions + JMMC libraries
- oidb
- oidb-data

eXist-db Setup

Activate java binding (used by jmal / oitools) : set xquery/@enable-java-binding to yes in exist-db/conf.xml
Restrict guest access to exide : set @guest to no in existdb:///db/apps/exide/configuration.xml

OiDB Build

Requires: ant, mvn, eXist-db 2.2

TODO: oiexplorer-existdb/install-existdb-artifacts.sh once !

jmmc-realm module

install local dependency: exist-x.y.jar (see lib/README.txt) : TODO: merge with oiexplorer-existdb/install-existdb-artifacts.sh

cd exist/jmmc-realm/
mvn install:install-file -Dfile=lib/exist-2.2.jar -DgroupId=org.exist-db -DartifactId=exist-exist -Dversion=2.2 -Dpackaging=jar

maven build

cd exist/jmmc-realm/
mvn install

get the jmmc-realm.jar package in exist/jmmc-realm/target/exist-security-jmmc.jar

jmmc-resource

build the jmcs and jmal modules (mvn install) (out of scope)
build the oitools module (mvn install)
build the oiexplorer-existdb (requires oitools)
- install local dependency: exist.jar and xmldb.jar (see install-existdb-artifacts.sh)
- mvn install
build the jmmc-resource (requires jmal oitools oiexplorer-existdb) * mvn install (only gathers previously built modules) * ant
get the jmmc-resource.xar package in exist/jmmc-resources/build/jmmc-resources-x.y.xar

distribution package

TODO: gather all items (jar, xar, ...) into a single zip file ...

OiDB Setup

jmmc-realm module

Install jmmc-realm module to enable JMMC's authentication :

- copy the JAR file exist-security-jmmc.jar in /home/existdb/exist-db/lib/extension
- edit the configuration for the Security Manager from eXide /db/system/security/config.xml and add:

     
       https://jmmc.obs.ujf-grenoble.fr/account/manage.php

- restart exist-db
- check it's working: grep JMMC /home/existdb/exist-db/webapp/WEB-INF/logs/exist.log

install jmmc-resource.xar library
install oidb-data.xar (only if not yet present!!)
install oidb.xar

TAP services:

TAPLib

Code pour wrapping

TBD

prepare directory for taplib logs

mkdir -p /var/log/oidb/taplib
chmod -R 775 /var/log/oidb
chgrp -R jmmc /var/log/oidb

Note: for the time being, log location (OiDBServiceConnection.java), JDBC url and user identification (OiDBDBConnection.java) are hardcoded in OiDBTAP source code.

installation en tant que servlet dans jetty

copy .jar to eXist-db lib directory

cp /usr/share/java/postgresql-8.4-703.jdbc4.jar ~existdb/exist-db/lib/user/
cp taplib.jar binarySavot.jar cds.savot.common.jar cds.savot.model.jar cds.savot.pull.jar cds.savot.writer.jar cos.jar kxml2-min.jar ~existdb/exist-db/lib/user/
cp OiDBTAP.jar ~existdb/exist-db/lib/user/

eXist-db configuration for taplib

#   add to ~existdb/exist-db/webapp/WEB-INF/web.xml
    <servlet>
        <servlet-name>OiDBTAPServlet</servlet-name>
        <servlet-class>fr.jmmc.oidb.tap.OiDBTAPServlet</servlet-class>
    </servlet>

#   add to ~existdb/exist-db/webapp/WEB-INF/controller-config.xml
    <forward pattern="/tap" servlet="OiDBTAPServlet"/>

Note: test for taplib installation (after eXist-db restart and DB configuration)

curl -v http://localhost:9090/exist/tap/capabilities

Dependance avec pgsphere (legacy)

Installing PostgreSQL 8.4.x on CentOS 5.5

# http://heatware.net/linux-unix/how-install-postgresql-8-4-centos-5/

#    Download the PostgreSQL 8.4 repository RPM
wget http://yum.pgrpms.org/reporpms/8.4/pgdg-centos-8.4-2.noarch.rpm
#    Add the repository
rpm -ivh pgdg-centos-8.4-2.noarch.rpm
#    Use yum to install PostgresSQL
yum install -y postgresql postgresql-server
#    Initialize the database
service postgresql initdb
#    Start the database
service postgresql restart
#    Automatically start database on boot
chkconfig postgresql on

Installing PGSphere

# install PGSphere
yum install pgsphere

Install PostgreSQL JDBC driver

# install postgresql jdbc driver
yum install postgresql-jdbc

PostgreSQL configuration

Create database and database user

#    create PostgreSQL user
sudo -u postgres createuser --no-createdb --no-createrole --no-superuser --password --encrypted oidb

#    create PostgreSQL database
sudo -u postgres createdb --owner=oidb --encoding=UTF-8 oidb

#    init pgsphere on new database
sudo -u postgres psql -d oidb -f /usr/share/pgsphere/pg_sphere.sql

# set pass

Setup database permissions

#    add to the top of /var/lib/pgsql/data/pg_hba.conf the following lines:
local   oidb        postgres                          ident
local   oidb        all                               md5

#    restart postgres
service postgres restart

Create table

psql -U oidb -d oidb -W -f oidb.postgres.sql

oidb.postgres.sql: PostgreSQL table creation script

Attachments

Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
sql	oidb.postgres.sql	r3 r2 r1	manage	2.1 K	2014-11-18 - 10:39	PatrickBernaud	PostgreSQL table creation script

Topic revision: r32 - 2014-11-26 - LaurentBourges