This page is listing technical discussions and notes on the OiDB project. See the definition of the
OiDB project in the document from the
2013-06-26 meeeting or in the
wiki page.
Notes
Collection
Define a collection as a way for a submitter (dataPI) submitting data to group entries together.
Push/Pull for database insertion
Methods for adding data :
- submit files/collection: the user make HTTP PUT request to an URL on the server for his OIFits files or collection of OIFits files. The portal displays a Web form for manually importing new data.
- download files/collection: the application is in charge of automatically downlaading the files from the server specified by the user. TODO: define format: XML files with resources to analyze ? parsing HTML file listing ? The portal let the user specify the address of his server and the access conditions.
Data update
Files/collections may be modified by the owner from time to time. The application should regularly download and reprocess the distant repository to stay up to date.
Ask the user for reimporting the data (user resend data)? Automatic import (identify modifications on files: checksum)?
Is it possible to use the name of the file as observation ID?
Data display
To retrieve external data :
- turn the portal into a proxy: at user request, the portal check the repository, download the file and return it to the user. The application knows for sure that the user has download the file.
- the user perform a request on the application that return the URL for the requested resource. The user then make an HTTP GET request on this URL to download the document. The application can log the request (the user may have downloaded the file but not guaranteed).
- make a redirection (similar to Google search results): slightly different from the previous proposition, the user performs a request on the portal that issue a redirect (URL forwarding, HTTP 303) to the external repository. Do all clients and libs support 303 redirections ?
Private data
Some data may not be downloadable until a given amount of time. Still should let people get in contact with the dataPI.
Changes to the current database schema
- add fields for descriptions and comments on data (file, collection target).
- make link between observation in the database and scientific publication (bibliographic reference)
- keep versions from a same source.
Data model
This information is now in a
dedicated topic
OIFITS
Define/extract metadata from submitted OIFits files.
- manage datum with flag=true (OI_VIS, OI_VIS2, OI_T3).
- take into account repeated wavelength (more than one OI_WAVELENGTH in a single OIFits with the same wavelength value).
- measurements from more than one instrument in the same file (OI_WAVELENGTH tables with different INSNAME while observing the same target).
Publishing the the VO
See :
Web portal examples
Infrastructure documentation
Linux packages (as root)
cat > /etc/apt/sources.list.d/gavo.list << EOF
# oidb requirements
# http://docs.g-vo.org/DaCHS/install.html#debian-systems
deb http://vo.ari.uni-heidelberg.de/debian stable main
deb-src http://vo.ari.uni-heidelberg.de/debian stable main
EOF
wget -qO - http://docs.g-vo.org/archive-key.asc | apt-key add -
apt-get update
- apt-get install -y postgresql postgresql-9.1-pgsphere libpostgresql-jdbc-java openjdk-7-jdk apache2 vim indent sudo subversion ant
Accounts (as root)
- create the
existdb
user account with bash account (can be done during install process)
Manual software package (as existdb user)
existdb@oidb:~/exist-db$ java -jar ~/eXist-db-setup-2.2.jar -console
Select target path [/home/existdb/exist-db]
press 1 to continue, 2 to quit, 3 to redisplay
1
Set Data Directory
Please select a directory where eXist-db will keep its data files. On Windows, this should be outside the 'Program Files' directory. Please make sure eXist can write to the directory it is installed in.
Data dir: [webapp/WEB-INF/data]
/home/existdb/data
press 1 to continue, 2 to quit, 3 to redisplay
1
Set Admin Password and Configure Memory
Enter password: []
XXXX
Enter password: [XXXX]
XXXX
------------------------------------------
Maximum memory in mb: [1024]
2048
Cache memory in mb: [128]
512
press 1 to continue, 2 to quit, 3 to redisplay
1
[ Starting to unpack ]
...
--- Initialization complete. Shutdown embedded database instance ---
[ Console installation done ]
- open dahsboard/package-manager and remove useless applications (demo/client...)
- install exist-db wrapper (root)
cd /etc/init.d
ln -s ~existdb/exist-db/tools/wrapper/bin/exist.sh .
update-rc.d exist.sh defaults
- set memory options for wrapper:
root@oidb:/home/existdb/exist-db# grep 2048 /home/existdb/exist-db/tools/wrapper/bin/../conf/wrapper.conf
wrapper.java.initmemory=2048
wrapper.java.maxmemory=2048
- Tell jetty's servlets to use the frontend server name -> set forwarded property of the SelectChannelConnector to true
<New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
...
<Set name="forwarded">true</Set>
...
Apache setup
- create the virtual host file for oidb:
<VirtualHost *:80>
ServerName oidb.ujf-grenoble.fr
DocumentRoot /var/www/
AllowEncodedSlashes on
# see doc on https://wiki.eclipse.org/Jetty/Howto/Configure_mod_proxy
# http://httpd.apache.org/docs/2.2/mod/mod_proxy.html
# http://exist-db.org/exist/apps/doc/production_web_proxying.xml#D2.2.3.5
# https://wiki.eclipse.org/Jetty/Tutorial/Apache#Configuring_mod_proxy
# https://wiki.eclipse.org/Jetty/Tutorial/Apache#Proxying_SSL_on_Apache_to_HTTP_on_Jetty
ProxyRequests off
# does not yet work on next interpolate (requires mod_rewrite ?) ProxyPassInterpolateEnv On
# ProxyStatus On
# do not redirect server-status
ProxyPass /server-status !
# uncomment to allow direct access on exide, dashboard and every exist applications (still reachable over 8080)
# ProxyPass /exist/ http://localhost:8080/exist/ nocanon
# ProxyPassReverse /exist/ http://${SERVER_NAME}/exist/ interpolate
# register tap servlet as /tap
# and map /exist/tap to the taplib servlet because async results contain ${SERVER_NAME}:80/exist/tap using getContextPath in the java code
ProxyPass /tap/ http://localhost:8080/exist/tap/ nocanon
ProxyPass /exist/tap/ http://localhost:8080/exist/tap/ nocanon
# ProxyPassReverse /tap/ http://${SERVER_NAME}:80/exist/tap/ interpolate
ProxyPassReverse /tap/ http://oidb.ujf-grenoble.fr:80/exist/tap/
# leave access to uploaded files
ProxyPass /exist/apps/oidb-data/oifits/ http://localhost:8080/exist/apps/oidb-data/oifits/ nocanon
# ProxyPass /exist/apps/oidb/ http://localhost:8080/exist/apps/oidb/ nocanon
ProxyPass / http://localhost:8080/exist/apps/oidb/ nocanon
# ProxyPassReverse / http://${SERVER_NAME}/exist/apps/oidb/ interpolate
ProxyPassReverse / http://oidb.ujf-grenoble.fr/exist/apps/oidb/
# ProxyPassReverseCookieDomain localhost ${SERVER_NAME} interpolate
ProxyPassReverseCookieDomain localhost oidb.ujf-grenoble.fr
ProxyPassReverseCookiePath /exist /
ErrorLog ${APACHE_LOG_DIR}/oidb-error.log
LogFormat "%h %l %u %t \"%r\" %>s %b" common
CustomLog ${APACHE_LOG_DIR}/oidb-access.log common
</VirtualHost>
- copy the same lines (but log files) in the /etc/apache2/sites-available/default-ssl
- a2enmod ssl
- a2ensite oidb
- a2ensite default-ssl
- a2dissite 000-default
- a2enmod proxy proxy_http deflate
- Set MaxRequestsPerChild to 10000 in /etc/apache2/apache2.conf
Creation / installation of a certificate
/etc/apache2/sites-enabled/default-ssl: SSLCertificateFile /etc/ssl/certs/cert-26285-oidb.ujf-grenoble.fr.pem
/etc/apache2/sites-enabled/default-ssl: SSLCertificateKeyFile /etc/ssl/private/oidb.ujf-grenoble.fr.key
/etc/apache2/sites-enabled/default-ssl: SSLCertificateChainFile /etc/ssl/certs/chain-26285-oidb.ujf-grenoble.fr.pem
PostgreSQL configuration (as postgres)
- Create database and database user
# create PostgreSQL user
createuser --no-createdb --no-createrole --no-superuser --encrypted oidb
# create PostgreSQL database
createdb --owner=oidb --encoding=UTF-8 oidb
# init pgsphere on new database
psql -d oidb -f //usr/share/postgresql/9.1/contrib/pg_sphere.sql
# set pass database permissions
# add as second rules in pg_hba.conf the following lines:
# (/etc/postgresql/9.1/main/pg_hba.conf) 'local' can't be use with JDBC
host oidb oidb 127.0.0.1/32 trust
# restart postgres
service postgres restart
Retrieve setup data (as existdb)
psql -U oidb -d oidb -f oidb.sql
Sources repository
Main oidb sources are located under the next repository:
- https://svn.jmmc.fr/jmmc-sw/oiTools/trunk/
- jmal : JMMC Astronomical library (ALX HMS / DMS conversion)
- oitools : OIFits reader / writer library
- oiexplorer-existdb : oitools wrapper (in future OIExplorer wrapper ...)
It creates a new eXist-db account with the registration information from the JMMC service.
Accounts are created on-demand.
The user password is never stored in eXist-db, user identification is still performed by the underlying JMMC user database.
All authenticated users are added to a 'jmmc' group. Users with specific credentials are also added to groups named after the credential.
-
- jmmc-resource : shared XQuery functions + JMMC libraries
- oidb : web application module
- oidb-data : data layer definition (collection, comments, logs, uploaded oifits files)
eXist-db Setup
- Activate java binding (used by jmal / oitools) : set xquery/@enable-java-binding to yes in file exist-db/conf.xml (existdb user)
- Restrict guest access to exide : set @guest to no in existdb:///db/apps/exide/configuration.xml
Requires: ant, mvn, eXist-db 2.2
First time: install exist-db libraries (exist.jar and xmldb.jar) into the maven local repository
see oiexplorer-existdb/install-existdb-artifacts.sh
jmmc-realm module
- mvn install
- get the jmmc-realm package in exist/jmmc-realm/target/exist-security-jmmc.jar
jmmc-resource
- build the jmcs and jmal modules (mvn install) (out of scope)
- build the oitools module (mvn install)
- build the oiexplorer-existdb (requires oitools)
- build the jmmc-resource (requires jmal oitools oiexplorer-existdb)
- mvn install (only gathers previously built modules)
- ant
- get the jmmc-resource package in exist/jmmc-resources/build/jmmc-resources-x.y.xar
oidb-data
- build the oidb-data (use svn to get aspro configuration)
- get the oidb-data package in exist/oidb-data/build/oidb-data-x.y.xar
oidb
- build the oidb module
- create the file : config.properties
jndi-name = jdbc/oidb
sql-table = oidb
tap-baseurl=http://oidb.ujf-grenoble.fr/exist/tap
- ant -Dconfig.properties=config.properties
- get the oidb package in exist/oidb/build/oidb-x.y.xar
oidb-tap
- build the oidb-tap
- get the oidb-tap package in exist/oidb-tap/target/oidb-tap-TRUNK.jar
distribution package
TODO: gather all items (jar, xar, ...) into a single zip file ...
- jmmc-realm : jmmc-realm/target/exist-security-jmmc.jar
- jmmc-resource : jmmc-resources/build/jmmc-resources-x.y.xar
- oidb-data : oidb-data/build/oidb-data-x.y.xar
- oidb : oidb/build/oidb-x.y.xar
- oidb-tap : /oidb-tap/target/oidb-tap-TRUNK.jar and /oidb-tap/target/runtime-libs/tap-x.y.jar
jmmc-realm module
Install jmmc-realm module to enable JMMC's authentication :
-
- copy the JAR file exist-security-jmmc.jar in /home/existdb/exist-db/lib/extensions/
- edit the configuration for the Security Manager from eXide
/db/system/security/config.xml
and add:
<realm id="JMMC">
<url>https://jmmc.obs.ujf-grenoble.fr/account/manage.php</url>
</realm>
-
- restart exist-db
- check it's working:
grep JMMC /home/existdb/exist-db/webapp/WEB-INF/logs/exist.log
oidb-tap
Note: for the time being, log location (OiDBServiceConnection.java), JDBC url and user identification (OiDBDBConnection.java) are hard-coded in oidb-tap source code.
- prepare directory for taplib logs (root user)
mkdir -p /var/log/oidb/taplib
chmod -R 775 /var/log/oidb
chown -R existdb /var/log/oidb
- install postgres driver to eXist-db lib directory (first time only)
cp /usr/share/java/postgresql-jdbc4.jar ~existdb/exist-db/lib/user/
- install oidb-tap files to eXist-db lib directory
cp taplib*.jar oidb-tap*.jar ~existdb/exist-db/lib/user/
- Jetty setup edit ~existdb/exist-db/webapp/WEB-INF/web.xml
<servlet>
<servlet-name>OiDBTAPServlet</servlet-name>
<servlet-class>fr.jmmc.oidb.tap.OiDBTAPServlet</servlet-class>
</servlet>
- exist-db setup add to ~existdb/exist-db/webapp/WEB-INF/controller-config.xml
<forward pattern="/tap" servlet="OiDBTAPServlet"/>
- restart exist-db service (root)
- test:
wget http://localhost:8080/exist/tap/availability
wget 'http://localhost:8080/exist/tap/sync?MAXREC=1&QUERY=SELECT * FROM oidb&FORMAT=votable&REQUEST=doQuery&LANG=ADQL'
exist-db packages
Use the package manager (admin):
-
JMMC Commons module
: install jmmc-resources-x.y.xar (may require on dev machines the http proxy : -Dhttp.proxyHost=... -Dhttp.proxyPort=...)
-
oidb-data
: install oidb-data-x.y.xar (only if not yet present!) TODO: how to handle migration / data model lifecycle ?
-
OIData portal
: install oidb-x.y.xar
That's all folks !
Links
http://oidb.ujf-grenoble.fr:8080/exist/tap/sync?QUERY=SELECT%20COUNT%28*%29%20FROM%20%28SELECT%20ALL%20%20*%20FROM%20oidb%20AS%20t%20WHERE%20%28%20t.calib_level=0%20%29%20%20%29%20AS%20e&FORMAT=votable&REQUEST=doQuery&LANG=ADQL
TODO
Backup
- Store db content on local files :
- SQL backup
- existbd backup
- oidata backup
- Store files on an external medium
Migration / Maintenance
- describe procedure
- provide readonly mode
- provide a common message displayed on every pages
- ...