Tags:
create new tag
view all tags
This page is listing technical discussions and notes on the OiDB project. See the definition of the OiDB project in the document from the 2013-06-26 meeeting or in the wiki page.

Notes

Collection

Define a collection as a way for a submitter (dataPI) submitting data to group entries together.

Push/Pull for database insertion

Methods for adding data :

  • submit files/collection: the user make HTTP PUT request to an URL on the server for his OIFits files or collection of OIFits files. The portal displays a Web form for manually importing new data.

  • download files/collection: the application is in charge of automatically downlaading the files from the server specified by the user. TODO: define format: XML files with resources to analyze ? parsing HTML file listing ? The portal let the user specify the address of his server and the access conditions.

Data update

Files/collections may be modified by the owner from time to time. The application should regularly download and reprocess the distant repository to stay up to date.

Ask the user for reimporting the data (user resend data)? Automatic import (identify modifications on files: checksum)?

Is it possible to use the name of the file as observation ID?

Data display

To retrieve external data :

  • turn the portal into a proxy: at user request, the portal check the repository, download the file and return it to the user. The application knows for sure that the user has download the file.

  • the user perform a request on the application that return the URL for the requested resource. The user then make an HTTP GET request on this URL to download the document. The application can log the request (the user may have downloaded the file but not guaranteed).

  • make a redirection (similar to Google search results): slightly different from the previous proposition, the user performs a request on the portal that issue a redirect (URL forwarding, HTTP 303) to the external repository. Do all clients and libs support 303 redirections ?

Private data

Some data may not be downloadable until a given amount of time. Still should let people get in contact with the dataPI.

Changes to the current database schema

  • sort by collection.

  • link data and dataPI.

  • add fields for descriptions and comments on data (file, collection target).

  • make link between observation in the database and scientific publication (bibliographic reference)

  • keep versions from a same source.

Data model

This information is now in a dedicated topic

OIFITS

Define/extract metadata from submitted OIFits files.

  • manage datum with flag=true (OI_VIS, OI_VIS2, OI_T3).

  • take into account repeated wavelength (more than one OI_WAVELENGTH in a single OIFits with the same wavelength value).

  • measurements from more than one instrument in the same file (OI_WAVELENGTH tables with different INSNAME while observing the same target).

Publishing the the VO

See :

Web portal examples

Web Site MultiCriterii ADQL TAP/ObsCore Samp Description/comments
http://portal-nvo.noao.edu/adql/query X X      
http://voparis-srv.obspm.fr/portal/vo.php X   ? X  
http://saada.u-strasbg.fr/taphandle/   X   X  
http://labs.adsabs.harvard.edu/adsabs/ X NC NC NC Not for observation data wink
http://polarbase.irap.omp.eu/ X   X    
http://herschel.esac.esa.int/Science_Archive.shtml         javaWebStart interface see Johan screenshots on the mailinglist : 05/02/2014 14:54
http://telbib.eso.org/?boolany=or&boolaut=or&boolti=or&yearto=2014&boolins=or&telescope[]=%22ALMA%22&booltel=or&search=Search X ? ? ? base de données de publi qui renvoie vers l'archive alma
http://almascience.eso.org/aq/ X   ?  ?  ? Archive Eso. simple d'utilisation recuperation de tgz sur pages web+ script shell
http://giraffe-archive.obspm.fr/search.html X     coté VO bien développé A voir!!

Infrastructure documentation

Linux packages (as root) 

cat > /etc/apt/sources.list.d/gavo.list << EOF 
# oidb requirements
# http://docs.g-vo.org/DaCHS/install.html#debian-systems
deb http://vo.ari.uni-heidelberg.de/debian stable main
deb-src http://vo.ari.uni-heidelberg.de/debian stable main
EOF 
wget -qO - http://docs.g-vo.org/archive-key.asc | apt-key add -
apt-get update

  • apt-get install -y postgresql postgresql-9.1-pgsphere libpostgresql-jdbc-java openjdk-7-jdk apache2 vim indent sudo subversion ant

Accounts (as root)

  • create the existdb user account with bash account (can be done during install process)

Manual software package (as existdb user) 

existdb@oidb:~/exist-db$ java -jar ~/eXist-db-setup-2.2.jar -console
Select target path [/home/existdb/exist-db] 

press 1 to continue, 2 to quit, 3 to redisplay
1
Set Data Directory
Please select a directory where eXist-db will keep its data files. On Windows, this should be outside the 'Program Files' directory. Please make sure eXist can write to the directory it is installed in.
Data dir:  [webapp/WEB-INF/data] 
/home/existdb/data
press 1 to continue, 2 to quit, 3 to redisplay
1
Set Admin Password and Configure Memory
Enter password:  [] 
XXXX
Enter password:  [XXXX] 
XXXX
------------------------------------------

Maximum memory in mb: [1024] 
2048
Cache memory in mb: [128] 
512
press 1 to continue, 2 to quit, 3 to redisplay
1
[ Starting to unpack ]
...
--- Initialization complete. Shutdown embedded database instance ---
[ Console installation done ]

  • open dahsboard/package-manager and remove useless applications (demo/client...)

  • install exist-db wrapper (root) 
   cd /etc/init.d
   ln -s ~existdb/exist-db/tools/wrapper/bin/exist.sh .
   update-rc.d exist.sh defaults

  • set memory options for wrapper: 
root@oidb:/home/existdb/exist-db# grep 2048 /home/existdb/exist-db/tools/wrapper/bin/../conf/wrapper.conf
wrapper.java.initmemory=2048
wrapper.java.maxmemory=2048

  • Tell jetty's servlets to use the frontend server name -> set forwarded property of the SelectChannelConnector to true 
<New class="org.eclipse.jetty.server.nio.SelectChannelConnector">
   ...
   <Set name="forwarded">true</Set>
   ...

Apache setup

  • create the virtual host file for oidb: 
<VirtualHost *:80>

   ServerName                   oidb.ujf-grenoble.fr

   DocumentRoot                 /var/www/
   AllowEncodedSlashes          on

   # see doc on https://wiki.eclipse.org/Jetty/Howto/Configure_mod_proxy
   #            http://httpd.apache.org/docs/2.2/mod/mod_proxy.html
   #            http://exist-db.org/exist/apps/doc/production_web_proxying.xml#D2.2.3.5
        #            https://wiki.eclipse.org/Jetty/Tutorial/Apache#Configuring_mod_proxy
        #            https://wiki.eclipse.org/Jetty/Tutorial/Apache#Proxying_SSL_on_Apache_to_HTTP_on_Jetty 

   ProxyRequests                off

# does not yet work on next interpolate (requires mod_rewrite ?)   ProxyPassInterpolateEnv On

   # ProxyStatus On

   # do not redirect server-status
   ProxyPass                    /server-status !

# uncomment to allow direct access on exide, dashboard and every exist applications (still reachable over 8080)      
#   ProxyPass                    /exist/ http://localhost:8080/exist/ nocanon
#   ProxyPassReverse             /exist/ http://${SERVER_NAME}/exist/ interpolate

   # register tap servlet as /tap
   # and map /exist/tap to the taplib servlet because async results contain ${SERVER_NAME}:80/exist/tap using getContextPath in the java code 
   ProxyPass                    /tap/ http://localhost:8080/exist/tap/ nocanon
   ProxyPass                    /exist/tap/ http://localhost:8080/exist/tap/ nocanon
#   ProxyPassReverse             /tap/ http://${SERVER_NAME}:80/exist/tap/ interpolate
   ProxyPassReverse             /tap/ http://oidb.ujf-grenoble.fr:80/exist/tap/
   
        # leave access to uploaded files
   ProxyPass                    /exist/apps/oidb-data/oifits/ http://localhost:8080/exist/apps/oidb-data/oifits/ nocanon

#   ProxyPass                    /exist/apps/oidb/ http://localhost:8080/exist/apps/oidb/ nocanon
   ProxyPass                    / http://localhost:8080/exist/apps/oidb/ nocanon
#   ProxyPassReverse             / http://${SERVER_NAME}/exist/apps/oidb/ interpolate
   ProxyPassReverse             / http://oidb.ujf-grenoble.fr/exist/apps/oidb/

#   ProxyPassReverseCookieDomain localhost ${SERVER_NAME} interpolate
   ProxyPassReverseCookieDomain localhost oidb.ujf-grenoble.fr
   ProxyPassReverseCookiePath   /exist /

   ErrorLog                     ${APACHE_LOG_DIR}/oidb-error.log
   LogFormat "%h %l %u %t \"%r\" %>s %b" common
   CustomLog            ${APACHE_LOG_DIR}/oidb-access.log common

</VirtualHost>

  • copy the same lines (but log files) in the /etc/apache2/sites-available/default-ssl

  • a2enmod ssl
  • a2ensite oidb
  • a2ensite default-ssl
  • a2dissite 000-default
  • a2enmod proxy proxy_http deflate

  • Set MaxRequestsPerChild to 10000 in /etc/apache2/apache2.conf

  • apache2ctl -S

Creation / installation of a certificate 

/etc/apache2/sites-enabled/default-ssl:   SSLCertificateFile /etc/ssl/certs/cert-26285-oidb.ujf-grenoble.fr.pem
/etc/apache2/sites-enabled/default-ssl:   SSLCertificateKeyFile /etc/ssl/private/oidb.ujf-grenoble.fr.key
/etc/apache2/sites-enabled/default-ssl:   SSLCertificateChainFile /etc/ssl/certs/chain-26285-oidb.ujf-grenoble.fr.pem

PostgreSQL configuration (as postgres)

  • Create database and database user 
#    create PostgreSQL user
createuser --no-createdb --no-createrole --no-superuser --encrypted oidb

#    create PostgreSQL database
createdb --owner=oidb --encoding=UTF-8 oidb

#    init pgsphere on new database
psql -d oidb -f //usr/share/postgresql/9.1/contrib/pg_sphere.sql

# set pass database permissions
# add as second rules in pg_hba.conf the following lines:
# (/etc/postgresql/9.1/main/pg_hba.conf) 'local' can't be use with JDBC
host   oidb             oidb             127.0.0.1/32           trust

#    restart postgres
service postgres restart

Retrieve setup data (as existdb)

  • Create table 
psql -U oidb -d oidb -f oidb.sql

Sources repository

Main oidb sources are located under the next repository:
  • https://svn.jmmc.fr/jmmc-sw/oiTools/trunk/
    • jmal : JMMC Astronomical library (ALX HMS / DMS conversion)
    • oitools : OIFits reader / writer library
    • oiexplorer-existdb : oitools wrapper (in future OIExplorer wrapper ...)

It creates a new eXist-db account with the registration information from the JMMC service. Accounts are created on-demand. The user password is never stored in eXist-db, user identification is still performed by the underlying JMMC user database. All authenticated users are added to a 'jmmc' group. Users with specific credentials are also added to groups named after the credential.

    • jmmc-resource : shared XQuery functions + JMMC libraries
    • oidb : web application module
    • oidb-data : data layer definition (collection, comments, logs, uploaded oifits files)

eXist-db Setup

  • Activate java binding (used by jmal / oitools) : set xquery/@enable-java-binding to yes in file exist-db/conf.xml (existdb user)
  • Restrict guest access to exide : set @guest to no in existdb:///db/apps/exide/configuration.xml

OiDB Build

Requires: ant, mvn, eXist-db 2.2

First time: install exist-db libraries (exist.jar and xmldb.jar) into the maven local repository see oiexplorer-existdb/install-existdb-artifacts.sh

jmmc-realm module

  • mvn install
  • get the jmmc-realm package in exist/jmmc-realm/target/exist-security-jmmc.jar

jmmc-resource

  • build the jmcs and jmal modules (mvn install) (out of scope)
  • build the oitools module (mvn install)
  • build the oiexplorer-existdb (requires oitools)
    • mvn install
  • build the jmmc-resource (requires jmal oitools oiexplorer-existdb)
    • mvn install (only gathers previously built modules)
    • ant
  • get the jmmc-resource package in exist/jmmc-resources/build/jmmc-resources-x.y.xar

oidb-data

  • build the oidb-data (use svn to get aspro configuration)
    • ant
  • get the oidb-data package in exist/oidb-data/build/oidb-data-x.y.xar

oidb

  • build the oidb module
    • create the file : config.properties 
jndi-name = jdbc/oidb
sql-table = oidb
tap-baseurl=http://oidb.ujf-grenoble.fr/exist/tap
  • ant -Dconfig.properties=config.properties
  • get the oidb package in exist/oidb/build/oidb-x.y.xar

oidb-tap

  • build the oidb-tap
    • mvn install
  • get the oidb-tap package in exist/oidb-tap/target/oidb-tap-TRUNK.jar

distribution package

TODO: gather all items (jar, xar, ...) into a single zip file ...

  • jmmc-realm : jmmc-realm/target/exist-security-jmmc.jar
  • jmmc-resource : jmmc-resources/build/jmmc-resources-x.y.xar
  • oidb-data : oidb-data/build/oidb-data-x.y.xar
  • oidb : oidb/build/oidb-x.y.xar
  • oidb-tap : /oidb-tap/target/oidb-tap-TRUNK.jar and /oidb-tap/target/runtime-libs/tap-x.y.jar

OiDB Setup

jmmc-realm module

Install jmmc-realm module to enable JMMC's authentication :
    • copy the JAR file exist-security-jmmc.jar in /home/existdb/exist-db/lib/extensions/
    • edit the configuration for the Security Manager from eXide /db/system/security/config.xml and add: 
     <realm id="JMMC">
       <url>https://jmmc.obs.ujf-grenoble.fr/account/manage.php</url>
     </realm>
    • restart exist-db
    • check it's working: 
grep JMMC /home/existdb/exist-db/webapp/WEB-INF/logs/exist.log

oidb-tap

Note: for the time being, log location (OiDBServiceConnection.java), JDBC url and user identification (OiDBDBConnection.java) are hard-coded in oidb-tap source code.

  • prepare directory for taplib logs (root user) 
mkdir -p /var/log/oidb/taplib
chmod -R 775 /var/log/oidb
chown -R existdb /var/log/oidb
  • install postgres driver to eXist-db lib directory (first time only) 
cp /usr/share/java/postgresql-jdbc4.jar ~existdb/exist-db/lib/user/
  • install oidb-tap files to eXist-db lib directory 
cp taplib*.jar oidb-tap*.jar ~existdb/exist-db/lib/user/
  • Jetty setup edit ~existdb/exist-db/webapp/WEB-INF/web.xml 
    <servlet>
        <servlet-name>OiDBTAPServlet</servlet-name>
        <servlet-class>fr.jmmc.oidb.tap.OiDBTAPServlet</servlet-class>
    </servlet>
  • exist-db setup add to ~existdb/exist-db/webapp/WEB-INF/controller-config.xml 
    <forward pattern="/tap" servlet="OiDBTAPServlet"/>
  • restart exist-db service (root)
  • test: 
wget http://localhost:8080/exist/tap/availability
wget 'http://localhost:8080/exist/tap/sync?MAXREC=1&QUERY=SELECT * FROM oidb&FORMAT=votable&REQUEST=doQuery&LANG=ADQL'

exist-db packages

Use the package manager (admin):
  • JMMC Commons module : install jmmc-resources-x.y.xar (may require on dev machines the http proxy : -Dhttp.proxyHost=... -Dhttp.proxyPort=...)
  • oidb-data : install oidb-data-x.y.xar (only if not yet present!) TODO: how to handle migration / data model lifecycle ?
  • OIData portal : install oidb-x.y.xar

That's all folks !

Links 

http://oidb.ujf-grenoble.fr:8080/exist/tap/sync?QUERY=SELECT%20COUNT%28*%29%20FROM%20%28SELECT%20ALL%20%20*%20FROM%20oidb%20AS%20t%20WHERE%20%28%20t.calib_level=0%20%29%20%20%29%20AS%20e&FORMAT=votable&REQUEST=doQuery&LANG=ADQL

TODO

Backup

  • Store db content on local files :
    • SQL backup
    • existbd backup
    • oidata backup
  • Store files on an external medium

Migration / Maintenance

  • describe procedure
    • provide readonly mode
    • provide a common message displayed on every pages
    • ...

Attachments Attachments
Topic attachments
I Attachment History Action Size Date Who Comment
Texttxt ExistDbInstallationSample.txt r1 manage 7.4 K 2015-09-24 - 06:44 GuillaumeMella quick install procedure sample for existdb
Unknown file formatsql oidb.postgres.sql r3 r2 r1 manage 2.1 K 2014-11-18 - 10:39 PatrickBernaud PostgreSQL table creation script
Edit | Attach | Watch | Print version | History: r53 < r52 < r51 < r50 < r49 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r53 - 2016-11-23 - GuillaumeMella
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback