This page is listing technical discussions and notes on the OiDB project. See the definition of the OiDB project in the document from the 2013-06-26 meeeting or in the wiki page.

Notes

Collection

Define a collection as a way for a submitter (dataPI) submitting data to group entries together.

Push/Pull for database insertion

Methods for adding data :

• submit files/collection: the user make HTTP PUT request to an URL on the server for his OIFits files or collection of OIFits files. The portal displays a Web form for manually importing new data.

• download files/collection: the application is in charge of automatically downlaading the files from the server specified by the user. TODO: define format: XML files with resources to analyze ? parsing HTML file listing ? The portal let the user specify the address of his server and the access conditions.

Data update

Files/collections may be modified by the owner from time to time. The application should regularly download and reprocess the distant repository to stay up to date.

Ask the user for reimporting the data (user resend data)? Automatic import (identify modifications on files: checksum)?

Is it possible to use the name of the file as observation ID?

Data display

To retrieve external data :

• turn the portal into a proxy: at user request, the portal check the repository, download the file and return it to the user. The application knows for sure that the user has download the file.

• the user perform a request on the application that return the URL for the requested resource. The user then make an HTTP GET request on this URL to download the document. The application can log the request (the user may have downloaded the file but not guaranteed).

• make a redirection (similar to Google search results): slightly different from the previous proposition, the user performs a request on the portal that issue a redirect (URL forwarding, HTTP 303) to the external repository. Do all clients and libs support 303 redirections ?

Private data

Some data may not be downloadable until a given amount of time. Still should let people get in contact with the dataPI.

Changes to the current database schema

• sort by collection.

• link data and dataPI.

• add fields for descriptions and comments on data (file, collection target).

• make link between observation in the database and scientific publication (bibliographic reference)

• keep versions from a same source.

Data model

This information is now in a dedicated topic

OIFITS

Define/extract metadata from submitted OIFits files.

• manage datum with flag=true (OI_VIS, OI_VIS2, OI_T3).

• take into account repeated wavelength (more than one OI_WAVELENGTH in a single OIFits with the same wavelength value).

• measurements from more than one instrument in the same file (OI_WAVELENGTH tables with different INSNAME while observing the same target).

Publishing the the VO

See :

• http://wiki.ivoa.net/twiki/bin/view/IVOA/PublishingInTheVONew

Web portal examples

Web Site	MultiCriterii	ADQL	TAP/ObsCore	Samp	Description/comments
http://portal-nvo.noao.edu/adql/query	X	X
http://voparis-srv.obspm.fr/portal/vo.php	X		?	X
http://saada.u-strasbg.fr/taphandle/		X		X
http://labs.adsabs.harvard.edu/adsabs/	X	NC	NC	NC	Not for observation data
http://polarbase.irap.omp.eu/	X		X
http://herschel.esac.esa.int/Science_Archive.shtml					javaWebStart interface see Johan screenshots on the mailinglist : 05/02/2014 14:54
http://telbib.eso.org/?boolany=or&boolaut=or&boolti=or&yearto=2014&boolins=or&telescope[]=%22ALMA%22&booltel=or&search=Search	X	?	?	?	base de données de publi qui renvoie vers l'archive alma
http://almascience.eso.org/aq/	X	?	?	?	Archive Eso. simple d'utilisation recuperation de tgz sur pages web+ script shell

Infrastructure documentation

Accounts

• create the existdb user account with bash account (can be done during install process)

Linux packages (as root)

• apt-get install -y postgresql libpostgresql-jdbc-java openjdk-7-jdk apache2 vim indent
• Add gavo deb sources (http://soft.g-vo.org/repo)

cat > /etc/apt/sources.list.d/gavo.list << EOF 
# oidb requirements
# http://docs.g-vo.org/DaCHS/install.html#debian-systems
deb http://vo.ari.uni-heidelberg.de/debian stable main
deb-src http://vo.ari.uni-heidelberg.de/debian stable main
EOF 
wget -qO - http://docs.g-vo.org/archive-key.asc | apt-key add -
apt-get update

• apt-get install postgresql-9.1-pgsphere

Manual software package (as existdb user)

• wget http://optimate.dl.sourceforge.net/project/exist/Stable/2.2/eXist-db-setup-2.2.jar
• mkdir exist-db ; cd exist-db/
• java -jar ~/eXist-db-setup-2.2.jar -console

existdb@oidb:~/exist-db$ java -jar ~/eXist-db-setup-2.2.jar -console
Select target path [/home/existdb/exist-db] 

press 1 to continue, 2 to quit, 3 to redisplay
1
Set Data Directory
Please select a directory where eXist-db will keep its data files. On Windows, this should be outside the 'Program Files' directory. Please make sure eXist can write to the directory it is installed in.
Data dir:  [webapp/WEB-INF/data] 
/home/existdb/data
press 1 to continue, 2 to quit, 3 to redisplay
1
Set Admin Password and Configure Memory
Enter password:  [] 
XXXX
Enter password:  [XXXX] 
XXXX
------------------------------------------

Maximum memory in mb: [1024] 
2048
Cache memory in mb: [128] 
512
press 1 to continue, 2 to quit, 3 to redisplay
1
[ Starting to unpack ]
...
--- Initialization complete. Shutdown embedded database instance ---
[ Console installation done ]

   * open dahsboard/package-manager and remove useless applications (demo/client...)

as root:

• cd /etc/init.d
• ln -s ~existdb/exist-db/tools/wrapper/bin/exist.sh .
• update-rc.d exist.sh defaults

• set memory options for wrapper:

root@oidb:/home/existdb/exist-db# grep 2048 /home/existdb/exist-db/tools/wrapper/bin/../conf/wrapper.conf
wrapper.java.initmemory=2048
wrapper.java.maxmemory=2048

Sources repository

Main oidb sources are located under the next repository:

• https://svn.jmmc.fr/jmmc-web/trunk/exist/
• https://svn.jmmc.fr/jmmc-sw/oiTools/trunk/

eXist-db Setup

• Activate java binding for jmal : set xquery/@enable-java-binding to yes in exist/conf.xml
• in the futur: wait Patrick's green signal install jmmc-realm module to enable JMMC's authentication mecanism (look installation details in JMMCRealm.java)
• Restrict guest access to exide : set @guest to yes in existdb:///db/apps/exide/configuration.xml
• Run daemon as existd user : set in /etc/init.d/exist

• install jmmc-resource.xar library
• install oidb-data.xar (only if not yet present!!)
• install oidb.xar

TAP services:

TAPLib

Code pour wrapping

TBD

• prepare directory for taplib logs

mkdir -p /var/log/oidb/taplib
chmod -R 775 /var/log/oidb
chgrp -R jmmc /var/log/oidb

Note: for the time being, log location (OiDBServiceConnection.java), JDBC url and user identification (OiDBDBConnection.java) are hardcoded in OiDBTAP source code.

installation en tant que servlet dans jetty

• copy .jar to eXist-db lib directory

cp /usr/share/java/postgresql-8.4-703.jdbc4.jar ~existdb/exist-db/lib/user/
cp taplib.jar binarySavot.jar cds.savot.common.jar cds.savot.model.jar cds.savot.pull.jar cds.savot.writer.jar cos.jar kxml2-min.jar ~existdb/exist-db/lib/user/
cp OiDBTAP.jar ~existdb/exist-db/lib/user/

• eXist-db configuration for taplib

#   add to ~existdb/exist-db/webapp/WEB-INF/web.xml
    <servlet>
        <servlet-name>OiDBTAPServlet</servlet-name>
        <servlet-class>fr.jmmc.oidb.tap.OiDBTAPServlet</servlet-class>
    </servlet>

#   add to ~existdb/exist-db/webapp/WEB-INF/controller-config.xml
    <forward pattern="/tap" servlet="OiDBTAPServlet"/>

Note: test for taplib installation (after eXist-db restart and DB configuration)

curl -v http://localhost:9090/exist/tap/capabilities

Dependance avec pgsphere

• Installing PostgreSQL 8.4.x on CentOS 5.5

# http://heatware.net/linux-unix/how-install-postgresql-8-4-centos-5/

#    Download the PostgreSQL 8.4 repository RPM
wget http://yum.pgrpms.org/reporpms/8.4/pgdg-centos-8.4-2.noarch.rpm
#    Add the repository
rpm -ivh pgdg-centos-8.4-2.noarch.rpm
#    Use yum to install PostgresSQL
yum install -y postgresql postgresql-server
#    Initialize the database
service postgresql initdb
#    Start the database
service postgresql restart
#    Automatically start database on boot
chkconfig postgresql on

• Installing PGSphere

# install PGSphere
yum install pgsphere

• Install PostgreSQL JDBC driver

# install postgresql jdbc driver
yum install postgresql-jdbc

PostgreSQL configuration

• Create database and database user

#    create PostgreSQL user
sudo -u postgres createuser --no-createdb --no-createrole --no-superuser --password --encrypted oidb

#    create PostgreSQL database
sudo -u postgres createdb --owner=oidb --encoding=UTF-8 oidb

#    init pgsphere on new database
sudo -u postgres psql -d oidb -f /usr/share/pgsphere/pg_sphere.sql

# set pass

• Setup database permissions

#    add to the top of /var/lib/pgsql/data/pg_hba.conf the following lines:
local   oidb        postgres                          ident
local   oidb        all                               md5

#    restart postgres
service postgres restart

• Create table

psql -U oidb -d oidb -W -f oidb.postgres.sql

• oidb.postgres.sql: PostgreSQL table creation script