(r27) OiDbImplementationsNotes < Jmmc/Software

Tags: view all tags
This page is listing technical discussions and notes on the !OiDB project. See the definition of the OiDB project in the document from the [[OiDbReunion26June2013][2013-06-26 meeeting]] or in the [[OiDbSpecs][wiki page]].


%TOC%


---+ Notes

---++ Collection

Define a collection as a way for a submitter (dataPI) submitting data to group entries together.

---++ Push/Pull for database insertion

Methods for adding data :

   * submit files/collection: the user make HTTP PUT request to an URL on the server for his OIFits files or collection of OIFits files. The portal displays a Web form for manually importing new data.

   * download files/collection: the application is in charge of automatically downlaading the files from the server specified by the user. TODO: define format: XML files with resources to analyze ? parsing HTML file listing ? The portal let the user specify the address of his server and the access conditions.


---++ Data update

Files/collections may be modified by the owner from time to time. The application should regularly download and reprocess the distant repository to stay up to date.

Ask the user for reimporting the data (user resend data)? Automatic import (identify modifications on files: checksum)?

Is it possible to use the name of the file as observation ID?


---++ Data display

To retrieve external data :

   * turn the portal into a proxy: at user request, the portal check the repository, download the file and return it to the user. The application knows for sure that the user has download the file.

<img src="%ATTACHURLPATH%/external_data_proxy.png"/>

   * the user perform a request on the application that return the URL for the requested resource. The user then make an HTTP GET request on this URL to download the document. The application can log the request (the user may have downloaded the file but not guaranteed).

<img src="%ATTACHURLPATH%/external_data_direct.png"/>

   * make a redirection (similar to Google search results): slightly different from the previous proposition, the user performs a request on the portal that issue a redirect (URL forwarding, HTTP 303) to the external repository. Do all clients and libs support 303 redirections ?


---++ Private data

Some data may not be downloadable until a given amount of time. Still should let people get in contact with the dataPI.


---++ Changes to the current database schema

   * sort by collection.

   * link data and dataPI.

   * add fields for descriptions and comments on data (file, collection target).

   * make link between observation in the database and scientific publication (bibliographic reference)

   * keep versions from a same source.

---++ Data model

This information is now in a [[OiDbColumns][dedicated topic]]

---++ OIFITS

Define/extract metadata from submitted OIFits files.

   * manage datum with flag=true (OI_VIS, OI_VIS2, OI_T3).

   * take into account repeated wavelength (more than one OI_WAVELENGTH in a single OIFits with the same wavelength value).

   * measurements from more than one instrument in the same file (OI_WAVELENGTH tables with different INSNAME while observing the same target).


---++ Publishing the the VO
See : 
   * http://wiki.ivoa.net/twiki/bin/view/IVOA/PublishingInTheVONew

---++ Web portal examples

|*Web Site*|*MultiCriterii*|*ADQL*|*TAP/ObsCore*|*Samp*|*Description/comments*|
| http://portal-nvo.noao.edu/adql/query | X | X | | | |
| http://voparis-srv.obspm.fr/portal/vo.php | X | | ? | X | |
| http://saada.u-strasbg.fr/taphandle/ |  | X | | X | |
| http://labs.adsabs.harvard.edu/adsabs/ | X | NC | NC | NC | Not for observation data ;) |
| http://polarbase.irap.omp.eu/ | X | | X | |  |
| http://herschel.esac.esa.int/Science_Archive.shtml |  | | | | javaWebStart interface  see Johan screenshots on the mailinglist : 05/02/2014 14:54 |
| http://telbib.eso.org/?boolany=or&boolaut=or&boolti=or&yearto=2014&boolins=or&telescope[]=%22ALMA%22&booltel=or&search=Search | X | ? | ? | ? | base de données de publi qui renvoie vers l'archive alma |
| http://almascience.eso.org/aq/ | X | ? | ? | ? | Archive Eso. simple d'utilisation recuperation de tgz sur pages web+ script shell |


---+ Infrastructure documentation

---++ Accounts
   * create the =existdb= user account with bash account (can be done during install process)

---++ Sources repository
Main oidb sources are located under the next repository:
   * https://svn.jmmc.fr/jmmc-web/trunk/exist/
   * https://svn.jmmc.fr/jmmc-sw/oiTools/trunk/

---++ eXist-db Setup
   * Activate java binding for jmal : set xquery/@enable-java-binding to yes in exist/conf.xml
   * =in the futur: wait Patrick's green signal= install jmmc-realm module to enable JMMC's authentication mecanism (look installation details in JMMCRealm.java)
   * Restrict guest access to exide : set @guest to yes in existdb:///db/apps/exide/configuration.xml
   * Run daemon as existd user : set in /etc/init.d/exist

   * install jmmc-resource.xar library 
   * install oidb-data.xar (only if not yet present!!)
   * install oidb.xar


---++ TAP services:
---+++ TAPLib
---++++ Code pour wrapping
TBD

   * prepare directory for taplib logs
<verbatim>
mkdir -p /var/log/oidb/taplib
chmod -R 775 /var/log/oidb
chgrp -R jmmc /var/log/oidb
</verbatim>

Note: for the time being, log location (!OiDBServiceConnection.java), JDBC url and user identification (!OiDBDBConnection.java) are hardcoded in !OiDBTAP source code.

---++++ installation en tant que servlet dans jetty

   * copy .jar to eXist-db lib directory
<verbatim>
cp /usr/share/java/postgresql-8.4-703.jdbc4.jar ~existdb/exist-db/lib/user/
cp taplib.jar binarySavot.jar cds.savot.common.jar cds.savot.model.jar cds.savot.pull.jar cds.savot.writer.jar cos.jar kxml2-min.jar ~existdb/exist-db/lib/user/
cp OiDBTAP.jar ~existdb/exist-db/lib/user/
</verbatim>

   * eXist-db configuration for taplib
<verbatim>
#   add to ~existdb/exist-db/webapp/WEB-INF/web.xml
    <servlet>
        <servlet-name>OiDBTAPServlet</servlet-name>
        <servlet-class>fr.jmmc.oidb.tap.OiDBTAPServlet</servlet-class>
    </servlet>

#   add to ~existdb/exist-db/webapp/WEB-INF/controller-config.xml
    <forward pattern="/tap" servlet="OiDBTAPServlet"/>
</verbatim>

Note: test for taplib installation (after eXist-db restart and DB configuration)
<verbatim>
curl -v http://localhost:9090/exist/tap/capabilities
</verbatim>

---++++ Dependance avec pgsphere
   * Installing !PostgreSQL 8.4.x on !CentOS 5.5
<verbatim>
# http://heatware.net/linux-unix/how-install-postgresql-8-4-centos-5/

#    Download the PostgreSQL 8.4 repository RPM
wget http://yum.pgrpms.org/reporpms/8.4/pgdg-centos-8.4-2.noarch.rpm
#    Add the repository
rpm -ivh pgdg-centos-8.4-2.noarch.rpm
#    Use yum to install PostgresSQL
yum install -y postgresql postgresql-server
#    Initialize the database
service postgresql initdb
#    Start the database
service postgresql restart
#    Automatically start database on boot
chkconfig postgresql on
</verbatim>

   * Installing PGSphere
<verbatim>
# install PGSphere
yum install pgsphere
</verbatim>

   * Install !PostgreSQL JDBC driver
<verbatim>
# install postgresql jdbc driver
yum install postgresql-jdbc
</verbatim>

---++++ !PostgreSQL configuration

   * Create database and database user
<verbatim>
#    create PostgreSQL user
sudo -u postgres createuser --no-createdb --no-createrole --no-superuser --password --encrypted oidb

#    create PostgreSQL database
sudo -u postgres createdb --owner=oidb --encoding=UTF-8 oidb

#    init pgsphere on new table
sudo -u postgres psql -d oidb -f /usr/share/pgsphere/pg_sphere.sql
</verbatim>

# set pass

   * Setup database permissions
<verbatim>
#    add to the top of /var/lib/pgsql/data/pg_hba.conf the following lines:
local   oidb        postgres                          ident
local   oidb        all                               md5

#    restart postgres
service postgres restart
</verbatim>

   * Create table
<verbatim>
psql -U oidb -d oidb -W -f oidb.postgres.sql
</verbatim>


   * [[%ATTACHURL%/oidb.postgres.sql][oidb.postgres.sql]]: !PostgreSQL table creation script
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
sql	oidb.postgres.sql	r3 r2 r1	manage	2.1 K	2014-11-18 - 10:39	PatrickBernaud	PostgreSQL table creation script
Topic revision: r27 - 2014-11-19 - GuillaumeMella