Obsportal <-> L0 OiDB coupling

Since Obsportal is now online and makes a lot of obslogdata massage it now makes sense to load this source of information into OiDB

Here comes a rough mapping table to start the link on top of current search.votable request (2min query / 27MB of gzip xml)

dataproduct_type String     High level scientific classification of the data product meta.id;class
OK constant (even if empty could be replaced by visibility)
calib_level Integer     Amount of data processing that has been applied to the data (0,1,2,3) meta.code;obs.calib
OK constant : 0
target_name String     Object of interest meta.id;src
OK obsportal.target_name
obs_id String     Unique identifier for an observation meta.id

OK obsportal.exp_header_id

KO in OiDB -> PIONIER's obs_id values should be placed in progid and replaced by proper value

obs_collection String     Name of the data collection (e.g. project name) this data belongs to meta.id

OK constant : jmmc_obsportal TBC

In fact obsportal may complete multiples collection. The first one will be the eso-l0 one that was synced by CDS.

obs_creator_name String     Name of the creator of the data meta.id
OK constant: jmmc-tech-group / Bourgès TBC
obs_release_date String   Iso8601->Time Observation release date time.release
obsportal.exp_date_release
obs_publisher_did String     Dataset identifier given by the publisher meta.ref.url;meta.curation
unused on OiDB
bib_reference String     Service bibliographic reference meta.bib.bibcode
NA
data_rights String     Public/Secure/Proprietary meta.code
OK constant : secure
access_url String     URL used to obtain the data set. meta.ref.url
OK concatenation : http://obs.jmmc.fr/detail/exposure/${obsportal.exp_id}
access_format String     MIME type of the resource at access_url meta.code.mime

NA for L0 (no data) or html ?

Remark: ObsCore1.1 mentions application/x-fits-uvfits that we can follow for oifits ? GD: Yes we should use /x-fits-oifits

access_estsize Long kbyte   Estimated size of data product phys.size;meta.file
NA for L0
s_ra Double deg   Right ascension of (center of) observation, ICRS pos.eq.ra;meta.main
OK obsportal.target_ra
s_dec Double deg   Declination of (center of) observation, ICRS pos.eq.dec;meta.main
OK obsportal.target_dec
s_fov Float deg   Approximate spatial extent for the region covered by the observation phys.angSize;instr.fov
NA
s_region Float     Region covered by the observation, as a polygon phys.angArea;obs
NA
s_resolution Float arcsec   Best spatial resolution within the data set pos.angResolution
NA
t_min Float d MJD->Time Lower bound of times represented in the data set, as MJD time.start;obs.exposure
obsportal.exp_mjd_start
t_max Float d MJD->Time Upper bound of times represented in the data set, as MJD time.end;obs.exposure
obsportal.exp_mjd_end
t_exptime Float s   Total exposure time time.duration;obs.exposure
obsportal.exp_mjd_end - obsportal.exp_mjd_start
t_resolution Float s   Minimal significant time interval along the time axis time.resolution
NA
em_min Float m   Minimal wavelength represented within the data set em.wl;stat.min
obsportal.instrument_em_min
em_max Float m   Maximal wavelength represented within the data set em.wl;stat.max
obsportal.instrument_em_max
em_res_power Float     Spectral resolving power delta_lambda / lamda spect.resolution
obsportal.instrument_em_res_power
o_ucd String     UCD for the product's observable meta.ucd
NA
pol_states String     List of polarization states in the data set meta.code;phys.polarization
??
facility_name String     Name of the facility at which data was taken meta.id;instr.tel
OK obsportal.interferometer_name
instrument_name String     Name of the instrument that produced the data meta.id;instr
OK obsportal.instrument_name
instrument_mode String     Instrument mode  
OK obsportal.instrument_mode . add obsportal.instrument_submode ?
quality_level Integer     Quality data level estimated by data provider meta.code.qual
computation from obsportal.exp_tau0 obsportal.exp_seeing ?
nb_channels Integer     number of spectral channels  
inferred from AsproConfig using obsportal.instrument_name/obsportal.instrument_mode/obsportal.instrument_submode ?
nb_vis Integer     number of OI_VIS data (complex visiblity)  
NA
nb_vis2 Integer     number of OI_VIS2 data (square visiblity)  
NA
nb_t3 Integer     number of OI_T3 data (closure phase)  
NA
keywords String        
??
subdate String   Iso8601->Time submission date  
current datetime
id Long internal ID (granule ID)   Object ID  
AUTO
progid String     program or proposal identifier  

OK obsportal.obs_program

TBC : obs_id goes from 1 to 9999 in obsportal for the whole 319791 records

datapi String     Data PI  
NA for L0 TBC
access_md5 String     md5sum of the resource at access_url  
NA
Some records should be filtered during the injest :
  • keep science only -> obsportal.obs_type='science' ?
  • ???

Add interferometer_stations as oidb metadata.

Obsportal votable could also propose to export mode a full one that carry all and a more lite one without verbose validation informations.

Obsportal <-> L2 OiDB coupling

L2 injest of PIONIER collection requires to complete some metadata not present in the OIFITS keywords.

At present time PIs are retrieved from the archive jmmc-eso:get-pi-from-progid($progid)

The change operated using obsportal as intermediate source of information could be indirect. The missing L2 information could be retrieved on top of already present L0 data using proper key ( obsportal.exp_header_id == OIFITS.ARCFILE ). However OiDB decided not to store datapis of L0 then DataPis could not be retrieved for L2 on top of L0 records frown

How to fix that ?

Add pi-coi in obsportal votable.

Do not store in the OiDB L0 records but in xml database : prog-id - pi-cois

Table metadata samples :

Table name: Statistics for ESO_TAP dbo.raw

dataLocation: TAP_5_dbo.raw
statRows: 361
Name Mean SD Minimum Maximum nGood
access_url     https://dataportal.eso.org/dataportal... https://dataportal.eso.org/dataportal... 361
datalink_url     http://archive.eso.org/datalink/links... http://archive.eso.org/datalink/links... 361
date_obs     2004-12-25T21:37:00.9989 2005-02-26T10:16:18.1015 361
dec -301.0227 287.67047 -596.52323555 15.87714 361
dec_pnt -22.998003 34.063705 -70.19476 15.87714 186
det_chip1id         0
det_chop_ncycles         0
det_dit 0.45429364 0.8346642 0 2 361
det_expid         0
det_ndit 147.39058 284.34305 1 1000 361
dp_cat     CALIB SCIENCE 361
dp_id     AMBER.2004-12-25T21:37:00.999 AMBER.2005-02-26T10:16:18.102 361
dp_tech     INTERFEROMETRY INTERFEROMETRY 361
dp_type     3P2V WAVE,3TEL 361
ecl_lat -30.475166 26.049498 -66.309153 0.654471 186
ecl_lon 192.56058 132.98537 0.0 348.802413 186
exp_start     2004-12-25T21:37:00Z 2005-02-26T10:16:18Z 361
exposure 8.481276 16.470871 0.058 82.0 356
filter_path         0
gal_lat -15.380623 15.691006 -60.188552 1.646137 186
gal_lon 211.57108 80.30247 21.535596 296.367403 186
grat_path     GHR PRISM 361
gris_path     NAR_SLT,3T_JHK NAR_SLT,3T_K 361
ins_mode     3Tstd_High_K_1_2.166 3Tstd_Medium_K_1_2.1 361
instrument     AMBER AMBER 361
last_mod_date     2010-11-30T09:20:26Z 2010-11-30T09:20:26Z 361
mjd_obs 53396.875 30.500704 53364.902 53427.43 361
ob_id 5.5059866E8 4.99765952E8 200146604 1268183793 361
ob_name     HR_K_ETA_CAR MR_K_HD100546_NATTA_1 361
object     3P2V WAVE,3TEL 343
obs_mode     s s 361
origfile     AMBER_3TSTD_ACQ055_0015.fits AMBER_P2VM_3T_2005-02-26T10:16:28.fits 361
period         0
pi_coi     AMBER SCIENCE DEMONS AMBER SCIENCE DEMONS 361
prog_id     60.A-9055(J) 60.A-9055(J) 361
prog_title     AMBER SCIENCE DEMONSTRATION AMBER SCIENCE DEMONSTRATION 361
prog_type 0.0 0.0 0 0 361
ra -226.02208 363.05057 -596.52323555 299.53385388 361
ra_pnt 122.567726 71.754524 0.0 299.533854 186
release_date     2004-12-25T21:37:00Z 2006-02-26T09:31:53Z 361
slit_path         0
target     ETA-CAR SAO237916 361
tel_airm_end         0
tel_airm_start         0
tel_alt         0
tel_ambi_fwhm_end         0
tel_ambi_fwhm_start         0
tel_ambi_pres_end         0
tel_ambi_pres_start         0
tel_ambi_rhum         0
tel_az         0
telescope     ESO-VLTI-U234 NOT_SPECIFIED 343
tpl_expno 7.218659 4.244221 1 15 343
tpl_id     AMBER_3T_cal_frnsrc AMBER_3Tstd_obs_1row 343
tpl_name     AMBER 3 telescopes / 1 row standard o... AMBER 3T standard acquisition 343
tpl_nexp 13.233236 3.6161854 1 15 343
tpl_seqno 1.8227147 1.7381604 1 7 361
tpl_start     2004-12-25T21:29:26 2005-02-26T10:09:59 343

Table name: Statistics for ESO TAP ivoa.ObsCore

dataLocation: TAP_7_ivoa.ObsCore
statRows: 1000

Name Mean SD Minimum Maximum nGood
abmaglim         0
access_estsize 85.64 2.0426455 80 92 1000
access_format     application/x-votable+xml;content=dat... application/x-votable+xml;content=dat... 1000
access_url     http://archive.eso.org/datalink/links... http://archive.eso.org/datalink/links... 1000
bib_reference         0
calib_level 2.0 0.0 2 2 1000
dataproduct_subtype         0
dataproduct_type     visibility visibility 1000
dp_id     ADP.2017-04-04T14:22:18.367 ADP.2018-10-22T14:40:47.123 1000
em_max 1.7599808E-6 1.8412344E-8 1.654116E-6 1.774278E-6 1000
em_min 1.5240973E-6 2.4750758E-8 1.511285E-6 1.6799380000000002E-6 1000
em_res_power 33.349228 4.976259 5.5137 34.6358 1000
em_xel 1.0 0.0 1 1 1000
facility_name     ESO-VLTI-A1234 ESO-VLTI-U1234 1000
filter         0
gal_lat -15.536459 27.223131 -82.701899 72.176321 1000
gal_lon 223.69342 116.30873 0.80439 359.933529 1000
instrument_name     PIONIER PIONIER 1000
multi_ob     S S 1000
n_obs 1.0 0.0 1 1 1000
o_calib_status     any any 1000
o_ucd         0
obs_collection     PIONIER PIONIER 1000
obs_creator_did     ivo://eso.org/origfile?PIONI.2015-10-... ivo://eso.org/origfile?PIONI.2018-06-... 1000
obs_creator_name     ANDERSON, RICHARD WITTKOWSKI, MARKUS 1000
obs_id     1236337 2027314 1000
obs_publisher_did     ivo://eso.org/ID?ADP.2017-04-04T14:22... ivo://eso.org/ID?ADP.2018-10-22T14:40... 1000
obs_release_date     2015-11-29T02:31:44Z 2019-06-05T07:25:39Z 1000
obs_title         0
obstech     INTERFEROMETRY INTERFEROMETRY 1000
p3orig     EDP IDP 1000
pol_states         0
pol_xel         0
proposal_id     0100.C-0278(J) 597.D-0727(A) 1000
publication_date     2017-04-04T14:53:52Z 2018-10-22T17:08:02Z 1000
s_dec -37.595093 26.249413 -76.61186 27.46021 1000
s_fov         0
s_pixel_scale         0
s_ra 189.17336 95.98031 0.455556 359.97951 1000
s_region     POSITION J2000 0.4555560000000014 -3.... POSITION J2000 97.967765 5.0331 1000
s_resolution         0
s_xel1         0
s_xel2         0
snr         0
t_exptime 234.39381 145.89099 51.2 1177.6 1000
t_max 57772.8 259.5562 57298.09441799 58274.30534132 1000
t_min 57772.797 259.55573 57298.09293651 58274.30385984 1000
t_resolution 241.3017 158.16522 51.199776 1338.315264 1000
t_xel         0
target_name     2MASS_J01373545-0645375 thetaSco 1000

Table name: Statistics for oidb

dataLocation: OiDB:votable.xql?perpage=25&order=t_min&caliblevel=1%2C2%2C3
statRows: 8736

+-------------------+--------------+--------------+------------------------------------------+------------------------------------------+-------+

Name Mean SD Minimum Maximum nGood

+-------------------+--------------+--------------+------------------------------------------+------------------------------------------+-------+

dataproduct_type         0
calib_level 2.1394231 0.34638748 2 3 8736
target_name     10_TAU zeta_Gem 8736
obs_id     0100.D-0024(A) Maintenance 6955
obs_collection     2727e73d-35d5-43e6-8a36-c9e1da31d294 iota 8736
obs_creator_name     Bernard Lazareff Nicolas Nardetto 8736
obs_release_date     1999-09-29T10:52:30 2021-03-01T08:29:42 8736
obs_publisher_did         0
bib_reference     2009A&A...493L..17C 2019ApJ...883...89O 807
data_rights     public secure 8736
access_url     /exist/apps/oidb-data/oifits/staging/... http://cdsarc.u-strasbg.fr/vizier/ftp... 8736
access_format     application/fits application/fits 7240
access_estsize 96.36161 73.9986 13 676 8736
s_ra 177.86417 94.429085 0.0 359.97951 8736
s_dec -31.354578 29.259974 -83.10761 68.163056 8736
s_fov         0
s_region         0
s_resolution         0
t_min 56167.242 5204.3647 -1.0 58909.355 8736
t_max 56167.246 5204.3633 -1.0 58909.355 8736
t_exptime 178431.16 526231.25 -561.22 7677132.5 8736
t_resolution         0
em_min 1.8477201E-6 1.0256063E-6 5.0E-7 7.622082E-6 8736
em_max 2.6952448E-6 3.2755117E-6 5.0E-7 1.3767216E-5 8736
em_res_power 495.37125 3623.3855 6.9476286E-5 35743.39 8736
o_ucd         0
pol_states         0
facility_name     CHARA VLTI 8734
instrument_name     6way_OB01 VEGA 8736
instrument_mode     High_K_1_2.172 Medium_K_1_2.3 152
quality_level 0.0 0.0 0 0 278
nb_channels 24.208677 65.90274 1 508 8736
nb_vis 0.21806319 0.7661262 0 10 8736
nb_vis2 7.485119 12.007391 0 300 8736
nb_t3 4.710165 7.7968287 0 200 8736
keywords     Astrophysics - Solar and Stellar Astr... techniques: interferometric ; galaxie... 807
subdate     2015-06-22T22:27:21 2020-05-11T13:08:12 8736
id 422905.6 160707.86 38659 525364 8736
progid     082.D-0146 60.A-9054 79
datapi     ABSIL ZUNIGA-FERNANDEZ, S 8715
access_md5         0

Table name: Statistics for obsportal

dataLocation: /home/mellag/tmp/search.votable
statRows: 319791

Name Mean SD Minimum Maximum nGood
exp_id     AMBER.2004-05-31T06:02:46.487_1 PIONI.2020-03-24T08:44:48.208_1 319791
exp_header_id     AMBER.2004-05-31T06:02:46.487 PIONI.2020-03-24T08:44:48.208 319791
exp_mjd_start 56545.707 1545.9994 52804.30111111 58932.3644468505 319791
exp_mjd_end 56545.71 1545.9995 52804.3016722211 58932.3650394431 319791
exp_projected_baselines     {'A0-B2': {'length': {'start': '0.000... {'U4-U3': {'length': {'start': '58.05... 319791
exp_validation_level 1.0 0.0 1 1 319791
exp_validation_log     ["Invalid value for 'HIERARCH ESO ISS... ["No value for 'HIERARCH ESO OBS TARG... 9761
exp_tau0 0.0042860596 0.008200745 2.925E-4 1.689503 319588
exp_temperature 12.737826 2.619762 0.34 20.66 319791
exp_seeing 0.87094575 0.53003204 -1.0 8.49 319741
exp_date_updated     2010-11-30 10:20:26+01:00 2020-03-27 09:26:53+01:00 319791
exp_date_release     2004-01-15 17:07:49+01:00 2021-03-24 09:45:18+01:00 319791
obs_id 23784.17 14976.115 1 45773 319791
obs_type     calibrator science 319791
obs_program     0100.B-0582(A) Maintenance 319791
obs_program_title     $DELTA$ SCO : FOLLOW-UP OF THE CIRCUM... ``THE ANSWER MY FRIEND... --- TESTING... 319717
obs_program_pi_coi     A JORISSEN/ BERGER/ HILLEN/ BOFFIN/ B... ZUNIGA-FERNANDEZ/ OLOFSSON/ BAYO/ COR... 319717
interferometer_name     VLTI VLTI 319791
interferometer_stations     A0 B2 C1 E0 U4 U3 319791
instrument_name     AMBER PIONIER 319791
instrument_mode     FREE-H SMALL-H 319791
instrument_submode     DUAL SINGLE 49491
instrument_em_min 3.381424 2.7538488 1.469 8.0 319791
instrument_em_max 5.1242223 4.724413 1.7959 13.03818 319791
instrument_em_res_power 1335.7579 3184.8386 6.0 12000.0 319791
target_id 18128.576 10688.66 1 34241 319791
target_name     05_07_42_000_24_47_5 zpup 319356
target_ra 180.03163 93.55215 0.026005 359.97951 319791
target_dec -31.22812 25.1721 -87.48218 43.94645 319791
-- Guillaume Mella - 2020-05-26

Comments

Edit | Attach | Watch | Print version | History: r10 < r9 < r8 < r7 < r6 | Backlinks | Raw View | Raw edit | More topic actions
Topic revision: r10 - 2020-06-09 - GuillaumeMella
 
This site is powered by the TWiki collaboration platform Powered by PerlCopyright © 2008-2024 by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding TWiki? Send feedback