Obsportal <-> L0 OiDB coupling

Since Obsportal is now online and makes a lot of obslogdata massage it now makes sense to load this source of information into OiDB

Here comes a rough mapping table to start the link on top of current search.votable request (2min query / 27MB of gzip xml)

dataproduct_type	String			High level scientific classification of the data product	meta.id;class
	OK constant (even if empty could be replaced by visibility)
calib_level	Integer			Amount of data processing that has been applied to the data (0,1,2,3)	meta.code;obs.calib
	OK constant : 0
target_name	String			Object of interest	meta.id;src
	OK obsportal.target_name
obs_id	String			Unique identifier for an observation	meta.id
	OK obsportal.exp_header_id KO in OiDB -> PIONIER's obs_id values should be placed in progid and replaced by proper value
obs_collection	String			Name of the data collection (e.g. project name) this data belongs to	meta.id
	OK constant : jmmc_obsportal TBC In fact obsportal may complete multiples collection. The first one will be the eso-l0 one that was synced by CDS.
obs_creator_name	String			Name of the creator of the data	meta.id
	OK constant: jmmc-tech-group / Bourgès TBC
obs_release_date	String		Iso8601->Time	Observation release date	time.release
	obsportal.exp_date_release
obs_publisher_did	String			Dataset identifier given by the publisher	meta.ref.url;meta.curation
	unused on OiDB
bib_reference	String			Service bibliographic reference	meta.bib.bibcode
	NA
data_rights	String			Public/Secure/Proprietary	meta.code
	OK constant : secure
access_url	String			URL used to obtain the data set.	meta.ref.url
	OK concatenation : http://obs.jmmc.fr/detail/exposure/${obsportal.exp_id}
access_format	String			MIME type of the resource at access_url	meta.code.mime
	NA for L0 (no data) or html ? Remark: ObsCore1.1 mentions application/x-fits-uvfits that we can follow for oifits ? GD: Yes we should use /x-fits-oifits
access_estsize	Long	kbyte		Estimated size of data product	phys.size;meta.file
	NA for L0
s_ra	Double	deg		Right ascension of (center of) observation, ICRS	pos.eq.ra;meta.main
	OK obsportal.target_ra
s_dec	Double	deg		Declination of (center of) observation, ICRS	pos.eq.dec;meta.main
	OK obsportal.target_dec
s_fov	Float	deg		Approximate spatial extent for the region covered by the observation	phys.angSize;instr.fov
	NA
s_region	Float			Region covered by the observation, as a polygon	phys.angArea;obs
	NA
s_resolution	Float	arcsec		Best spatial resolution within the data set	pos.angResolution
	NA
t_min	Float	d	MJD->Time	Lower bound of times represented in the data set, as MJD	time.start;obs.exposure
	obsportal.exp_mjd_start
t_max	Float	d	MJD->Time	Upper bound of times represented in the data set, as MJD	time.end;obs.exposure
	obsportal.exp_mjd_end
t_exptime	Float	s		Total exposure time	time.duration;obs.exposure
	obsportal.exp_mjd_end - obsportal.exp_mjd_start
t_resolution	Float	s		Minimal significant time interval along the time axis	time.resolution
	NA
em_min	Float	m		Minimal wavelength represented within the data set	em.wl;stat.min
	obsportal.instrument_em_min
em_max	Float	m		Maximal wavelength represented within the data set	em.wl;stat.max
	obsportal.instrument_em_max
em_res_power	Float			Spectral resolving power delta_lambda / lamda	spect.resolution
	obsportal.instrument_em_res_power
o_ucd	String			UCD for the product's observable	meta.ucd
	NA
pol_states	String			List of polarization states in the data set	meta.code;phys.polarization
	??
facility_name	String			Name of the facility at which data was taken	meta.id;instr.tel
	OK obsportal.interferometer_name
instrument_name	String			Name of the instrument that produced the data	meta.id;instr
	OK obsportal.instrument_name
instrument_mode	String			Instrument mode
	OK obsportal.instrument_mode . add obsportal.instrument_submode ?
quality_level	Integer			Quality data level estimated by data provider	meta.code.qual
	computation from obsportal.exp_tau0 obsportal.exp_seeing ?
nb_channels	Integer			number of spectral channels
	inferred from AsproConfig using obsportal.instrument_name/obsportal.instrument_mode/obsportal.instrument_submode ?
nb_vis	Integer			number of OI_VIS data (complex visiblity)
	NA
nb_vis2	Integer			number of OI_VIS2 data (square visiblity)
	NA
nb_t3	Integer			number of OI_T3 data (closure phase)
	NA
keywords	String
	??
subdate	String		Iso8601->Time	submission date
	current datetime
id	Long	internal ID (granule ID)		Object ID
	AUTO
progid	String			program or proposal identifier
	OK obsportal.obs_program TBC : obs_id goes from 1 to 9999 in obsportal for the whole 319791 records
datapi	String			Data PI
	NA for L0 TBC
access_md5	String			md5sum of the resource at access_url
	NA

• keep science only -> obsportal.obs_type='science' ?
• ???

Add interferometer_stations as oidb metadata.

Obsportal votable could also propose to export mode a full one that carry all and a more lite one without verbose validation informations.

Obsportal <-> L2 OiDB coupling

L2 injest of PIONIER collection requires to complete some metadata not present in the OIFITS keywords.

At present time PIs are retrieved from the archive jmmc-eso:get-pi-from-progid($progid)

The change operated using obsportal as intermediate source of information could be indirect. The missing L2 information could be retrieved on top of already present L0 data using proper key ( obsportal.exp_header_id == OIFITS.ARCFILE ). However OiDB decided not to store datapis of L0 then DataPis could not be retrieved for L2 on top of L0 records

How to fix that ?

Add pi-coi in obsportal votable.

Do not store in the OiDB L0 records but in xml database : prog-id - pi-cois

Table metadata samples :

Table name: Statistics for ESO_TAP dbo.raw

dataLocation: TAP_5_dbo.raw
statRows: 361

Name	Mean	SD	Minimum	Maximum	nGood
access_url			https://dataportal.eso.org/dataportal...	https://dataportal.eso.org/dataportal...	361
datalink_url			http://archive.eso.org/datalink/links...	http://archive.eso.org/datalink/links...	361
date_obs			2004-12-25T21:37:00.9989	2005-02-26T10:16:18.1015	361
dec	-301.0227	287.67047	-596.52323555	15.87714	361
dec_pnt	-22.998003	34.063705	-70.19476	15.87714	186
det_chip1id					0
det_chop_ncycles					0
det_dit	0.45429364	0.8346642	0	2	361
det_expid					0
det_ndit	147.39058	284.34305	1	1000	361
dp_cat			CALIB	SCIENCE	361
dp_id			AMBER.2004-12-25T21:37:00.999	AMBER.2005-02-26T10:16:18.102	361
dp_tech			INTERFEROMETRY	INTERFEROMETRY	361
dp_type			3P2V	WAVE,3TEL	361
ecl_lat	-30.475166	26.049498	-66.309153	0.654471	186
ecl_lon	192.56058	132.98537	0.0	348.802413	186
exp_start			2004-12-25T21:37:00Z	2005-02-26T10:16:18Z	361
exposure	8.481276	16.470871	0.058	82.0	356
filter_path					0
gal_lat	-15.380623	15.691006	-60.188552	1.646137	186
gal_lon	211.57108	80.30247	21.535596	296.367403	186
grat_path			GHR	PRISM	361
gris_path			NAR_SLT,3T_JHK	NAR_SLT,3T_K	361
ins_mode			3Tstd_High_K_1_2.166	3Tstd_Medium_K_1_2.1	361
instrument			AMBER	AMBER	361
last_mod_date			2010-11-30T09:20:26Z	2010-11-30T09:20:26Z	361
mjd_obs	53396.875	30.500704	53364.902	53427.43	361
ob_id	5.5059866E8	4.99765952E8	200146604	1268183793	361
ob_name			HR_K_ETA_CAR	MR_K_HD100546_NATTA_1	361
object			3P2V	WAVE,3TEL	343
obs_mode			s	s	361
origfile			AMBER_3TSTD_ACQ055_0015.fits	AMBER_P2VM_3T_2005-02-26T10:16:28.fits	361
period					0
pi_coi			AMBER SCIENCE DEMONS	AMBER SCIENCE DEMONS	361
prog_id			60.A-9055(J)	60.A-9055(J)	361
prog_title			AMBER SCIENCE DEMONSTRATION	AMBER SCIENCE DEMONSTRATION	361
prog_type	0.0	0.0	0	0	361
ra	-226.02208	363.05057	-596.52323555	299.53385388	361
ra_pnt	122.567726	71.754524	0.0	299.533854	186
release_date			2004-12-25T21:37:00Z	2006-02-26T09:31:53Z	361
slit_path					0
target			ETA-CAR	SAO237916	361
tel_airm_end					0
tel_airm_start					0
tel_alt					0
tel_ambi_fwhm_end					0
tel_ambi_fwhm_start					0
tel_ambi_pres_end					0
tel_ambi_pres_start					0
tel_ambi_rhum					0
tel_az					0
telescope			ESO-VLTI-U234	NOT_SPECIFIED	343
tpl_expno	7.218659	4.244221	1	15	343
tpl_id			AMBER_3T_cal_frnsrc	AMBER_3Tstd_obs_1row	343
tpl_name			AMBER 3 telescopes / 1 row standard o...	AMBER 3T standard acquisition	343
tpl_nexp	13.233236	3.6161854	1	15	343
tpl_seqno	1.8227147	1.7381604	1	7	361
tpl_start			2004-12-25T21:29:26	2005-02-26T10:09:59	343

Table name: Statistics for ESO TAP ivoa.ObsCore

dataLocation: TAP_7_ivoa.ObsCore
statRows: 1000

Name	Mean	SD	Minimum	Maximum	nGood
abmaglim					0
access_estsize	85.64	2.0426455	80	92	1000
access_format			application/x-votable+xml;content=dat...	application/x-votable+xml;content=dat...	1000
access_url			http://archive.eso.org/datalink/links...	http://archive.eso.org/datalink/links...	1000
bib_reference					0
calib_level	2.0	0.0	2	2	1000
dataproduct_subtype					0
dataproduct_type			visibility	visibility	1000
dp_id			ADP.2017-04-04T14:22:18.367	ADP.2018-10-22T14:40:47.123	1000
em_max	1.7599808E-6	1.8412344E-8	1.654116E-6	1.774278E-6	1000
em_min	1.5240973E-6	2.4750758E-8	1.511285E-6	1.6799380000000002E-6	1000
em_res_power	33.349228	4.976259	5.5137	34.6358	1000
em_xel	1.0	0.0	1	1	1000
facility_name			ESO-VLTI-A1234	ESO-VLTI-U1234	1000
filter					0
gal_lat	-15.536459	27.223131	-82.701899	72.176321	1000
gal_lon	223.69342	116.30873	0.80439	359.933529	1000
instrument_name			PIONIER	PIONIER	1000
multi_ob			S	S	1000
n_obs	1.0	0.0	1	1	1000
o_calib_status			any	any	1000
o_ucd					0
obs_collection			PIONIER	PIONIER	1000
obs_creator_did			ivo://eso.org/origfile?PIONI.2015-10-...	ivo://eso.org/origfile?PIONI.2018-06-...	1000
obs_creator_name			ANDERSON, RICHARD	WITTKOWSKI, MARKUS	1000
obs_id			1236337	2027314	1000
obs_publisher_did			ivo://eso.org/ID?ADP.2017-04-04T14:22...	ivo://eso.org/ID?ADP.2018-10-22T14:40...	1000
obs_release_date			2015-11-29T02:31:44Z	2019-06-05T07:25:39Z	1000
obs_title					0
obstech			INTERFEROMETRY	INTERFEROMETRY	1000
p3orig			EDP	IDP	1000
pol_states					0
pol_xel					0
proposal_id			0100.C-0278(J)	597.D-0727(A)	1000
publication_date			2017-04-04T14:53:52Z	2018-10-22T17:08:02Z	1000
s_dec	-37.595093	26.249413	-76.61186	27.46021	1000
s_fov					0
s_pixel_scale					0
s_ra	189.17336	95.98031	0.455556	359.97951	1000
s_region			POSITION J2000 0.4555560000000014 -3....	POSITION J2000 97.967765 5.0331	1000
s_resolution					0
s_xel1					0
s_xel2					0
snr					0
t_exptime	234.39381	145.89099	51.2	1177.6	1000
t_max	57772.8	259.5562	57298.09441799	58274.30534132	1000
t_min	57772.797	259.55573	57298.09293651	58274.30385984	1000
t_resolution	241.3017	158.16522	51.199776	1338.315264	1000
t_xel					0
target_name			2MASS_J01373545-0645375	thetaSco	1000

Table name: Statistics for oidb

dataLocation: OiDB:votable.xql?perpage=25&order=t_min&caliblevel=1%2C2%2C3
statRows: 8736

+-------------------+--------------+--------------+------------------------------------------+------------------------------------------+-------+

+-------------------+--------------+--------------+------------------------------------------+------------------------------------------+-------+

dataproduct_type					0
calib_level	2.1394231	0.34638748	2	3	8736
target_name			10_TAU	zeta_Gem	8736
obs_id			0100.D-0024(A)	Maintenance	6955
obs_collection			2727e73d-35d5-43e6-8a36-c9e1da31d294	iota	8736
obs_creator_name			Bernard Lazareff	Nicolas Nardetto	8736
obs_release_date			1999-09-29T10:52:30	2021-03-01T08:29:42	8736
obs_publisher_did					0
bib_reference			2009A&A...493L..17C	2019ApJ...883...89O	807
data_rights			public	secure	8736
access_url			/exist/apps/oidb-data/oifits/staging/...	http://cdsarc.u-strasbg.fr/vizier/ftp...	8736
access_format			application/fits	application/fits	7240
access_estsize	96.36161	73.9986	13	676	8736
s_ra	177.86417	94.429085	0.0	359.97951	8736
s_dec	-31.354578	29.259974	-83.10761	68.163056	8736
s_fov					0
s_region					0
s_resolution					0
t_min	56167.242	5204.3647	-1.0	58909.355	8736
t_max	56167.246	5204.3633	-1.0	58909.355	8736
t_exptime	178431.16	526231.25	-561.22	7677132.5	8736
t_resolution					0
em_min	1.8477201E-6	1.0256063E-6	5.0E-7	7.622082E-6	8736
em_max	2.6952448E-6	3.2755117E-6	5.0E-7	1.3767216E-5	8736
em_res_power	495.37125	3623.3855	6.9476286E-5	35743.39	8736
o_ucd					0
pol_states					0
facility_name			CHARA	VLTI	8734
instrument_name			6way_OB01	VEGA	8736
instrument_mode			High_K_1_2.172	Medium_K_1_2.3	152
quality_level	0.0	0.0	0	0	278
nb_channels	24.208677	65.90274	1	508	8736
nb_vis	0.21806319	0.7661262	0	10	8736
nb_vis2	7.485119	12.007391	0	300	8736
nb_t3	4.710165	7.7968287	0	200	8736
keywords			Astrophysics - Solar and Stellar Astr...	techniques: interferometric ; galaxie...	807
subdate			2015-06-22T22:27:21	2020-05-11T13:08:12	8736
id	422905.6	160707.86	38659	525364	8736
progid			082.D-0146	60.A-9054	79
datapi			ABSIL	ZUNIGA-FERNANDEZ, S	8715
access_md5					0

Table name: Statistics for obsportal

dataLocation: /home/mellag/tmp/search.votable
statRows: 319791

Guillaume Mella - 2020-05-26

Comments