OiDbObsMapping < Jmmc/Software

Tags: view all tags
%TOC%

---+ Obsportal &lt;-&gt; L0 OiDB coupling

Since Obsportal is now online and makes a lot of obslogdata massage it now makes sense to load this source of information into OiDB

Here comes a rough mapping table to start the link on top of current search.votable request (2min query / 27MB of gzip xml)

| dataproduct_type | String | | | High level scientific classification of the data product | meta.id;class |
| ^ | %GREEN%OK%ENDCOLOR% constant (even if empty could be replaced by visibility) |||||
| calib_level | Integer | | | Amount of data processing that has been applied to the data (0,1,2,3) | meta.code;obs.calib |
| ^ | %GREEN%OK%ENDCOLOR% constant : 0 |||||
| target_name | String | | | Object of interest | meta.id;src |
| ^ | %GREEN%OK%ENDCOLOR% obsportal.target_name |||||
| obs_id | String | | | Unique identifier for an observation | meta.id |
| ^ | <p>%GREEN%OK%ENDCOLOR% obsportal.exp_header_id</p> <p>KO in OiDB -&gt; PIONIER's obs_id values should be placed in progid and replaced by proper value</p> |||||
| obs_collection | String | | | Name of the data collection (e.g. project name) this data belongs to | meta.id |
| ^ | <p>%GREEN%OK%ENDCOLOR% constant : jmmc_obsportal TBC</p> <p>In fact obsportal may complete multiples collection. The first one will be the eso-l0 one that was synced by CDS.</p> |||||
| obs_creator_name | String | | | Name of the creator of the data | meta.id |
| ^ | %GREEN%OK%ENDCOLOR% constant: jmmc-tech-group / Bourgès TBC |||||
| obs_release_date | String | | Iso8601-&gt;Time | Observation release date | time.release |
| ^ | obsportal.exp_date_release |||||
| obs_publisher_did | String | | | Dataset identifier given by the publisher | meta.ref.url;meta.curation |
| ^ | unused on OiDB |||||
| bib_reference | String | | | Service bibliographic reference | meta.bib.bibcode |
| ^ | NA |||||
| data_rights | String | | | Public/Secure/Proprietary | meta.code |
| ^ | %GREEN%OK%ENDCOLOR% constant : secure |||||
| access_url | String | | | URL used to obtain the data set. | meta.ref.url |
| ^ | %GREEN%OK%ENDCOLOR% concatenation : http://obs.jmmc.fr/detail/exposure/${obsportal.exp_id} |||||
| access_format | String | | | MIME type of the resource at access_url | meta.code.mime |
| ^ | <p>NA for L0 (no data) or html ?</p> <p>Remark: ObsCore1.1 mentions application/x-fits-uvfits that we can follow for oifits ? GD: Yes we should use /x-fits-oifits</p> |||||
| access_estsize | Long | kbyte | | Estimated size of data product | phys.size;meta.file |
| ^ | NA for L0 |||||
| s_ra | Double | deg | | Right ascension of (center of) observation, ICRS | pos.eq.ra;meta.main |
| ^ | %GREEN%OK%ENDCOLOR% obsportal.target_ra |||||
| s_dec | Double | deg | | Declination of (center of) observation, ICRS | pos.eq.dec;meta.main |
| ^ | %GREEN%OK%ENDCOLOR% obsportal.target_dec |||||
| s_fov | Float | deg | | Approximate spatial extent for the region covered by the observation | phys.angSize;instr.fov |
| ^ | NA |||||
| s_region | Float | | | Region covered by the observation, as a polygon | phys.angArea;obs |
| ^ | NA |||||
| s_resolution | Float | arcsec | | Best spatial resolution within the data set | pos.angResolution |
| ^ | NA |||||
| t_min | Float | d | MJD-&gt;Time | Lower bound of times represented in the data set, as MJD | time.start;obs.exposure |
| ^ | obsportal.exp_mjd_start |||||
| t_max | Float | d | MJD-&gt;Time | Upper bound of times represented in the data set, as MJD | time.end;obs.exposure |
| ^ | obsportal.exp_mjd_end |||||
| t_exptime | Float | s | | Total exposure time | time.duration;obs.exposure |
| ^ | obsportal.exp_mjd_end - obsportal.exp_mjd_start |||||
| t_resolution | Float | s | | Minimal significant time interval along the time axis | time.resolution |
| ^ | NA |||||
| em_min | Float | m | | Minimal wavelength represented within the data set | em.wl;stat.min |
| ^ | obsportal.instrument_em_min |||||
| em_max | Float | m | | Maximal wavelength represented within the data set | em.wl;stat.max |
| ^ | obsportal.instrument_em_max |||||
| em_res_power | Float | | | Spectral resolving power delta_lambda / lamda | spect.resolution |
| ^ | obsportal.instrument_em_res_power |||||
| o_ucd | String | | | UCD for the product's observable | meta.ucd |
| ^ | NA |||||
| pol_states | String | | | List of polarization states in the data set | meta.code;phys.polarization |
| ^ | ?? |||||
| facility_name | String | | | Name of the facility at which data was taken | meta.id;instr.tel |
| ^ | %GREEN%OK%ENDCOLOR% obsportal.interferometer_name |||||
| instrument_name | String | | | Name of the instrument that produced the data | meta.id;instr |
| ^ | OK obsportal.instrument_name |||||
| instrument_mode | String | | | Instrument mode | |
| ^ | OK obsportal.instrument_mode . add obsportal.instrument_submode ? |||||
| quality_level | Integer | | | Quality data level estimated by data provider | meta.code.qual |
| ^ | computation from obsportal.exp_tau0 obsportal.exp_seeing ? |||||
| nb_channels | Integer | | | number of spectral channels | |
| ^ | inferred from AsproConfig using obsportal.instrument_name/obsportal.instrument_mode/obsportal.instrument_submode ? |||||
| nb_vis | Integer | | | number of OI_VIS data (complex visiblity) | |
| ^ | NA |||||
| nb_vis2 | Integer | | | number of OI_VIS2 data (square visiblity) | |
| ^ | NA |||||
| nb_t3 | Integer | | | number of OI_T3 data (closure phase) | |
| ^ | NA |||||
| keywords | String | | | | |
| ^ | ?? |||||
| subdate | String | | Iso8601-&gt;Time | submission date | |
| ^ | current datetime |||||
| id | Long | internal ID (granule ID) | | Object ID | |
| ^ | AUTO |||||
| progid | String | | | program or proposal identifier | |
| ^ | <p>OK obsportal.obs_program</p> <p>TBC : obs_id goes from 1 to 9999 in obsportal for the whole 319791 records</p> |||||
| datapi | String | | | Data PI | |
| ^ | NA for L0 TBC |||||
| access_md5 | String | | | md5sum of the resource at access_url | |
| ^ | NA |||||
Some records should be filtered during the injest :
   * keep science only -&gt; obsportal.obs_type='science' ?
   * ???

Add interferometer_stations as oidb metadata.

Obsportal votable could also propose to export mode a full one that carry all and a more lite one without verbose validation informations.

---+ Obsportal &lt;-&gt; L2 OiDB coupling

L2 injest of PIONIER collection requires to complete some metadata not present in the OIFITS keywords.

At present time PIs are retrieved from the archive jmmc-eso:get-pi-from-progid($progid)

The change operated using obsportal as intermediate source of information could be indirect. The missing L2 information could be retrieved on top of already present L0 data using proper key ( obsportal.exp_header_id == OIFITS.ARCFILE ). However OiDB decided not to store datapis of L0 then DataPis could not be retrieved for L2 on top of L0 records :(

How to fix that ?

Add pi-coi in obsportal votable.

Do not store in the OiDB L0 records but in xml database : prog-id - pi-cois

---+ Table metadata samples :
---++ Table name: Statistics for ESO_TAP dbo.raw
<verbatim>
dataLocation: TAP_5_dbo.raw
statRows: 361</verbatim>
| Name  | Mean  | SD  | Minimum  | Maximum  | nGood |
| access_url  | | | https://dataportal.eso.org/dataportal... | https://dataportal.eso.org/dataportal... | 361  |
| datalink_url  | | | http://archive.eso.org/datalink/links... | http://archive.eso.org/datalink/links... | 361  |
| date_obs  | | | 2004-12-25T21:37:00.9989  | 2005-02-26T10:16:18.1015  | 361  |
| dec  | -301.0227  | 287.67047  | -596.52323555  | 15.87714  | 361  |
| dec_pnt  | -22.998003  | 34.063705  | -70.19476  | 15.87714  | 186  |
| det_chip1id  | | | | | 0 |
| det_chop_ncycles  | | | | | 0 |
| det_dit  | 0.45429364  | 0.8346642  | 0 | 2  | 361  |
| det_expid  | | | | | 0 |
| det_ndit  | 147.39058  | 284.34305  | 1  | 1000  | 361  |
| dp_cat  | | | CALIB  | SCIENCE  | 361  |
| dp_id  | | | AMBER.2004-12-25T21:37:00.999  | AMBER.2005-02-26T10:16:18.102  | 361  |
| dp_tech  | | | INTERFEROMETRY  | INTERFEROMETRY  | 361  |
| dp_type  | | | 3P2V  | WAVE,3TEL  | 361  |
| ecl_lat  | -30.475166  | 26.049498  | -66.309153  | 0.654471  | 186  |
| ecl_lon  | 192.56058  | 132.98537  | 0.0  | 348.802413  | 186  |
| exp_start  | | | 2004-12-25T21:37:00Z  | 2005-02-26T10:16:18Z  | 361  |
| exposure  | 8.481276  | 16.470871  | 0.058  | 82.0  | 356  |
| filter_path  | | | | | 0 |
| gal_lat  | -15.380623  | 15.691006  | -60.188552  | 1.646137  | 186  |
| gal_lon  | 211.57108  | 80.30247  | 21.535596  | 296.367403  | 186  |
| grat_path  | | | GHR  | PRISM  | 361  |
| gris_path  | | | NAR_SLT,3T_JHK  | NAR_SLT,3T_K  | 361  |
| ins_mode  | | | 3Tstd_High_K_1_2.166  | 3Tstd_Medium_K_1_2.1  | 361  |
| instrument  | | | AMBER  | AMBER  | 361  |
| last_mod_date  | | | 2010-11-30T09:20:26Z  | 2010-11-30T09:20:26Z  | 361  |
| mjd_obs  | 53396.875  | 30.500704  | 53364.902  | 53427.43  | 361  |
| ob_id  | 5.5059866E8 | 4.99765952E8 | 200146604  | 1268183793  | 361  |
| ob_name  | | | HR_K_ETA_CAR  | MR_K_HD100546_NATTA_1  | 361  |
| object  | | | 3P2V  | WAVE,3TEL  | 343  |
| obs_mode  | | | s  | s  | 361  |
| origfile  | | | AMBER_3TSTD_ACQ055_0015.fits  | AMBER_P2VM_3T_2005-02-26T10:16:28.fits  | 361  |
| period  | | | | | 0 |
| pi_coi  | | | AMBER SCIENCE DEMONS  | AMBER SCIENCE DEMONS  | 361  |
| prog_id  | | | 60.A-9055(J)  | 60.A-9055(J)  | 361  |
| prog_title  | | | AMBER SCIENCE DEMONSTRATION  | AMBER SCIENCE DEMONSTRATION  | 361  |
| prog_type  | 0.0  | 0.0  | 0 | 0 | 361  |
| ra  | -226.02208  | 363.05057  | -596.52323555  | 299.53385388  | 361  |
| ra_pnt  | 122.567726  | 71.754524  | 0.0  | 299.533854  | 186  |
| release_date  | | | 2004-12-25T21:37:00Z  | 2006-02-26T09:31:53Z  | 361  |
| slit_path  | | | | | 0 |
| target  | | | ETA-CAR  | SAO237916  | 361  |
| tel_airm_end  | | | | | 0 |
| tel_airm_start  | | | | | 0 |
| tel_alt  | | | | | 0 |
| tel_ambi_fwhm_end  | | | | | 0 |
| tel_ambi_fwhm_start | | | | | 0 |
| tel_ambi_pres_end  | | | | | 0 |
| tel_ambi_pres_start | | | | | 0 |
| tel_ambi_rhum  | | | | | 0 |
| tel_az  | | | | | 0 |
| telescope  | | | ESO-VLTI-U234  | NOT_SPECIFIED  | 343  |
| tpl_expno  | 7.218659  | 4.244221  | 1  | 15  | 343  |
| tpl_id  | | | AMBER_3T_cal_frnsrc  | AMBER_3Tstd_obs_1row  | 343  |
| tpl_name  | | | AMBER 3 telescopes / 1 row standard o... | AMBER 3T standard acquisition  | 343  |
| tpl_nexp  | 13.233236  | 3.6161854  | 1  | 15  | 343  |
| tpl_seqno  | 1.8227147  | 1.7381604  | 1  | 7  | 361  |
| tpl_start  | | | 2004-12-25T21:29:26  | 2005-02-26T10:09:59  | 343  |

---++ Table name: Statistics for ESO TAP ivoa.ObsCore
<verbatim>
dataLocation: TAP_7_ivoa.ObsCore
statRows: 1000</verbatim>

| Name  | Mean  | SD  | Minimum  | Maximum  | nGood |
| abmaglim  | | | | | 0 |
| access_estsize  | 85.64  | 2.0426455  | 80  | 92  | 1000  |
| access_format  | | | application/x-votable+xml;content=dat... | application/x-votable+xml;content=dat... | 1000  |
| access_url  | | | http://archive.eso.org/datalink/links... | http://archive.eso.org/datalink/links... | 1000  |
| bib_reference  | | | | | 0 |
| calib_level  | 2.0  | 0.0  | 2  | 2  | 1000  |
| dataproduct_subtype | | | | | 0 |
| dataproduct_type  | | | visibility  | visibility  | 1000  |
| dp_id  | | | ADP.2017-04-04T14:22:18.367  | ADP.2018-10-22T14:40:47.123  | 1000  |
| em_max  | 1.7599808E-6 | 1.8412344E-8 | 1.654116E-6  | 1.774278E-6  | 1000  |
| em_min  | 1.5240973E-6 | 2.4750758E-8 | 1.511285E-6  | 1.6799380000000002E-6  | 1000  |
| em_res_power  | 33.349228  | 4.976259  | 5.5137  | 34.6358  | 1000  |
| em_xel  | 1.0  | 0.0  | 1  | 1  | 1000  |
| facility_name  | | | ESO-VLTI-A1234  | ESO-VLTI-U1234  | 1000  |
| filter  | | | | | 0 |
| gal_lat  | -15.536459  | 27.223131  | -82.701899  | 72.176321  | 1000  |
| gal_lon  | 223.69342  | 116.30873  | 0.80439  | 359.933529  | 1000  |
| instrument_name  | | | PIONIER  | PIONIER  | 1000  |
| multi_ob  | | | S  | S  | 1000  |
| n_obs  | 1.0  | 0.0  | 1  | 1  | 1000  |
| o_calib_status  | | | any  | any  | 1000  |
| o_ucd  | | | | | 0 |
| obs_collection  | | | PIONIER  | PIONIER  | 1000  |
| obs_creator_did  | | | ivo://eso.org/origfile?PIONI.2015-10-... | ivo://eso.org/origfile?PIONI.2018-06-... | 1000  |
| obs_creator_name  | | | ANDERSON, RICHARD  | WITTKOWSKI, MARKUS  | 1000  |
| obs_id  | | | 1236337  | 2027314  | 1000  |
| obs_publisher_did  | | | ivo://eso.org/ID?ADP.2017-04-04T14:22... | ivo://eso.org/ID?ADP.2018-10-22T14:40... | 1000  |
| obs_release_date  | | | 2015-11-29T02:31:44Z  | 2019-06-05T07:25:39Z  | 1000  |
| obs_title  | | | | | 0 |
| obstech  | | | INTERFEROMETRY  | INTERFEROMETRY  | 1000  |
| p3orig  | | | EDP  | IDP  | 1000  |
| pol_states  | | | | | 0 |
| pol_xel  | | | | | 0 |
| proposal_id  | | | 0100.C-0278(J)  | 597.D-0727(A)  | 1000  |
| publication_date  | | | 2017-04-04T14:53:52Z  | 2018-10-22T17:08:02Z  | 1000  |
| s_dec  | -37.595093  | 26.249413  | -76.61186  | 27.46021  | 1000  |
| s_fov  | | | | | 0 |
| s_pixel_scale  | | | | | 0 |
| s_ra  | 189.17336  | 95.98031  | 0.455556  | 359.97951  | 1000  |
| s_region  | | | POSITION J2000 0.4555560000000014 -3.... | POSITION J2000 97.967765 5.0331  | 1000  |
| s_resolution  | | | | | 0 |
| s_xel1  | | | | | 0 |
| s_xel2  | | | | | 0 |
| snr  | | | | | 0 |
| t_exptime  | 234.39381  | 145.89099  | 51.2  | 1177.6  | 1000  |
| t_max  | 57772.8  | 259.5562  | 57298.09441799  | 58274.30534132  | 1000  |
| t_min  | 57772.797  | 259.55573  | 57298.09293651  | 58274.30385984  | 1000  |
| t_resolution  | 241.3017  | 158.16522  | 51.199776  | 1338.315264  | 1000  |
| t_xel  | | | | | 0 |
| target_name  | | | 2MASS_J01373545-0645375  | thetaSco  | 1000  |

---++ Table name: Statistics for oidb
<verbatim>
dataLocation: OiDB:votable.xql?perpage=25&order=t_min&caliblevel=1%2C2%2C3
statRows: 8736</verbatim>

+-------------------+--------------+--------------+------------------------------------------+------------------------------------------+-------+
| Name  | Mean  | SD  | Minimum  | Maximum  | nGood |

+-------------------+--------------+--------------+------------------------------------------+------------------------------------------+-------+
| dataproduct_type  | | | | | 0 |
| calib_level  | 2.1394231  | 0.34638748  | 2  | 3  | 8736  |
| target_name  | | | 10_TAU  | zeta_Gem  | 8736  |
| obs_id  | | | 0100.D-0024(A)  | Maintenance  | 6955  |
| obs_collection  | | | 2727e73d-35d5-43e6-8a36-c9e1da31d294  | iota  | 8736  |
| obs_creator_name  | | | Bernard Lazareff  | Nicolas Nardetto  | 8736  |
| obs_release_date  | | | 1999-09-29T10:52:30  | 2021-03-01T08:29:42  | 8736  |
| obs_publisher_did | | | | | 0 |
| bib_reference  | | | 2009A&A...493L..17C  | 2019ApJ...883...89O  | 807  |
| data_rights  | | | public  | secure  | 8736  |
| access_url  | | | /exist/apps/oidb-data/oifits/staging/... | http://cdsarc.u-strasbg.fr/vizier/ftp... | 8736  |
| access_format  | | | application/fits  | application/fits  | 7240  |
| access_estsize  | 96.36161  | 73.9986  | 13  | 676  | 8736  |
| s_ra  | 177.86417  | 94.429085  | 0.0  | 359.97951  | 8736  |
| s_dec  | -31.354578  | 29.259974  | -83.10761  | 68.163056  | 8736  |
| s_fov  | | | | | 0 |
| s_region  | | | | | 0 |
| s_resolution  | | | | | 0 |
| t_min  | 56167.242  | 5204.3647  | -1.0  | 58909.355  | 8736  |
| t_max  | 56167.246  | 5204.3633  | -1.0  | 58909.355  | 8736  |
| t_exptime  | 178431.16  | 526231.25  | -561.22  | 7677132.5  | 8736  |
| t_resolution  | | | | | 0 |
| em_min  | 1.8477201E-6 | 1.0256063E-6 | 5.0E-7  | 7.622082E-6  | 8736  |
| em_max  | 2.6952448E-6 | 3.2755117E-6 | 5.0E-7  | 1.3767216E-5  | 8736  |
| em_res_power  | 495.37125  | 3623.3855  | 6.9476286E-5  | 35743.39  | 8736  |
| o_ucd  | | | | | 0 |
| pol_states  | | | | | 0 |
| facility_name  | | | CHARA  | VLTI  | 8734  |
| instrument_name  | | | 6way_OB01  | VEGA  | 8736  |
| instrument_mode  | | | High_K_1_2.172  | Medium_K_1_2.3  | 152  |
| quality_level  | 0.0  | 0.0  | 0 | 0 | 278  |
| nb_channels  | 24.208677  | 65.90274  | 1  | 508  | 8736  |
| nb_vis  | 0.21806319  | 0.7661262  | 0 | 10  | 8736  |
| nb_vis2  | 7.485119  | 12.007391  | 0 | 300  | 8736  |
| nb_t3  | 4.710165  | 7.7968287  | 0 | 200  | 8736  |
| keywords  | | | Astrophysics - Solar and Stellar Astr... | techniques: interferometric ; galaxie... | 807  |
| subdate  | | | 2015-06-22T22:27:21  | 2020-05-11T13:08:12  | 8736  |
| id  | 422905.6  | 160707.86  | 38659  | 525364  | 8736  |
| progid  | | | 082.D-0146  | 60.A-9054  | 79  |
| datapi  | | | ABSIL  | ZUNIGA-FERNANDEZ, S  | 8715  |
| access_md5  | | | | | 0 |

---++ Table name: Statistics for obsportal
<verbatim>
dataLocation: /home/mellag/tmp/search.votable
statRows: 319791</verbatim>

| Name  | Mean  | SD  | Minimum  | Maximum  | nGood  |
| exp_id  | | | AMBER.2004-05-31T06:02:46.487_1  | PIONI.2020-03-24T08:44:48.208_1  | 319791 |
| exp_header_id  | | | AMBER.2004-05-31T06:02:46.487  | PIONI.2020-03-24T08:44:48.208  | 319791 |
| exp_mjd_start  | 56545.707  | 1545.9994  | 52804.30111111  | 58932.3644468505  | 319791 |
| exp_mjd_end  | 56545.71  | 1545.9995  | 52804.3016722211  | 58932.3650394431  | 319791 |
| exp_projected_baselines | | | {'A0-B2': {'length': {'start': '0.000... | {'U4-U3': {'length': {'start': '58.05... | 319791 |
| exp_validation_level  | 1.0  | 0.0  | 1  | 1  | 319791 |
| exp_validation_log  | | | ["Invalid value for 'HIERARCH ESO ISS... | ["No value for 'HIERARCH ESO OBS TARG... | 9761  |
| exp_tau0  | 0.0042860596 | 0.008200745 | 2.925E-4  | 1.689503  | 319588 |
| exp_temperature  | 12.737826  | 2.619762  | 0.34  | 20.66  | 319791 |
| exp_seeing  | 0.87094575  | 0.53003204  | -1.0  | 8.49  | 319741 |
| exp_date_updated  | | | 2010-11-30 10:20:26+01:00  | 2020-03-27 09:26:53+01:00  | 319791 |
| exp_date_release  | | | 2004-01-15 17:07:49+01:00  | 2021-03-24 09:45:18+01:00  | 319791 |
| obs_id  | 23784.17  | 14976.115  | 1  | 45773  | 319791 |
| obs_type  | | | calibrator  | science  | 319791 |
| obs_program  | | | 0100.B-0582(A)  | Maintenance  | 319791 |
| obs_program_title  | | | $DELTA$ SCO : FOLLOW-UP OF THE CIRCUM... | ``THE ANSWER MY FRIEND... --- TESTING... | 319717 |
| obs_program_pi_coi  | | | A JORISSEN/ BERGER/ HILLEN/ BOFFIN/ B... | ZUNIGA-FERNANDEZ/ OLOFSSON/ BAYO/ COR... | 319717 |
| interferometer_name  | | | VLTI  | VLTI  | 319791 |
| interferometer_stations | | | A0 B2 C1 E0  | U4 U3  | 319791 |
| instrument_name  | | | AMBER  | PIONIER  | 319791 |
| instrument_mode  | | | FREE-H  | SMALL-H  | 319791 |
| instrument_submode  | | | DUAL  | SINGLE  | 49491  |
| instrument_em_min  | 3.381424  | 2.7538488  | 1.469  | 8.0  | 319791 |
| instrument_em_max  | 5.1242223  | 4.724413  | 1.7959  | 13.03818  | 319791 |
| instrument_em_res_power | 1335.7579  | 3184.8386  | 6.0  | 12000.0  | 319791 |
| target_id  | 18128.576  | 10688.66  | 1  | 34241  | 319791 |
| target_name  | | | 05_07_42_000_24_47_5  | zpup  | 319356 |
| target_ra  | 180.03163  | 93.55215  | 0.026005  | 359.97951  | 319791 |
| target_dec  | -31.22812  | 25.1721  | -87.48218  | 43.94645  | 319791 |
-- %USERSIG{GuillaumeMella - 2020-05-26}%
---++ Comments

%COMMENT%
Topic revision: r10 - 2020-06-09 - GuillaumeMella