Update the JSDC in 2020 (v2020 ?)

This page gathers notes on the overall process to obtain the updated JSDC catalog (v2020) based on Simbad SPTYPE + VJHK mags (tycho2+2MASS).

SearchCal changes (2020)

The JSDC_BRIGHT scenario retrieves GAIA information (I/345 (DR2), I/347 (dist)) and the SearchCal server was updated to give more details on its crossmatch algorithm.

Problem: as GAIA coordinates are the most accuracte, RA/DE pmRA/DE are overwritten with values from the corresponding GAIA target. However it leads to confusion for close binaries or ASCC duplicates (within 1 as), as several rows in the JSDC output will have the same coordinates (and same GAIA ID).

2020.09: filter out all these potential conflicting sources (duplicates) in the sclcatFilterCatalog.sh script.

Meta-data of New SearchCal columns:

Only main column are listed (xx.confidence and xx.origin are ommitted):

• Added GAIA columns:

GAIA(String) - GAIA DR2 Release Catalog name, click to call VizieR on this object

Bp(Double)/mag - GAIA: Integrated Bp mean magnitude (Vega)
e_Bp(Double)/mag - GAIA: Standard error of BP mean magnitude (Vega)

G(Double)/mag - GAIA: G-band mean magnitude (Vega)
e_G(Double)/mag - GAIA: Standard error of G-band mean magnitude (Vega)

Rp(Double)/mag - GAIA: Integrated Rp mean magnitude (Vega)
e_Rp(Double)/mag - GAIA: Standard error of Rp mean magnitude (Vega)

e_RadVel(Double)/km/s - Radial velocity error 

gaia_dist(Double)/pc - GAIA: Estimated distance
b_gaia_dist(Double)/pc - GAIA: Lower bound on the confidence interval of the estimated distance
B_gaia_dist(Double)/pc - GAIA: Upper bound on the confidence interval of the estimated distance

Teff(Double)/K - GAIA: Stellar effective temperature (estimate from Apsis-Priam)
b_Teff(Double)/K - GAIA: Uncertainty (lower) on Teff estimate from Apsis-Priam (16th percentile)
B_Teff(Double)/K - GAIA: Uncertainty (upper) on Teff estimate from Apsis-Priam (84th percentile)

Note: columns with 'b_' or 'B_' prefixes may cause problems in IDL scripts (Uppercase in MWRFITS)

• New XMATCH columns (XM_ prefix):

The new XMATCH_LOG column gives lots of information about the xmatch steps performed on main catalogs (ASCC, HIP, GAIA, 2MASS, WISE) for every target.

XMATCH_LOG(String) - xmatch log (internal JMMC)
XM_ASCC_n_mates(Integer) - Number of mates within 5 as in ASCC catalog
XM_ASCC_sep(Double)/as - Angular Separation of the first object in ASCC catalog
XM_ASCC_sep_2nd(Double)/as - Angular Separation between first and second objects in ASCC catalog

XM_HIP_n_mates(Integer) - Number of mates within 5 as in HIP1/2 catalogs
XM_HIP_sep(Double)/as - Angular Separation of the first object in HIP1/2 catalogs
XM_HIP_sep_2nd(Double)/as - Angular Separation between first and second objects in HIP1/2 catalogs

XM_GAIA_n_mates(Integer) - Number of mates within 5 as in GAIA catalog
XM_GAIA_sep(Double)/as - Angular Separation of the first object in GAIA catalog
XM_GAIA_sep_2nd(Double)/as - Angular Separation between first and second objects in GAIA catalog

XM_2MASS_n_mates(Integer) - Number of mates within 5 as in 2MASS catalog
XM_2MASS_sep(Double)/as - Angular Separation of the first object in 2MASS catalog
XM_2MASS_sep_2nd(Double)/as - Angular Separation between first and second objects in 2MASS catalog

XM_WISE_n_mates(Integer) - Number of mates within 5 as in WISE catalog
XM_WISE_sep(Double)/as - Angular Separation of the first object in WISE catalog
XM_WISE_sep_2nd(Double)/as - Angular Separation between first and second objects in WISE catalog

JMDC inputs

See the full procedure at /SCALIB/DEVELOPMENT/alx/config/JMDC/scriptHandleLastVersionOf_JMDC_dot_xls_ToProduceJMDC_dot_fits.sh

This script has the following dependencies: stilts + gdl (astro lib ie MRDFITS + gdl mpfit procedures). export GDL_PATH="/home/bourgesl/apps/astron/pro/:/home/bourgesl/dev/gdl/src/pro/:/home/bourgesl/dev/gdl/src/pro/CMprocedures/"

Inputs:

• JMDC: get latest JMDC catalog from http://jmdc.jmmc.fr/ by clicking on the CSV button (top)
• JSDC (input catalog) to compute LDD/UD using gdl script (all SearchCal columns needed)

Commands:

    ./scriptHandleLastVersionOf_JMDC_dot_xls_ToProduceJMDC_dot_fits.sh JMDC_full_20200730.csv

 gdl -e "make_jsdc_script_simple,\"JMDC_full_20200730_final_lddUpdated.fits\",verbose=1,nocatalog=1"

 gdl -e "make_jsdc_script_simple,\"JMDC_full_20200730_final_lddUpdated.fits\",\"catalog2020.fits\",verbose=1"

JSDC input catalog

See procedure at /vobs/config/prepare_candidates/proc.sh

This script generates the 2 input files in the vobs/config folder for the JSDC_BRIGHT (vobsascc_simbad_sptype.cfg) / JSDC_FAINT (vobsascc_simbad_no_sptype.cfg) input catalogs (candidates) used to run SearchCal scenarios and obtain the JSDC full databases (data + computations).

How to update the input catalogs ?

Run proc.sh

Overall description: - get both ASCC & SIMBAD full catalogs via cdsxmatch.u-strasbg.fr/QueryCat queries (vot) - crossmatch catalogs using radius = 2 as (all from 1) on J2000 coords (corrected by PM) + compute group size at 5 as in both catalogs (GROUP_SIZE) - produce input catalogs in SearchCal vobsCatalog format (ascii):

• RAJ2000/DEJ2000, pmRA, pmDE (ASCC)
• MAIN_ID, SP_TYPE, OTYPES (Simbad at 2as)
• GROUP_SIZE (max GroupSize in both catalogs at 5as)

- split the full ASCC in 2 lists (NULL_SP_TYPE / NULL_SP_TYPE) (2017-04-14)

• vobsascc_simbad_sptype.cfg: 494401 rows
• vobsascc_simbad_no_sptype.cfg: 2006912 rows

Comments:

• ASCC is a catalog compilation (TYCHO2 + others) that contains many duplicates with 2as (tycho2 components or other known binaries ...)
• Simbad coordinates (used by crossmatch) now use Gaia DR2 ones for 3M objects where SIMBAD retrieved the 'unique' GAIA object

NOTE: these files were produced at 2017-04-14, not updated in 2020 as a cds-healpix-java bug (in 2020 stilts/topcat releases) make the procedure to fail (simbad 11M objects):

processing @ date: mardi 15 septembre 2020, 09:59:32 (UTC+0000)
Params: Max Error(Number)/arcsec=5.0
Tuning: HEALPix k(Integer)=11
Binning rows for table 2.............................................
Error: Hash value 13421764095 must be in [0, 12884901888[

How to run SearchCal scenarios ?

* Server side (~bourgesl)

* Publish results at

* Run filter.sh

JSDC 2020 generation

-- Bourges Laurent - 2020-09-15