JMMCSearchCalCDSCatalog2020 < Jmmc/Software

---+ Update the JSDC in 2020 (v2020 ?)

This page gathers notes on the overall process to obtain the updated JSDC catalog (v2020) based on Simbad SPTYPE + VJHK mags (tycho2+2MASS).


---++ SearchCal changes (2020)

The JSDC_BRIGHT scenario retrieves GAIA information (I/345 (DR2), I/347 (dist)) and the SearchCal server was updated to give more details on its crossmatch algorithm.

Problem: as GAIA coordinates are the most accuracte, RA/DE pmRA/DE are overwritten with values from the corresponding GAIA target. However it leads to confusion for close binaries or ASCC duplicates (within 1 as), as several rows in the JSDC output will have the same coordinates (and same GAIA ID).

2020.09: filter out all these potential conflicting sources (duplicates) in the sclcatFilterCatalog.sh script.


---+++ Meta-data of New SearchCal columns:
Only main column are listed (xx.confidence and xx.origin are ommitted):

   * Added GAIA columns:
<verbatim>
GAIA(String) - GAIA DR2 Release Catalog name, click to call VizieR on this object

Bp(Double)/mag - GAIA: Integrated Bp mean magnitude (Vega)
e_Bp(Double)/mag - GAIA: Standard error of BP mean magnitude (Vega)

G(Double)/mag - GAIA: G-band mean magnitude (Vega)
e_G(Double)/mag - GAIA: Standard error of G-band mean magnitude (Vega)

Rp(Double)/mag - GAIA: Integrated Rp mean magnitude (Vega)
e_Rp(Double)/mag - GAIA: Standard error of Rp mean magnitude (Vega)

e_RadVel(Double)/km/s - Radial velocity error 

gaia_dist(Double)/pc - GAIA: Estimated distance
b_gaia_dist(Double)/pc - GAIA: Lower bound on the confidence interval of the estimated distance
B_gaia_dist(Double)/pc - GAIA: Upper bound on the confidence interval of the estimated distance

Teff(Double)/K - GAIA: Stellar effective temperature (estimate from Apsis-Priam)
b_Teff(Double)/K - GAIA: Uncertainty (lower) on Teff estimate from Apsis-Priam (16th percentile)
B_Teff(Double)/K - GAIA: Uncertainty (upper) on Teff estimate from Apsis-Priam (84th percentile)

</verbatim>

Note: columns with 'b_' or 'B_' prefixes may cause problems in IDL scripts (Uppercase in MWRFITS)


   * New XMATCH columns (XM_ prefix):
The new XMATCH_LOG column gives lots of information about the xmatch steps performed on main catalogs (ASCC, HIP, GAIA, 2MASS, WISE) for every target.

<verbatim>
XMATCH_LOG(String) - xmatch log (internal JMMC)
XM_ASCC_n_mates(Integer) - Number of mates within 5 as in ASCC catalog
XM_ASCC_sep(Double)/as - Angular Separation of the first object in ASCC catalog
XM_ASCC_sep_2nd(Double)/as - Angular Separation between first and second objects in ASCC catalog

XM_HIP_n_mates(Integer) - Number of mates within 5 as in HIP1/2 catalogs
XM_HIP_sep(Double)/as - Angular Separation of the first object in HIP1/2 catalogs
XM_HIP_sep_2nd(Double)/as - Angular Separation between first and second objects in HIP1/2 catalogs

XM_GAIA_n_mates(Integer) - Number of mates within 5 as in GAIA catalog
XM_GAIA_sep(Double)/as - Angular Separation of the first object in GAIA catalog
XM_GAIA_sep_2nd(Double)/as - Angular Separation between first and second objects in GAIA catalog

XM_2MASS_n_mates(Integer) - Number of mates within 5 as in 2MASS catalog
XM_2MASS_sep(Double)/as - Angular Separation of the first object in 2MASS catalog
XM_2MASS_sep_2nd(Double)/as - Angular Separation between first and second objects in 2MASS catalog

XM_WISE_n_mates(Integer) - Number of mates within 5 as in WISE catalog
XM_WISE_sep(Double)/as - Angular Separation of the first object in WISE catalog
XM_WISE_sep_2nd(Double)/as - Angular Separation between first and second objects in WISE catalog
</verbatim>


---++ JMDC inputs
See the full procedure at /SCALIB/DEVELOPMENT/alx/config/JMDC/scriptHandleLastVersionOf_JMDC_dot_xls_ToProduceJMDC_dot_fits.sh

This script has the following dependencies: stilts + gdl (astro lib ie MRDFITS + gdl mpfit procedures).
    export GDL_PATH="/home/bourgesl/apps/astron/pro/:/home/bourgesl/dev/gdl/src/pro/:/home/bourgesl/dev/gdl/src/pro/CMprocedures/"

Inputs:
   * JMDC: get latest JMDC catalog from http://jmdc.jmmc.fr/ by clicking on the CSV button (top)
   * JSDC (input catalog) to compute LDD/UD using gdl script (all SearchCal columns needed)

Commands:
<verbatim>
    ./scriptHandleLastVersionOf_JMDC_dot_xls_ToProduceJMDC_dot_fits.sh JMDC_full_20200730.csv

 gdl -e "make_jsdc_script_simple,\"JMDC_full_20200730_final_lddUpdated.fits\",verbose=1,nocatalog=1"

 gdl -e "make_jsdc_script_simple,\"JMDC_full_20200730_final_lddUpdated.fits\",\"catalog2020.fits\",verbose=1"
</verbatim>

---++ JSDC input catalog
See procedure at /vobs/config/prepare_candidates/proc.sh

This script generates the 2 input files in the vobs/config folder for the JSDC_BRIGHT (vobsascc_simbad_sptype.cfg) / JSDC_FAINT (vobsascc_simbad_no_sptype.cfg) input catalogs (candidates) used to run SearchCal scenarios and obtain the JSDC full databases (data + computations).

---+++ How to update the input catalogs ?

Run proc.sh 

Overall description:
- get both ASCC & SIMBAD full catalogs via cdsxmatch.u-strasbg.fr/QueryCat queries (vot)
- crossmatch catalogs using radius = 2 as (all from 1) on J2000 coords (corrected by PM) + compute group size at 5 as in both catalogs (GROUP_SIZE)
- produce input catalogs in SearchCal vobsCatalog format (ascii):
   * RAJ2000/DEJ2000, pmRA, pmDE (ASCC)
   * MAIN_ID, SP_TYPE, OTYPES (Simbad at 2as)
   * GROUP_SIZE (max GroupSize in both catalogs at 5as)
- split the full ASCC in 2 lists (!NULL_SP_TYPE / NULL_SP_TYPE) (2017-04-14)
   * vobsascc_simbad_sptype.cfg: 494401 rows 
   * vobsascc_simbad_no_sptype.cfg: 2006912 rows

Comments:
   * ASCC is a catalog compilation (TYCHO2 + others) that contains many duplicates with 2as (tycho2 components or other known binaries ...)
   * Simbad coordinates (used by crossmatch) now use Gaia DR2 ones for 3M objects where SIMBAD retrieved the 'unique' GAIA object

NOTE: these files were produced at 2017-04-14, not updated in 2020 as a cds-healpix-java bug (in 2020 stilts/topcat releases) make the procedure to fail (simbad 11M objects):
<verbatim>
processing @ date: mardi 15 septembre 2020, 09:59:32 (UTC+0000)
Params: Max Error(Number)/arcsec=5.0
Tuning: HEALPix k(Integer)=11
Binning rows for table 2.............................................
Error: Hash value 13421764095 must be in [0, 12884901888[
</verbatim>



---+++ How to run SearchCal scenarios ?

* Server side (~bourgesl)

* Publish results at

* Run filter.sh


---++ JSDC 2020 generation




---++ 


-- %USERSIG{LaurentBourges - 2020-09-15}%
This topic: Jmmc/Software > WebHome > SearchCal > JMMCSearchCalCDSCatalog > JMMCSearchCalCDSCatalog2020
Topic revision: r3 - 2020-09-15 - LaurentBourges