Gentlemen and Ladies
Below a description of the EWS (Expert Work Station: pronounced "Ooze") software for coherent estimation of MIDI visibilities. The package consists of C-routines that do most of the work and IDL front-ends forselecting data and viewing results.
Walter
07-Sep-2012
I. Summary of C-Language programs II. Summary of IDL routines calling C routines (probably what you want to know!) III. Summary of IDL Calibration Database Routines IV. Summary of other IDL utilities V. More details about (I) VI. More details about (II) VII. More details about (III) VIII. More details about (IV) IX. Description of file selection GUI (Gorgonzola) X. Standard file names and contents used by scripts
1. oir1dCompressData "inFiles" maskFile outFile 2. oirFormFringes infile outfile [-smooth nSmooth] [-averageOrder order] 3. oirRotateInsOpd infile outfile [waveRefFile] 4. oirGroupDelay infile outfile [-smooth gSmooth] [-weight weightFile] [-first firstFile ]
[-maxopd maxOpd] [-search] 5. oirFGroupDelay infile outfile [-smooth gSmooth] [-medsmooth medsmooth] [-maxopd maxOpd]
[-first firstFile] [-ngrad nGrad] [-search]
[-sigout sigOut] [-exinterval exInterval] 6. oirPowerDelay infile outfile [-smooth gSmooth] [-ampsmooth ampSmooth]
[-delaysmooth delaySmooth ] [-medsmooth medSmooth] 7. oirRotateGroupDelay infile delayFile outfile [-smooth smooth] [-ngrad nGrad] [-frames]
[-phase phaseFile] [-nophase] [-nogroup] [-noopd] 8. oirAutoFlag infile delayFile flagFile [-maxopd maxOpd] [-minopd minOpd]
[-jump jumpOpd] [-jitteropd jitterOpd] [-SN SN] 9. oirAverageVis infile flagFile outFile 10. oirChopPhotoImages -in "inFiles" -out outFile [-nDrop nDrop] [-dSky dSky] [-order order]
[-ref curveFile] [-skymask skymaskfile] [-sky0] 11. oirMakePhotoSpectra -A Afile -B Bfile [-AB ABfile] [-mask maskFile] -out outFile [-shift maskShift]
[-outmask outMaskFile] [-autoshift] 12. oirRedCal sourceTag 13. oirCalibrateVis targTag calTag [-calspec calSpecFile] [-calflux calFlux10] [-cald calDiameter]
[-nophot] 14. oirSci2HiPhot -AB ABfile -outbase outFileBase -ref refCoordFile -cross CrossCoeffFile 15 oirRescaleSpectra templateFile scaleFile outFile 16. oirCrossCoeff baseName -cross outputFile -mask maskFile [-autoshift] [-sky] [-dSky dSkyValue ] 17. oirMeanRMS infile outfile [-float] 18. oirCurveCal -A AFile -B BFile -out outFile [-fudge fudge]
These are the "routine" routines that most people will use for standarized analysis of MIDI data.
1. midiPipe, tag, filelist [,mask=maskfile][,smooth=smooth][,gsmooth=gsmooth][,ave=ave][,/twopass]
[,mindopd=minopd][,maxopd=maxopd][/twopass][,msmooth=msmooth]
[,ngrad=ngrad][,skymask=skymask][,dSky=dSky]
2. midiVisPipe, tag, filelist [,mask=maskfile][,smooth=smooth][,gsmooth=gsmooth][,ave=ave]
[,minopd=minopd][,maxopd=maxopd][,msmooth=msmooth][,ngrad=ngrad][,noc=noc]
3. midiPhotoPipe, tag, filelist [,mask=maskfile][,skyMask=skyMask][,dSky=dSky]
4. midiSearch, tag, fileList [,mask=mask][,smooth=smooth][,gsmooth=gsmooth][,ave=ave]
[,minopd=minopd][,maxopd=maxopd][,msmooth=msmooth][,ngrad=ngrad]
5. midiCalibrate, sourceTag, calTag, [,calDatabase=calDatabase][,calAscii=calAscii][,calFlux10=calFlux10]
[,diam=diam][,/noPhot][,/print]
6. midiSPipe, tag, files, cross=cross [,photoFile=photoFile][,minopd=minopd][,maxopd=maxopd][,curve=curve]
[,mask=maskfile][,smooth=smooth][,gsmooth=gsmooth][,dSky=dSky][,/dAve]
7. midiMakeMask, tag, filelist [,initialMask=initialMask][,shiftOnly=shiftOnly][,ave=ave]
[,smooth=smooth][,gsmooth=gsmooth][,factor=factor][,/noDelete
8. midiCrossCoeff, tag, fileList [,curveFile=curveFile][,sky=sky]
9. midiProcess2Vis, targetFiles, calFiles, targTag, calTag, [db=db],[mask=mask],[parmfile=parmfile]
10. midiFringeImage, tag, file, [/nodelete],[smooth=smooth],[noAve=noAve]
11. simData=midiDecorrelate(targetdata, caldata, targtag, caltag, ratios [,parmfile=parmfile][,mask=mask] ...)
1. cdb = midiCalDatabase(fitsfile) 2. cdb = vboekelbase() 3. nSource = cdb->sourceByName(name, sourceData, diamData, photData, specData) 4. nSource = cdb->sourceByAlias(alias, sourceData, diamData, photData, specData) 5. radii = cdb->sourcesNearPosition(ra, dec, sources, nmax=nmax, rmax=rmax) 6. sourceData = cdb->sourceFromFile(fitsFile, diamData=diamData, photData=photData, specData=specData, rad=rad, /silent) 7. sourceFlux = cdb->spectrumFromFile(fitsFile, diamData=diamData, outwave, rad=rad, /silent)
1. midiDelayPlot, tag, title, twopass=twopass 2. data = oirGetData(file[,rows][,col=col]) 3. vis = oirGetVis(file[,wave=wave]) 4. corr = midiGetCorr(tag[,wave=wave]) 5. phase = midiGetPhase(tag[,wave=wave]) 6. phot = midiGetPhot(tag) 7. opd = midiGetOpd(tag) 8. delay = midiGetDelay(tag) 9. choppedData = midiChopImage(inputFiles, before=before, after=after) 10. keyValue = midiGetKeyword(keyword, inputFiles, comment=comment, extna=extna) 11. complexVisData = midiCVis(oirVisibilityStruct) 12. p = midiPhase(cArray) 13. c = pseudoComplex(pcArray) 14. c = midiGetComplex(tag, qualifier)
Purpose:
Compress MIDI raw (PRISM/GRISM) files by multiplying each MIDI window
by a floating point mask (which has zeros in rows not containing useful
data) and summing in the y-direction (perpendicular to the PRISM/GRISM
dispersion). The result is a 1-dimensional spectrum for each window
region.
Parameters:
"inFiles": a list of raw MIDI (PRISM/GRISM) input files representing
one exposure. There may be more than one file because they were
broken up into 100MB files by the on-line system. If there is more
than one file they should be separated by spaces and included
between double quotes: "file1 file2 file3"
maskFile: a MIDI style FITS binary table whose imaging_data section
contains the masks described above. If this is not specified it
defaults to an environmental variable specified externally,
typically in the vltisetup script:
for PRISM/HIGH_SENS $prismhmask
for PRISM/SCI_PHOT $prismsmask
for GRISM/HIGH_SENS $grismhmask
for GRISM/SCI_PHOT $grismsmask
outFile: the name of the output file. This will be a MIDI style FITS
binary table in exactly the same format as "inFiles" and maskFile
except that the y-dimension of each DATA array is 1.
Purpose:
Take the output of oir1dCompressData and form spectrally dispersed
fringes by subtraction of the two interferometic channels and
supression of the background. The latter is accomplished by the
subtraction itself, by a high-pass filter (in the time domain) applied
to the individual spectral channels, and finally, optionally, by
removal of a polynomial fit to the flux (over wavelength) from the
individual spectral channels. This removes instrumental and sky
backgrounds that vary more slowly than the fringe. The fringe should
vary quickly because of the OPD modulation imposed by the MIDI piezos.
Parameters:
inFile: name of the output from oir1dCompressData
outFile: output file name. The format is a MIDI style FITS table
imaging_data file with only one DATA region (the combined I1-I2
channels).
nSmooth: (optional; default = 50) width of boxcar used in highpass
filtering. Data is smoothed in time with boxcar, and this smoothed
version is subtracted from input data, yielding a highpass filter.
order: If specified, a polynomial fit of the specified order is made to
the pixels in each frame between 7 and 13.8 microns, and this fit
is subtracted from this individual pixels. The sky background often
generates a slowly varying offset of the spectrum, which is not
entirely removed by the preceding filtering actions. This offset
generates an artefact at zero OPD that can confuse oirGroupDelay or
oirFGroupDelay if the source flux is very weak. Specifying
-averageOrder ≥0 removes this artefact, but it also removes your
source if it crosses zero during a OPD scan.
Default: no removal of average
Comments:
values >3 are not accepted and values <0 are the same as 0
values >0 (e.g. 1 or 2) are possible but probably not advisable
specify -averageOrder 0 if you observed with offset OPD tracking
(then there are no OPD zero crossings) and if your source is
very weak and the delay solutions returned by oirGroupDelay show
a zigzag pattern typical of zero OPD artefacts.
If you use this option be sure to use it also on your calibrators.
Purpose:
Remove the known Instrumental OPD components from the fringe spectra
that oirFormFringes produced. Each data point at frequency k (in
spatial units: 2 pi/lambda) is multiplied by exp(-i*k* OPD) where OPD
is the sum of the OPD from the MIDI piezos and the VLTI delay lines.
Special Notes:
1. The result is a FITS imaging_data file in "pseudo-complex"
format. I.e. it is in REAL format but each row is twice as long
as the original data: each value is a (real, imaginary) pair.
2. The VLTI Delay values may contain large fictitious offsets of
up to 1cm if the system was not correctly zeroed at setup. To
avoid applying these large fictitious offsets, the program
determines the MODE of the tracking positions during the run and
subtracts this before applying the OPD shift. This modal value
is printed by the program and also stored in the output header as
"OPD0" in meters.
Parameters:
inFile: name of the output from oirFormFringes
outFile: output file name. The format is a MIDI style FITS table
imaging_data file in pseudo-complex format.
Purpose:
Estimate the group delay for each measured spectrum. The group delay
is the peak of the Fourier Transform of the spectrum (thus in the
delay domain, not the frequency domain). Each spectrum is fourier
transformed. Note that because the MIDI beam-combiner has only two
output phases it is essentially a cosine correlator and not a complex
correlator. This means that the group-delay has TWO peaks: at
positive and negative delays.
Because the actual OPD was modulated by the MIDI piezos and this
modulation was deRotated in oirRotateInsOpd, one of the two peaks is
nearly stationary from frame to frame (only atmospheric OPD movement
remains) while the other peak moves around with TWICE the instrumental
modulation. Thus if we average a few frames together the wrong side
band is strongly suppressed. In this program a gaussian smoothing of
sigma = gSmooth frames (default 4) used in this averaging.
Parameters:
infile: output of oirRotateInsOpd
outFile: output file containing TWO interesting tables:
1. A pseudo-complex imaging_data table containing the raw FFTs of
the input data
2. A DELAY table containing the value of the peak delay (after
smoothing) of the above table at each frame. This DELAY table has
three columns: TIME (same as raw data), TELESCOPE (always 1,2) and
DELAY (peak of group delay, with OPD0 added), in seconds (IAU
standard). Thus you have to multiply by the speed of light to get
meters.
gSmooth (optional; default: 4 frames)
Specifies the smoothing in the frame domain before looking for a
delay peak. Default is gaussian sigma = 4 frames. Choose larger
value, like 10-15 for weak sources in good weather, shorter for
strong sources in bad weather.
maxOpd (optional; default: 200 microns) Do not look for delay peaks
more than maxOpd away from either
a. OPD of delayline tracking system (default)
b. OPD calculated by a first pass through data and specified by
the firstFile option
firstFile (optional; default: not used)
If specified it contains a the filename of the output from a
previous run of oirGroupDelay or oirPowerDelay. The delay table
from this file is used as a first guess before searching for the
group delay. If maxopd is fairly small, this limits the search
range of oirGroupDelay and it is thus less likely to misidentify
noise peaks as delay peaks.
weightFile (optional; default: uniform spectral weighting)
If specified this is the name of a MIDI type file containing an
IMAGING_DATA table with at least one record containing a
1 × nwavelength array. This will be used as relative weights
for the input spectra before looking for the group delay. This
can improve the S/N for sources with a non-flat spectrum.
-search (optional; default: tracking mode)
If specified oirGroupDelay assumes that this is SEARCH mode data
covering a large OPD range, rather than TRACK mode data. In
SEARCH mode, the program slides its relative OPD zero point along
with each scan to avoid running out of range during the FFT.
Data of this type can be analyzed with the IDL program midiSearch.
Purpose:
A more modern version of oirGroupDelay with special features to
produce more sensitive and reliable estimates of the group delay for
faint sources. The basic procedure is the same as in
oirGroupDelay, but there are many new features. Among these:
a. Smoothing intervals etc. are specified as times in seconds
rather than frame counts.
b. The delay table is not calulated at every frame but only at a
fraction of the smoothing interval specified by sigOut.
c. If requested the program will fit, in each averaging period
specified by smooth, an OPD that changes linearly with
time, rather than being constant. This allows the integration
interval used to search for the delay peak to be lengthened.
The range of OPD rates to be searched is specified by the
parameter ngrad.
d. The final delay track can be median smoothed over a time interval
medSmooth before being written to the output.
e. The search for the delay peak can be restricted to maxopd
from either the telescope delay tracking positions or a first
guess estimate of the delay positions typically provided by
oirPowerDelay. This reduces the chance of finding a noise peak.
f. When finding the value of the delay in a time interval around a
specific time, you can exclude a small amount of data at the
center of the interval. This has the effect of removing a bias
when the delay value is used to rotate the phases of the data at
the center. The bias is introduced because the noise in the data
influences the correction to the phase of the same data, tending
to cause all phases to approach zero. The size of the exclusion
interval is set by exInterval.
Parameters:
infile: output of oirRotateInsOpd
outFile: output file containing TWO interesting tables:
1. A pseudo-complex imaging_data table containing the raw FFTs of
the input data
2. A DELAY table containing the value of the peak delay (after
smoothing) of the above table at each frame. This DELAY table has
three columns: TIME (same as raw data), TELESCOPE (always 1,2) and
DELAY (peak of group delay, with OPD0 added), in seconds (IAU
standard). Thus you have to multiply by the speed of light to get
meters. The delay table contains output sampled at a fraction
(sigOut) of the specified gSmooth interval.
gSmooth (optional; default: 0.2 sec)
If specified can change the smoothing in the frame domain before
looking for a delay peak. Default is gaussian sigma = 0.2 sec
frames. Choose a larger value, like 0.6-0.8 for weak sources in
good weather, shorter for strong sources in bad weather.
maxOpd (optional; default: 200 microns) Do not look for
delay peaks more than maxOpd away from either
a. OPD of delayline tracking system (default)
b. OPD calculated by a first pass through data and specified by
the firstFile option
medSmooth (optional; default: no median smoothing) Before writing the
output delay table, smooth the values with a median filter of
width medSmooth (seconds). For low signal sources a fairly large
value e.g. 1 sec, is advisable.
nGrad (optional; default: 1) Check for linear OPD variations
within the smoothing interval specified by gSmooth. The program
looks for maximum coherence over phase gradients in a range
of +/-nGrad * 1 radian/gSmooth interval.
exInterval (optional; default: 1.2) Exclude data within this interval
when estimating delays, to avoid noise bias. The excluded
interval = exInterval * 0.25 * gSmooth
firstFile (optional; default: not used)
If specified it contains the filename of the output from a
previous run of oirFGroupDelay or oirPowerDelay. The delay table
from this file is used as a first guess before searching for the
group delay. If maxOpd is fairly small, this limits the search
range of oirGroupDelay and it is thus less likely to misidentify
noise peaks as delay peaks
-search (optional; default: tracking mode)
If specified oirGroupDelay assumes that this is SEARCH mode data
covering a large OPD range, rather than TRACK mode data. In
SEARCH mode, the program slides its relative OPD zero point along
with each scan to avoid running out of range during the FFT.
Data of this type can be analyzed with the IDL program faintSearch.
Purpose:
Make a crude estimate of the group delay for very low S/N data as a
function of time by first smoothing the delay data coherently as
specified by gSmooth and then smoothing the the delay data
incoherently over a much longer period of time, ampSmooth, and
possibly smoothing in the delay direction by delaySmooth pixels.
Then search for the peak in the delay direction and write the delay
value at the peak into the output delay table. This incoherent track
can be used for a second pass of oirFGroupDelay.
Parameters:
infile: output of oirGroupDelay or oirFGroupDelay.
outFile: output file containing TWO interesting tables:
1. A real imaging_data table containing the FFTs of the input
data, first smooth coherently by gSmooth seconds, then con-
verted to ABS(f)^2 and smoothed by ampSmooth seconds.
2. A DELAY table containing the value of the peak delay (after
smoothing) of the above table at each frame. This DELAY table has
three columns: TIME (same as raw data), TELESCOPE (always 1,2) and
DELAY (peak of group delay, with OPD0 added), in seconds (IAU
standard). Thus you have to multiply by the speed of light to get
meters.
gSmooth: (optional; default: 0.4 sec)
Coherent averaging interval.
ampSmooth: (optional; default: 16 sec) after coherent averaging by
gSmooth, take the absolute square of the delay function
(FT of the spectra) and average in the time direction with a boxcar
with a width of ampSmooth seconds.
delaySmooth: (optional; default: 10 pixels) Smooth the delay function by
a gaussian with this sigma (pixels) in the delay direction. This
lowers the resolution in delay, but further improves the S/N.
medSmooth: (optional; default: 15 sec) Median smooth the positions of
the delay peaks over this interval before writing the output delay
table.
Purpose:
Remove the group delay measured by oirGroupDelay as well as the
instrumental OPD from fringe data (the output of oirFormFringes).
Same algorithm as specified in oirRotateInsOpd.
Additionally estimate an offset PHASE for each frame and remove it.
This phase is primarily the result of water vapor dispersion (i.e.
non-constant index of refraction). Several spectra are averaged
together to remove the beats from the off-side-band and improve the
signal/noise, then the phase offset is determined and removed.
Parameters:
infile: output of oirFormFringes
delayFile: output of oirGroupDelay or oirFGroupDelay
outfile: output, in format identical with output of oirRotateInsOpd
smooth: (optional; default: 0.2 sec) 1-sigma size of gaussian smoothing
function in estimating water vapor phases
frames: if set then smooth is specified in frames instead of seconds
ngrad: (optional; default: no gradients) as in oirFGroupDelay, try
to find best linear gradient of water vapor phase within smooth.
phasefile: instead of finding the water vapor phases myself, use the
output of a previous run of this program in specified file.
nophase, nogroup, noopd: suppress respectively: water vapor estimation,
removal of group delay estimates, removal of instrumental opd, for
diagnostic purposes.
Purpose:
Try to choose (automatically) which frames to include in the output
visibility estimation: This is currently NOT done on the basis of
estimated amplitude since this biases the result. The current
criteria are:
1. OPD tracking distance from 0-point. Experience shows significant
attenuation of the fringe (a few percent) if this is larger than
about 150 microns for the PRISM and 800 microns for the PRISM. So
if ABS(instrumental OPD-group OPD) > maxOpd the frame is flagged.
When the OPD tracking point crosses zero MIDI can get confused with
the sky background, especially if oirFormFringes is run with the
-removeAverage or -averageOrder options. So if
ABS(instrumental OPD-group OPD) < minOpd the frame is flagged.
2. Jumps in OPD. There are some instrumental OPD jumps (due to
telescope refocusing) where the fringe is presumably attenuated.
If the 2nd difference in the group OPD is > jumpOpd the frame is
flagged.
3. Low signal, poor tracking: If the OPD jitter is larger than the
specified jitterOpd, this is considered poor tracking and the frames
are flagged. The OPD estimates from oirFGroupDelay are read. We
take the 2nd difference to remove trends, then take the absoute value
and then median smooth over 50 frames before comparing to jitterOpd.
Output is written into a FLAG Fits table that specifies the flagged
time intervals.
Parameters:
infile: output of oirFormFringes (only used to read the instrumental OPD)
delayFile: output of oirGroupDelay or oirFGroupDelay
flagFile: name of file to contain FLAG tables
maxOpd: maximum acceptable difference between the tracking and the
true OPD (microns)
minOpd: minimum acceptable difference between the tracking and the
true OPD (microns)
jumpOpd: OPD jumps larger than this (microns) are flagged
jitterOpd: (optional; default: 1.5 microns) if jitter of tracking is
larger than this value then data is flagged
Purpose:
Average all unflagged but phase rotated frames together to arrive at
a single (complex) visibility.
Output is in IAU/ESO OI_VIS format FITS tables which is different
from imaging_data. These can be accessed with my IDL oirGetVis
routine: data_array = oirGetVis(file, wave=wave) where wave contains
the channel wavelengths in microns. data_array is an array of
structures containing (among other things) visamp and visphi, i.e.
the amplitude and phase.
Parameters:
infile: output of oirRotateGroupDelay
flagFile: output of oirAutoFlag (or manual flagging)
outFile: name of the OI_VIS file
Purpose:
The first step of the operations for reducing chopping photometry.
Sky and target frames are averaged, target-sky computed, and an
on-slit estimate of residual sky is made and removed. The result
is output as a set of detector images for one, possibly multiple,
input files. For HIGH_SENS photometry this must be run sequentially
for the AOPEN and BOPEN photometry input. For SCI_PHOT photometry
it must be run once on the same file used for interferometry. The
output contains one set of images for the average of all good input
frames, and then a series of such sets made with (5) independent
subsets of the input frames so that the rms variations in calculated
parameters can be estimated.
Parameters:
"inFiles": a list of raw MIDI (PRISM/GRISM) input files representing
one exposure. There may be more than one file because they were
broken up into 100MB files by the on-line system. If there is more
than one file they should be separated by spaces and included
between double quotes: "file1 file2 file3"
outFile: output file name
nDrop: exclude this many frames after a transition from Sky to Target
while the chop mirror has not settled (default = 2)
dSky: Normally after Target-Sky subtraction, an additional sky
correction is estimated at the top and bottom of the slit. This is
necessary for faint sources (under 1Jy). The program normally
estimates the sky from rows 7-11 and 23-27 (PRISM) or 4-8 and 27-31
(GRISM). If specified, dSky indicates that the user wishes to move
these rows dSky pixels apart (dSky positive) or together (dSky
negative). If dSky is large enough so that the sky region moves out
of the measured window, no on-slit sky subtraction is performed
(this is indicated in the text output of the program).
maskfile: file containing image in standard MIDI format designating area
to fit sky background along spectrum. If specified dSky and refFile
are ignored
refFile: file containing an accurate description ofthe curvature of the
spectra on the detector in order to more accurately position the sky
subtraction zones. If not present the spectra are assumed to be
straight.
polyOrder: (optional; default: 1 for UTs, 0 for ATs) order of polynomial
fit to background along the slit.
sky0: if set in SCIPHOT mode, make a special attempt to debias the sky
images
Purpose:
Apply a mask to the output(s) of oirChopPhotoImages and compute all
kinds of geometric and arithmetic averages of the AOPEN and BOPEN
channels. They these are summed in the y-direction both with and
without masking (as described in oir1dCompressData) to produce
spectra. The mask should be IDENTICAL with the one specified there.
Input are two separate (sets of) files (AFile and Bfile) containing
raw chopping data for AOPEN and BOPEN shutter positions, or one file
in SCIPHOT mode (ABFile).
Output is a imaging_data FITS table file with 12 rows, each row
containing a single DATA1 array:
1. Total Flux (ADU/s/channel) for AOPEN shutter position
2. Total Flux (ADU/s/channel) for BOPEN shutter position
3. Total Flux GeometricMean of A&B (i.e. of rows 1 and 2).
4. Masked Flux (ADU/s/channel) for AOPEN shutter position
5. Masked Flux (ADU/s/channel) for BOPEN shutter position
6. Masked Flux GeometricMean of A&B (i.e. of rows 4 and 5).
7-12 are the (poorly) estimated rms of the above quantities.
Parameters:
"AFiles": AOPEN HISENS chopped photometry files from oirChopPhotoImages
"BFiles": BOPEN HISENS chopped photometry files fromoirChopPhotoImages
"ABFiles": ABOPEN SCIPHOT chopped photometry files from oirChopPhotoImages
outFile: FITS imaging_data file containing output
maskFile: mask file. Should be the same as for interferometric data.
maskShift: move the mask up/down by the specified number of pixels
before masking
-autoshift: figure out the shift by myself by crosscorrelating
the mask with the images. If specified, overrides shift.
outMaskFile: If specified, put result mask of -shift or -autoshift in
this file for later usage (e.g. in interferometric processing)
Purpose:
This program takes the output files from several of the previous
steps, as applied to a calibrator and computes the instrumental
visibility and spectrophotometric calibration.
In this and the programs described below I use "tags" to identify
all the files belonging to one observation set. The full file names
are tag.type.fits where type describes the file type (see complete
listing below). Thus oirRedCal expects all input files to have this
type of name.
The output of oirRedCal is an OI_VIS table with 3 rows:
1. Instrumental visibility (amplitude and phase)
2. Spectrophotometric calibration (ADU/s/Jy)
3. "Strehl Ratio": not really, but the ratio of masked
counts/total counts per spectral channel.
Parameters:
sourceTag: the initial part of input file names, may also include a
directory. e.g. hd10380 or /strw11/jaffe/ngc4151
The type flags actually used are photometry (output of
oirMakePhotoSpectra), and correlated flux (output of oirAverageVis),
thus the real file names would be hd10380.photometry.fits
and hd10380.corr.fits.
output: is not specified. The output is written into a file named
sourceTag.redcal.fits and contains the above mention OI_VIS table.
Purpose:
Combine reductions of a target and calibrator observations to produce
calibrated visibilities and plots. The output is three FITS table
files.
1. An OI_VIS table file: targetTag.calvis.fits containing one
row per input row in the file targetTag.corr.fits
This contains calibrated visamp and visphi.
2. An OI_VIS table file: targetTag.calcorr.fits containing one
row per input row in the file targetTag.corr.fits
This table contains calibrated correlated fluxes and phases.
3. An imaging_data file: targetTag.calphot.fits containing two
rows:
1. calibrated photometry (Jy) total
2. calibrated photometry (Jy) under mask
Parameters:
targetTag: descriptor for the science target. The types actually used
are .photometry and .corr
calTag: descriptor for the calibrator. The type actually used
is .redcal
output is as described above: targetTag.calvis and targetTag.calphot
-calSpecFile: a FITS file containing a MIDI IMAGING_DATA table with one
row containing the calibrator flux (Jy) at the same wavelengths as
the target being calibrated. If specified this data overrides the
specification of calflux.
-calflux calFlux10: specify the flux of the calibrator used for the
flux calibration in Jansky at 10 microns. A Rayleigh-Jeans spectrum
is assumed. If not specified, 1 Jy is used.
-cald calDiameter: visibility calibrator diameter in milliarcsec. If
not specified, 0.0 mas is used.
-nophot: Assume that source photometry is not very accurate. The effect
is that correlated fluxes are computed as:
Jy(calibrator) * raw_correlated_flux(target)/raw_correlated_flux(calibrator)
instead of
Jy(calibrator) * visibility(target) * raw_total_flux(target)/raw_total_flux(calibrator)
Purpose:
Convert chopped interferometric data in SCI_PHOT format into something
that looks like what you would have gotten if you had run a standard
photometric sequence (AOPEN/BOPEN) in SCI_PHOT mode and which can be
digested by oirMakePhotoSpectra. If you have chopped raw tracking
data in SCI_PHOT mode, first run it through oirChopPhotoImages to
subtract the sky from the target frames and average everything
together. The output is then the input (ABfile) to the
current program. The photometric channels (channels 1 and 4) are
bent according to the information in refCoordFile and rescaled
according to the coefficients in crossCoeffFile to what you would
have measured in channels 2 and 3 if you had observed with only one
shutter open. There are then two output files:
outFileBase.Aphotometry.fits and outFileBase.Bphotometry.fits that
simulate the result you would have gotten with shutters AOPEN and
BOPEN. These can now be fed into oirMakePhotoSpectra to produce
photometric spectra.
Parameters:
ABfile: output of oirChopPhotoImages for an ABOPEN file containing
chopped data.
outFileBase: prefix for the two output files. These will be named
outFileBase.Aphotometry.fits and outFileBase.Bphotometry.fits
refCoordFile: a "data" file containing an IMAGING_DETECTOR table that
describes the curvature of the spectra on the detector. This
information will be used to recurve the photometric channels to look
like the interferometric channels (so that the same masks can be
applied). This will typically be created by a run of oirCurveCal.
crossCoeffFile: a "data" file containing the cross coupling
coefficients between the interferometric photometric channels.
This will typically be created by a run of oirCrossCoeff.
Purpose: Take a reduced photometric data file (e.g. template.photometry.fits) produced by dispPhot or midiPhotoPipe from AOPEN/BOPEN chopped photometry data, and rescale it using photometry data on the same target taken in SCI_PHOT mode (produced by dispSciPhot or oirSci2HiPhot) so that output = (A + BX) * template, where A and B are chosen to give the best least squares fit of output to scale. Parameters: templateFile: a nice reduced standard output of chopped photometry data taken in standard AOPEN/BOPEN mode with the SCI_PHOT beam combiner. scaleFile: a nice reduced standard output of photometry data taken from the outer PA + PB channels of interferometry data taken in SCI_PHOT mode. output: name of the output file
Purpose:
Calculate cross-coupling coefficients (kappa coefficients)
for SCI_PHOT mode data
Parameters:
Afile output of oirChopPhotoImages representing data
taken in chopping photometry mode with the shutter in AOPEN position
Bfile output of oirChopPhotoImages representing data
taken in chopping photometry mode with the shutter in BOPEN position
coordFile: FITS file containing "imaging_detector" table specifying
distortions and wavelength information for this SCI_PHOT detector
mode.
outFile: output file name containing the kappa coefficients in the form of fits
IMAGING_DATA tables.
Purpose: Computes the temporal mean and RMS of the DATA sections
of a MIDI data file.
Inputs:
input: input data file, compressed or otherwise
output: output data file. IMAGING_DATA table contains only 2 rows, the mean and rms.
float: must be specified if the input data is raw, i.e. integer format
Purpose:
Read reduced chop imaging files (from oirChopPhotoImages).
Fit gaussians to the spectra at each x-position, then fit power laws to the centers
of these gaussians to represent the curvature of the spectra with x. When you
are done, write the coefficients of these power laws into the dmc/dmp parameters
in the IMAGING_DETECTOR tables in the output file. While you are at it, since
you've done most of the work already, write out an IMAGING_DATA table with
gaussians as specified by the power laws because these make a plausible mask for
extracting photometry and correlated flux from normal measurements.
Parameters:
AFile: output of oirChopPhotoImages for data taken on
a bright star with the shutter in AOPEN position.
BFile: output of oirChopPhotoImages for data taken on
a bright star with the shutter in BOPEN position.
output: name of FITS file to contain output tables
fudge: optional; The half widths of the exponential used to create the masks are increased
by this factor. If omitted, 1.0 is used.
Purpose:
Run dispVis, dispPhot, and oirRedCal on 3 files specifying
interferometric and photometric raw data. Effectively a complete but
uncalibrated reduction of this set of data. Plots tracking versus
estimated group delay and estimated instrumental visibility. After
running midiPipe on a science target and a calibrator you should run
midiCalibrate,tag, caltag to calibrate the science data.
This procedure produces many files, of which the most important for a
routine reduction are:
tag.corr.fits
containing the estimated correlated flux (instrumental units)
tag.photometry.fits
containing the estimated photometric fluxes (instrumental units)
tag.redcal.fits
containing the estimated instrumental visibility
For experts:
tag.groupdelay.fits
contains group delay tracking information.
Parameters:
tag: output tag to identify files to dispVisPhot
files (optional) a string array of 3 input file names, each of which
may be a blank-separated list of files. The files must specify
one interferometric observation and two (A- and B- shutter)
photometry observations.
If files is not specified, the GORGONZOLA GUI will come up to
allow a manual specification of files.
mask (optional): mask file name. If not specified a default,
depending on the instrumental setup, is used. For old prism
high_sens files (before Dec 2003) use:
mask = getenv('drsRoot')+'/maskfiles/prismmask_before_2003dec01.fits'
smooth: Delay high-pass smoothing width to supress background
fluctuations. Default is 50 but use a smaller value if
your source is very weak.
gsmooth: group delay estimation averageing half-width (frames). Default
is 4 but use larger value for weak sources during stable atmosphere,
smaller for strong sources during rapid OPD changes. For weak
sources during rapid OPD changes, give up.
dSky: move on-slit regions for sky estimation (c.f. oirChopPhotometry ).
A large positive number means no on-slit sky estimation.
/dAve: if specified the -removeAverage option is applied in
oirFormFringes. See the text there for details. This removes
fringe artifacts near the OPD=zero point which can be useful when
you are looking for weak sources but may reduce correlated flux.
If you specify /dAve, then the -minopd 15 microns option is
specified in oirAutoFlag, which then flags frames within 15 microns
of the zero OPD crossings.
Purpose:
Execute dispVis which reduces the interferometric scans specified by
fileList and produces Tag.corr.fits, and Tag.groupdelay.fits files.
This is same as midiPipe but the photometric reduction is skipped. See
midiPipe for details. Note the slightly different syntax for fileList.
Purpose:
Execute dispPhot which reduces the photometric scans specified by
fileList and produces Tag.photometry.fitsfile.
Purpose:
Similar to midiVisPipe but the fileList should point to data taken in
fringe Search mode instead of fringe Tracking mode. The output
contains the usual fits files. Additionally a plot of fringe amplitude
versus OPD during the search phase is produced.
Purpose:
Run oirCalibrateVis with specified parameters. If the calDatabase
parameter is specified as an existing database (c.f. Calibration
Database) then the database is searched for a calibrator within 1'
of the position in your calibration files, and if found, the
calibrator spectral and diameter data are taken from there.
In addition a whole bunch of plots is produced in a file named
sourceTag.ps. It also produces the following FITS files:
sourceTag.calphot.fits containing calibrated photometry,
sourceTag.calvis.fits containing the calibrated visibility.
sourceTag.calcorr.fits containing the calibrated corr. flux
(analogous to uncalibrated sourceTag.corr.fits)
If /print is specified, a dozen or so sourceTag.xxxx.dat files are
produced containing the information that was plotted as ASCII tables.
Purpose: run dispVis and dispSciPhot or dispSciPhotPhot to reduce a (chopped/interferometric/SCI_PHOT) input file, and optionally some chopped/photometry/SCI_PHOT data. Parameters: as in midiPipe with the addition of: file(s): an array of either 1 or 3 character strings specifying (blank separated) sets input files. If there is only one string, this contains data taken in interferometric mode with shutter in ABOPEN position. The interferometric signal is extracted with dispVis. If photoFile is NOT specified, the photometric signal is extracted from the same input files with dispSciPhot. If photoFile IS specified, dispSciPhotPhot is run. This extracts the photometric signal in the same way, but uses it to rescale previously reduced real photometric observations using oirRescaleSpectra. If 3 input arrays are given, the reduction proceeds with rescaling as just described, but the raw photometric observations in AOPEN/BOPEN mode are specified in the 2nd and 3rd strings. coord: file containing distortion/wavelength information for individual MIDI channels, not yet included automatically with data from Paranal. Defaults to either of : $drsRoot/software/minrts/config/mioSetup_FIELD_PRISM_SCI_PHOT_SLIT.tmp $drsRoot/software/minrts/config/mioSetup_SPECTRAL_GRISM_SCI_PHOT_SLIT.tmp cross: file containing kappa/cross-coupling coefficients. Should have been computed from a previous run of midiCrossCoeff. It is the User's responsibility to do this. photoFile: contains the reduced output of standard AOPEN/BOPEN photometric observations of the same target. Optional. If provided, and file[s] only contains one, interferometric data set, it will be used as a template for rescaling the photometric data, as described in oirRescaleSpectra.
Purpose: Create a new mask file by constructing an image of the correlated flux as a function of wavelength and slit position on the detector and fitting a set of smoothed gaussians to this image. This mask is then (near) optimized for S/N in looking for fringes. In this form the procedure is usually applied to a strong calibrator. The process is to track the calibrator fringe using an initial mask, e.g. the default mask, with the procedure faintVisPipe. The OPD information from this run is then used to construct a coherent image of the entire detector plane. This image is smoothed somewhat and then fit with a set of gaussians whose position and width are allowed to vary slowly with wavelength. This set, normalized to a peak of unity at each wavelength, forms the new mask. For low S/N sources, i.e. your targets, it is probably a bad idea to perform this procedure. Alternatively you can ask midiMakeMask to consider only shifting an initial mask (the one generated from a calibrator) to get the best overlap with the fringe image. Inputs: tag: The usual string to keep track of the output files fileList: As in faintVisPipe, the file containing raw MIDI interferometry data initialMask: (optional; default: system reference masks) a mask to use while looking for fringes during the first pass through the data. If you are using the shiftOnly option, this should be the output of a previous run of midiMakeMask on a strong source. shiftOnly: (optional: default: full-fit) If specified, consider only shifting the y-position of the initial mask, but do not change its shape. smooth,gsmooth,noAve: Parameters for faintVisPipe. factor: (optional: default=1.0) For a full fit to the coherent image, increase the width of the determined best fit gaussians by this factor noDelete: (optional: default=delete) Normally all the (large) scratch files used to calculate the new mask are deleted by the procedure. If you specify noDelete they are left intact for possible diagnostics.
Calculate the Cross Coupling (Kappa) coefficients between the interferometric and photometric channels for data taken with the SCI_PHOT beam combiner. The input raw data should consist of two file (sets) containing AOPEN and BOPEN data with this combiner for a bright star. This routine runs dispCrossCoeff after digesting the input arguments and filling in defaults. This dechops the input images, straightens out the curved photometry channel images, sums the photometry in the y-direction, and divides the interferometry images by the photometry images to yield the kappa coefficients in the output file. Inputs: tag: output tag to identify files to dispCrossCoeff. fileList: an array of two strings, containing the AOPEN and BOPEN files. As usual if these are multiple files, these should be concatenated and separated by blanks. curve: a file containing descriptions of the curvature of the spectra. You can generate this yourself using oirCurveCal. If not specified this midiCrossCoeff uses the environmental variable prismscurve or grismscurve which are usually to be found in $drsRoot/software/minrts/minrts/config with file names like: minrtsCurve_FIELD_PRISM_SCI_PHOT.fits or minrtsCurve_SPECTRAL_GRISM_SCI_PHOT.fits
Completely process target and calibrator correlated flux data to yield calibrated correlated fluxes. Does not handle photometry data. Runs midiVisPipe on both input data sets, and midiCalibrate to execute calibration. Default parameters for midiVisPipe are used unless you specify a parmfile. Inputs: targetdata: raw interferometry data for target. If multiple files these should be blank separated and surrounded by "" signs. caldata: same as above for calibrator targtag: character string to mark all output files for target caltag: character string to mark all output files for target db: previously created database containing calibrator spectra etc. default: van Boekel database. mask: file containing mask used for both data sets. Default: new mask will be created from calibrator data using midiMakeMask parmfile:A text file containing valid IDL commands to set (a few) parameters for midiVisPipe. These will be executed using the IDL execute command before running midiVisPipe. Parameters that can be set are smooth,gsmooth,msmooth Outputs: all the normal outputs from midiVisPipe for both target and calibrator, plus outputs of midiCalibrate.
Compute a 2-dimension (wavelength,y-position) complex image of correlated flux from a source. This in contrast to a normal reduction that collapses the
raw spectrum to 1-dimension before further processing. The principel
use of this routine is to create images of calibrators that can be
used for constructing masks.
Inputs:
tag: character string to mark output files
file: raw interferometric input data
smooth: input high-pass filter interval for oirFormFringes (seconds)
noAve: if specified, do not remove spectral average in oirFormFringes
noDelete: This routine generates large temporary files that are
deleted at the end. If nodelete is specified (for diagnostic purposes)
this is suppressed.
Outputs:
A pseudo-complex image file tag.fringeimages.fits that contains
the complex output separately for each interferometric input channel.
Produce a set of simulations to indicate how much flux MIDI/EWS loses as a source becomes weaker and the group delay tracking becomes inaccurate. The procedure assumes that you have already run midiVisPipe or a similar procedure on both your target data and on a strong calibrator. midiDecorrelate will then: Extract the reduced correlated fluxes from both data sets. Rescale the--assumed noiseless--calibrator data so that it will produce a spectrum identical to the target. Extract a compressed noise data set from the target data displaced along the slit. In a loop: attenuate the rescaled calibrator data by a given factor, add it to the noise data, and reduce it with midiVisPipe as if it were a normal observation. Inputs: targetdata: raw interferometry data for target. caldata: raw data for calibrator targtag: reference name for reduced data files for target caltag: reference name for reduced data files for calibrator tag: reference name for data files produced by this program (default="weak") ratios: vector of values by which to attenuate rescaled calibrator ...: typical midiVisPipe parameters may be specified: mask,gsmooth,smooth,msmooth,twopass,ave parmfile: A text file containing IDL commands to be executed before starting the midiVisPipe procedure. This is an alternative way to specify values to be used for gsmooth etc. i.e. the file may contain statements like: gsmooth=.6 msmooth=2. Outputs: An array of IDL oirVis structures, i.e. what you would get with oirGetVis(tag.corr.fits) on each of the simulations. You can run midiCvis on each element of the array. The first structure in the array corresponds to the unmodified calibrator. The second element corresponds to your unmodified target. The succeeding elements correspond to the rescaled calibrator multiplied by the elements of ratios and added to noise data. Thus if you specified ratios=[1.,.1] on a moderately strong target, the 3rd output element should look almost identical to the 2nd, differing by a small addition of noise. If the target is very strong the 4th element will look like the 2nd, multiplied by 0.1, but if the target is not so strong, the tracking errors at this low flux will reduce the output flux by even more. Running this procedure with a variety of ratios should allow you to estimate the importance of this loss of flux for your target.
These routines allow access to databases containing position, flux, diameter, and spectral data for calibrators. Here I describe only the routines for creating and IDL version of a database from a Fits representation, and for retrieving data.
cdb = midiCalDatabase(fitsDatabaseFile) Purpose: Create an IDL internal version of a cal database from a fits file. There is currently only one such Fits database: vBoekelDatabase.fits which is included in the EWS distribution. This database was provided in 2010 by Roy van Boekel, and updated in 2012. It comes with no guarantee on quality, and no obligation of being maintained. It contains Cohen model representations of spectrophotometric data for about 800 stars. This function returns an IDL object:cdb (you can choose any name you like for this variable), which can be used in the access routines described below. Parameters: fitsDatabaseFile: The name of a fits file containing calibrator data in "Jaffe Calibrator Table Format". I can provide documentation on this format to anyone who wants it. The file "vBoekelDatabase.fits" is contained in the directory "root"/idl/calibrators/databases in the distribution .tar file for MIA+EWS.
cdb = vboekelbase() Purpose: A short cut to midiCalDatabase that assumes that you want the database by Roy van Boekel. Parameters: None
nFound = cdb->sourceByName (name, sourceData, diamData, photData, specData) Purpose: Find a source in an existing database given its name and return the stored data. Note: In this routine the name must agree with the standard name stored in the database. These are typically HD names (uppercase, no blanks or underscores). In most cases it is better to call sourceByAlias. Parameters: name: source name, character string Returns directly: a vector of 4 integers describing the number of data sets with this name describing: 1. =1 if some sources are found; =0 if none found 2. Source Names 3. Photometry 4. SpectroPhotometry Returns in calling argument list: SourceData: a structure containing mostly positional data about source, e.g. name, ra, dec (equinox/epoch) and velocities, proper motions and parallaxes (if known) diamData: a structure containing diameter information (if known), e.g. name, wavelength band where measured, diameter (and error), date ... photData: photometry data: name, band, flux(Jy) (U,Q,V if known), quality specData: spectrophotometry data: name, band, date, quality, wavelength array, flux array(Jy).
nFound = cdb->sourceByAlias(name, sourceData, diamData, photData, specData) Purpose: Same as sourceByName. Actually this program calls the other program. First however the name is converted to "standard" format (uppercase, no blanks). Then an alias table is searched for equivalent names. For the moment this table is largely empty but the intent is to fill it with other names (e.g. HR, HIP, IRAS, 2mass, constellation names). However if you use this routine with HD names you don't have to worry about blank spaces and upper/lower case problems. Parameters: see sourceByName
opd = cdb->sourcesNearPosition(ra, dec, sources, nmax=nmax, rmax=rmax) Purpose: Get information on calibrator sources near a specified position Parameters: ra: Right Ascension. Can be specified as a scalar=position in degrees; an array of three numbers= [hours, minutes, seconds], or a blank- separated character string: "hh mm ss.ss" dec: Declination. Same choices as ra for formats but all in degrees. nmax: (optional; default: 30) maximum number of sources to return. rmax: (optional; default: 1 degree) largest acceptable distance from the specified position Returns: A vector with the distance (degrees) from the calibrators found to the specified position. Returns in calling argument list: sources: source structures for the calibrators found. You can use sources.name as inputs to sourceByName to get additional data about these sources.
sourceData = cdb->sourceFromFile(fitsFile, diamData=diamData, photData=photData, specData=specData, rad=rad, /silent) Purpose: Find the calibrator nearest to a position specified in a MIDI data file, either raw data or reduced data. The information returned is similar to that in sourceByName. Parameters: fitsFile: reference data file. The RA and DEC specified in the FITS header are used as reference positions. rad: (optional; default: 1 arc minute) Maximum acceptable distance (degrees) from the reference position. silent: (optional; default: noisy) Supress some of the message generated while looking for targets. Returns: A source structure as defined in source ByName Returns in argument list: diameter, photometry and spectrophotometry data
spectrum = cdb->spectrumFromFile(fitsfile, diamData=diamdata, wavelength, rad=rad, /silent) Purpose: Find a calibrator near the position specified in a reference file header and return its spectrum (if found) interpolated to the wavelengths specified in the MIDI FITS file. Parameters: see sourceFromFile Returns: Spectrum of the calibrator (Jy) found near the position specified in the file header, interpolated to the wavelengths specified in the wave- length tables contained in the file.
midiDelayPlot, tag, title [,twopass=twopass] Purpose: Make a plot of tracked OPD and estimated group delay as a function of time for data which has already been reduced using midiPipe or midiVisPipe. The plot is similar to the one shown by these functions, but by setting !x.range or !y.range before calling the routine, or setting other IDL plot variables, you can influence the appearance of the plot, or direct it to a file, etc. Parameters: tag: as in midiPipe or midiVisPipe, sets a directory and prefix that determines which data to use. title: a character string to be used at the top of the plot as a title. twopass: (optional; default: do not plot twopass delays) also plot the first guess group delay as well as the result of oirPowerDelay when midiPipe or midiVisPipe were executed with the option /twopass.
dataStruct = oirGetData(datafile [,rows][,col=col]) Purpose: Read data from a MIDI FITS ImageData table file into an internal IDL structure. Parameters: datafile: name of input file. e.g. CenA.compressed.fits If the input data consists of multiple, to-be-concatenated files, specify files as a single character string with a space between the successive files. rows: (optional; default: all rows in file(s)). If specified this is an IDL array containing the row numbers that you want. These start at 1 not 0 (FITS standard). This option cannot be used for multiple files. col: (optional) A character string array specifying which columns of the data table to read. E.g. col=['DATA1','TIME'] Specifying this does not save time, but it can save space inside IDL.
visStruct=oirGetVis(visFile[,wave=wave]) Purpose: Read data from a MIDI FITS OIR Vis table file into an internal IDL structure. Parameters: visFile: name of input file, e.g. CenA.corr.fits wave: (optional) Returns the wavelength information for the vis data.
corr = midiGetCorr(tag[,wave=wave]) Purpose: Return a vector containing the (uncalibrated) Correlated Flux computed by midiPipe or midiVisPipe, for the data specified by tag. I.e. this is the visamp vector stored in the file tag.corr.fits. Parameters: tag: as used everywhere. wave: returns the vector of wavelengths (microns) corresponding to the data arrays described below. Returns: a vector of values specifying the uncalibrated correlated flux+ amplitude (ADU/s) for the data specified by tag.
phase = midiGetPhase(tag[,wave=wave]) Purpose: Return a vector containing the (uncalibrated) phase of the correlated signal, as computed by midiPipe or midiVisPipe, for the data specified by tag. I.e. this is the visphi vector stored in tag.corr.fits. Parameters: tag: as used everywhere. wave: returns the vector of wavelengths (microns) corresponding to the data arrays described below. Returns: a vector of values specifying the uncalibrated correlated phase (degrees) the data specified by tag.
phot = midiGetPhot(tag) Purpose: Return a structure containing the (uncalibrated) photometry for the data specified by tag. Parameters: tag: as used everywhere. Returns: an IDL structure named whatever is on the left-hand side of the "=" sign containing the uncalibrated photometric data as computed by midiPipe or by midiPhotoPipe. This is the data contained in the file tag.photometry.fits. For example phot[0].data1 contains a vector of the estimated total flux (ADU/s) from telescope A.
opd = midiGetOpd(tag) Purpose: Get the OPD tracking positions for the data specified by tag. Parameters: tag: as used everywhere. Returns: a vector sign containing the OPD tracking position (microns).
mydelay = midiGetDelay(tag) Purpose: Get estimated Group Delay positions for the data specified by tag. Parameters: tag: as used everywhere. Returns: an IDL structure with the group delay information calculated by oirGroupDelay. The more interesting elements in this structure are: mydelay.delay: the estimated group delay (microns) for each frame mydelay.time: time (MJD days) of each frame mydelay.amplitude: amplitude of fringe signal found at this delay (ADUs)
choppedData = midiChopImage(inputFiles) Purpose: Average all "target" frames in a set of chopped data (e.g. target acquisition or photometry frames) and subtract the average of all "sky" frames. Parameters: inputFiles: name of files, if multiple files this should be a single, blank-separated list: e.g. "file1.fits file2.fits" Returns: an IDL structure with elements like "choppedData.DATA1" "choppedData.DATA1" that represent the signals in each of the MIDI windows for this data. The structure has two rows: choppedData[0] contains Target-Sky while choppedData[1] contains Sky.
keyValue = midiGetKeyword(keyword, inputFiles) Purpose: Get keyword values from primary headers of one or more FITS files. Parameters: keyword: a character string specifying keyword. For ESO-style HIERARCH keywords specify at least two of the keywords and enough to be unique, e.g. "FILT NAME". inputFiles: a single character string or array of character strings specifying a set of files to search. If an array is specified, an array of values will be returned. If the keyword is missing in the header, the program returns 0. If it is present in some files and missing in others the program might crash. Returns: an IDL array containing the values of the specified keywords.
complexVisibilitySpectrum = midiCVis(oirVisibilityStruct) Purpose: Convert visibility structure into IDL complex visibility vector Parameters: oirVisibilityStruct: The IDL structure representation of a visibility spectrum. This is what is returned by oirGetVis when reading the output of oirAverageVis. So you can combine these: spectrum=midiCVis(oirGetVis(visFile,wave=wave)). Returns: an IDL complex array containing the visibility spectrum.
p = midiPhase (cArray) Purpose: Compute the phase of an array of complex numbers. Parameters: cArray: an IDL complex array Returns: an array of the phases of each element of cArray, i.e. arctan(Im(cArray)/Re(cArray)), in the range -pi <-> pi. Note: p = midiPhase(cArray) returns values in degrees (-180->180).
c = pseudoComplex (pcArray) Purpose: Convert a 2-dimensional array of pseudoComplex numbers into IDL complex numbers and transpose the array. Several of the C-programs, such as oirGroupDelay produce output data arrays that are in fact complex. These are stored in the FITS files as pseudoComplex, i.e. as pairs of real numbers. Inside IDL it is easier to deal with true com- plex numbers, so this routine converts the FITS pairs into IDL complex numbers. In addition, most of the FITS files are stored with channel number as the X-axis and frame number, or time as the Y-axis. For visualization it us usually more convenient the other way around, so pseudoComplextransposes the array. Parameters: pcArray: a 2-dimensional array of real numbers, each two values repre- senting one complex number, arranged in channel-time order. Returns: an array of the complex numbers arranged in time-channel order.
c = midiGetComplex (tag,qualifier) Purpose: Read a MIDI image file stored in pseudoComplex format. Parameters: tag: 1st component of a MIDI FITS file name qualifier: 2nd component of a MIDI FITS file name. For example, the output of oirFGroupDelay might be CenA.groupdelay.fits. The data1 element of the file contents can be read into an IDL complex array with: carray = midiGetComplex('CenA','groupdelay') Returns: an array of the complex numbers arranged in time-channel order.
EWS includes a data selection GUI, somewhat like GASGANO. It brings up a GUI and returns a list of files selected. To browse your raw data, and select some of it type:
FileList = midiGui()
There are alternate versions described below. There is a faster version that caches the contents of the file headers on disk, but it only works if you have write permission on the data disks:
FileList = midiGuis()
Note that IDL is indifferent to upper/lower case letters in commands and variable names (but not indifferent to case in FileNames).
The GUI is pretty spartan. Up at the top there are 10 buttons that let you select files, modify the display a bit, and quit.
Below this there is a scrollable area that shows information on each of the available files and on the left a column of ASTERISKs (*) showing which files are currently selected, or blanks if they are not. Initially files are brought up all un-selected.
In the next column you see the file name. Due to ESO limitations on file lengths, MIDI files cannot be longer than 100 MB or so, while a single observations may generate Gigabytes of data. The on-line data system breaks the data up into 100 MB files. Gorgonzola recognizes this process, and the displayed file name is only the FIRST file of all the files that constitute an observation. When Gorgonzola returns, all the sub-files for an observations are concatenated into a single character string with spaces between the file names. Most of the EWS programs can recognize such a string and process the files sequentially. NOTE, however, that if you use such a string to call a C-program directly (e.g. oir1dCompressData), you have to enclose the concatenated string in double quotes "...".
The other columns show keyword values for each observations, gleaned from the file primary FITS headers.
Two things:
We start with the easy ones:
QUIT causes the GUI to leave without making a selection
SELECT causes the GUI to leave and returns the selected files
UP/DOWN IDL and X-windows crash if thousands of widgets are put
on the screen, so I have limited the display to 100 rows.
If the rows you want to see are outside this range you
can push UP or DOWN to scroll. NOTE that the scroll bar
on the right only allows scrolling within the 100 displayed
rows.
HIDE Remove all currently unselected rows from the display. This
can considerably simplify things if you've already de-selected
a lot of files.
SHOW Reverse of HIDE: show all files, selected or not. This gives
you the opportunity to selectively re-select unselected files.
Sometimes the first few non-selected files are not displayed
after hitting the SHOW button. Try clicking the UP button.
To the right of this row the current HIDE/SHOW mode is displayed
Then the real meat: the BOOLEAN buttons. These are not strictly boolean but sort of. When you click one of them, the "caught" value is compared to all other values in the same column and the currently list of selected files is modified:
In the second row down on the GUI are six comparison operators:
= Equal != Not Equal > Greater than < Less than <= Greater than or equal > Less than or equal
To the right of this row the currently selected comparison is shown.
In the third row two combination operators:
AND The results of the comparison are
logically ANDed with the current selection
OR The results of the comparison are
logicall ORed with the current selection
Confused? It actually works fairly intuitively.
Suppose you have made a hundred or so exposures during a day, and suppose you are looking for a group of exposures with the PRISM in HIGH_SENS mode. The fifth column of selction buttons is labelled "INSGRIS" (shortend from INS GRIS NAME). Look down this column, with the slider if necessary until you find a button with PRISM displayed. Click on this and the words INGRIS and PRISM should appear below the AND/OR buttons to indicate your selection. Now click on the OR button. All observations with GRIS=PRISM are ORed with the initial selection (i.e. nothing), and asterisks should appear in the left hand column of all PRISM observations.
Now click on HIDE and all non-PRISM observations should disappear.
Now go down the INSOPT1 column and find an observations with OPT1=HIGH_SENS. Click on this, and then on AND. This will AND the set of HIGH_SENS observations with the current PRISM selection and any SCI_PHOT or OPEN observations should disappear. You can continue by selecting on OBSTARG (target name as given by the observer) or NRTSMODE (what the online program thought it was doing), or FILTER or SHUTTER position.
Using the other relational buttons, you can also select by time, because the file names are ordered by time. For example you can click on the file name of first observation you want, then click on the >= button and then on AND. All the earlier files will be deselected. Then you can click on the last observation, the <= button and AND, and the later observations will disappear.
When you have made the selection you want, click on SELECT.
At the top right the total number of files in the list, and the total number currently selected, are displayed.
If you click on a data item and then click HIDE, not much will happen (except that previously de-selected rows may disappear). You first have to press on of the boolean buttons to make something change.
files = midiGuiS()
Look for a file in the search directory called midiGui.SAV, and if you find it, assume that it contains the keyword values for all the MIDI files in this directory. This speeds things up a lot. If you don't find it, parse all the headers and store the summary in the abovenamed file upon exit. This will crash if you don't have write privliges.
If you can added/deleted files in the directory, so that the cached version is not up to date, just delete it and try midiGuiS again.
files = midiGui(dir=datadir) or midiGuiS(dir=datadir)
gorgonzola assumes that IDL is running in the directory where the data is. If this is not the case you can use the above forms to specify a different data directory. You can also use the IDL utility pushd:
pushd, datadir files = midigui() popd
The following standard file types are produced by the specified programs.
Some notes indicate how to access the contents of the programs.
Some of the lower level programs to access this data are described below
but almost all are listed here:
oirGetData(dataFileName [,rows] [,col=col],[ierr=ierr])
Get raw or semi-reduced detector data
oirGetVis(visFileName [,wave=wave])
Get reduced visibility data and wavelength information
oirGetDelay(groupDelayFileName)
Get estimated group delay for each reduced frame
oirGetOpd(fileName)
From raw or partially reduced data get piezo+VLTI delay
line OPD position
oirGetMeanRMS(inputfile)
Take pixel by pixel mean and rms of all frames in an observation
oirGetDetector(fileName)
Get data from IMAGING_DETECTOR table, specifying detector layout
oirGetWaveNo(detectorFile, region)
Get wavenumbers (here defined as 2*!pi/lambda(micron)) for
each pixel in specified detector region (1-relative)
midChopImage(datafile)
Return 2-row structure with mean of Target - mean of Sky
detector images in first row, and mean of Sky in 2nd row.
File types:
.compressed
Produced by oir1dCompressData, contains a header, an
imaging_detector table and a floating point imaging_data table.
The last has a 1-dimensional DATA array for each input frame
for each detector window.
This can be accessed with oirGetData(dataFileName):
dataArray = oirGetData(dataFileName)
oirGetData works on raw and partially reduced detector data.
It returns an array of structures. Each structure corresponds to
one data frame, and corresponds to ESO FITS interferometry
data tables. Each row of the table contains information on
the time of exposure (MJD days), exposure time (sec), OPD positions,
chopping mirror positions, and finally the data itself.
dataArray[15].time contains the MJD time of the 16th frame
(IDL is 0-relative).
dataArray.data1 is in general a 3-dimensional data cube, where
each plane represents the detector data from the 1st specified
detector region, and the 3rd dimension is frame number.
The data arrays are INT for raw data and FLOAT for partially
reduced data. Note that the raw data (16-bit integers) is
offset by -32768 from the true detector zero values.
.fringes
Produced by oirFormFringes, contains a header, an imaging_detector table
and a imaging_data table with a single DATA1 array for each input frame.
This can be accessed with
table = oirGetData(dataFileName)
data = table.data1
.insopd
Produced by oirRotateInsOpd. Contains a header and imaging_detector
table, and a pseudo-complex imaging_data table with a single DATA1 array
per row. Header contains keyword OPD0 describing offset subtracted from
all OPDs before rotation (=mode of tracking OPDs).
This can be accessed with
table = oirGetData(dataFileName)
data = pseudocomplex(table.data1)
data is then COMPLEX(nFrames, nFreq).
.groupdelay
Produced by oirGroupDelay. Contains a header and imaging_detector table,
and a pseudo-complex imaging_data table with the FFT of .insopd. The
header contains some special keywords:
OPD0: as in .insopd
OPD1,OPD2: coordinates of delay direction in output FFT. The delays
= OPD1 + OPD2*(xpix-1).
This table can be accessed with oirGetData(dataFileName). The
data section can be accessed using a call to pseudoComplex, e.g.
gd = oirGetData('mydata.groupdelay.fits')
gdc = pseudocomplex(gd.data1)
gdc is then COMPLEX(nFrames, 512), with the y-axis representing delay.
Additionally there is a 3rd table DELAY containing (per frame) the
time (MJD days), telescope numbers and estimated OPD (seconds) and
fringe amplitude. This can be accessed with oirGetDelay(fileName)
or midiGetDelay(tag).
.ungroupdelay
Produced by oirRotateGroupDelay . Contains the usual header and
detector tables and a imaging_data table in pseudo-complex format.
Can be accessed with
ugd = oirGetData(filename)
ugdc = pseudocomplex(ugd.data1) which is COMPLEX(nFrame, nFreq)
Additionally there is a 3rd table DISPERSION containing (per frame)
the value of the average phase subtracted from each row. The
table has three columns "TIME" "TELESCOPE" and "DISPERSION", which
for the purpose at hand, is the subtracted mean phase in degrees.
This is accessed by the generic fitstable routines:
t = obj_new('fitstable', filename, extname='DISPERSION')
dispData = t->readrows()
obj_destroy,t
.flag
Produced by oirAutoFlag. Contains a header and a FLAG table with the
time intervals that have been flagged. There is no oirGetFlag routine.
You can use:
t = obj_new('fitstable', filename, extname='FLAG')
flagData = t->readrows()
obj_destroy,t
.corr
Produced by oirAverageVis. Contains a header, an OI_WAVELENGTH table
specifying the wavelength for each channel, and an OI_VIS table
containing averaged correlated fluxes.
Can be accessed with
visTable = oirGetVis(filename, wave=wave) with wave optional
visTable.visamp is an array of (nWave) correlated fluxes (Jy)
visTable.visphi is an array of (nWave) differential phases (degree)
visTable.visamperr contains the estimated rms of visamp
visTable.visphierr contains the estimated rms of visphi
wave contains nWave values of the wavelength (microns)
.photometry
Produced by oirChopPhotometry. Header, detector tables, and
imaging_data table in float format
Can be accessed with photData = oirGetData(filename)
photData[0].data1 contains the photometry data for telescope A
photData[1].data1 contains the photometry data for telescope B
and so forth (see heading oirChopPhotometry above)
.redcal
Produced by oirRedCal. Header, OI_WAVELENGTH and OI_VIS tables
Can be accessed with oirGetVis(filename, wave=wave)
.calphot
Produced by oirCalibrateVis. Header, detector, imaging_data table
Can be accessed with oirGetData(filename)
.calvis
Produced by oirCalibrateVis. Header, OI_WAVELENGTH and OI_VIS tables
Can be accessed with oirGetVis(filename, wave=wave)
.calcorr
Produced by oirCalibrateVis. Header, OI_WAVELENGTH and OI_VIS tables
Can be accessed with oirGetVis(filename, wave=wave)
Table-of-Contents C-routines IDL-Routines