- Is it possible to have a single database file that stores all the
calibration information (eg spectra positions, flux calibration,
wavelength calibration), or do I need a different database file for
each different type of calibration data ?
- I think that, because the standard star is observed through a
single fibre, and then this calibration is applied to all fibres, you
should wavelength calibrate before flat fielding and flux
calibration. This means that you should wavelength calibrate the dome
flats before finding the normalised flat field spectra. But I haven't
managed to totally convince myself (and others) yet.
The data reduction package contains the tasks required to reduce the calibration and target data. It is assumed that PixCel outputs a FITS file for each read, which contains the four quadrants assembled into one image. PixCel can read an external FITS header fragment which means that the FITS files output from PixCel will have all the required header keywords (though not all with the correct values). The final data product for the target data is an x,y,lambda data cube, which can be visualised as a cube where the z-axis is wavelength and each plane is a picture of what the IFU observed at that wavelength.
The first version of the data reduction package will deal separately with the different types of data (eg dome flats, sky flats, arc lamps, flux standards and target observations. For each type of data there are one or more pipeline scripts to reduce the data. Each pipeline script runs a series of IRAF tasks. Each IRAF task can also run stand-alone.
Some tasks are always called by the pipeline scripts, others are called depending upon the value of FITS header keywords. This enables 'QuickLook' data reduction at the telescope to run a certain subset of the possible tasks in each pipeline. The initial values of the FITS header keywords which drive the data reduction are set by the user at the start of an observation through the instrument control software. There is a task to allow the user to easily change the values of these keywords is required.
Below is an illustration of the order of the pipeline tasks that would be run to produce a QuickLook data reduction (ie not wavelength or flux calibrated) from a beamswitched pair.
Make illustration for here
Pipeline task name | Brief description |
docirpass | basic processsing required by all observations |
findspecpos | trace the spectra and write positions to database file. extract the spectra. |
dodomeflat | make final dome flats |
doskyflat | find normalised throughput values for each fibre |
dofinalflat | make final flat field from final dome flat and normalised throughput values |
doarc | find wavelength calibration from arc observation |
dotarget | produce x,y,lambda data cube for target observation (could be object, sky, flux std) |
PARAMETERS
OUTPUT FILES
rootname_runno_raw.fits
rootname_runno_ima.#.fits
ILLUSTRATION
In this and the following illustrations, tasks in yellow boxes denote those that will be used for QuickLook data reduction.
task name | DR kwords read | QuickLook initial value | other kwords read | kwords updated | output files |
stack_read | _raw.fits | ||||
zerosub | zoffcorr | perform | zoffdone,samptime | ||
badmask | maskfile | " " | maskdone | ||
maskcorr | omit | ||||
mkerr | noisfile | " " | noisdone | ||
noiscalc | omit | ||||
mktime | timecalc | perform | timedone | ||
lincorr | linfile | " " | lindone | ||
lincorr | omit | ||||
ctrate | unitcorr | perform | samptime | unitdone,bunit | |
cosmic | cridcalc | perform | criddone | _cr.fits |
DESCRIPTION
The inputs to docirpass are a root name, and run number and a date. The raw data files output by PixCel have the format rootname_runno_readno.fits. The raw data directory is in /data1/cirpass/date. docirpass can output a MEF FITS file containing the stacked raw data, this is called rootname_runno_raw.fits. docirpass always outputs a MEF FITS file containing the reduced data, this is called rootname_runno_ima.#.fits where # is a number which is automatically incremented to allow several reduced versions of the same data.
docirpass checks if the _raw file already exists. If it does
then no stacking is carried out and the existing _raw file is used. If
_raw does not exist then stack_read is called to create a _raw
file from the raw data. The _raw file is a multi-extension FITS (MEF)
file where each read is one extension with extension name SCI.
Associated with each science extension are a variance extension,
a data quality extension and a time extension.
These four extensions are known as an imset.
stack_read creates
these extensions but does not populate them.
The structure of the _raw file is for the first read to be in the
last science extension and the final read to be in the first science
extension. If an observation contained 2 reads then the structure of
the resulting _raw file would be
primary header
SCI 1 - final read
VAR 1
DQ 1
TIME 1
SCI 2 - first read
VAR 2
DQ 2
TIME 2
docirpass reads the data reduction keyword values from the primary header of the _raw file.
If zoffcorr = perform, zerosub subtracts the zeroth read in the _raw file off all the other science extensions, and sets zoffdone = performed.
If maskcorr = perform, badmask reads the bad pixel mask file maskfile, populates the data quality image of each imset and sets maskdone to performed.
If noiscalc = perform, mkerr reads a file noisfile containing the read noise value at each pixel, populates the variance extension of each imset and sets noisdone = performed.
If lincorr = perform, lincorr reads the non-linearity correction file linfile, corrects each pixel for non-linearity and set lindone = performed.
If unitcorr = perform, ctrate divides each science image by the exposure time and sets unitdone = performed.
If cridcalc = perform, cosmic performs cosmic
ray rejection.
cosmic checks if the following conditions are true:
- bunit=counts/s
- nextend/4>=2
If either of these conditions are false then an error message is printed and
criddone=skipped.
The differences in total fibre throughput are found by taking sky flats. The illumination of the IFU by the sky is assumed to be flat. However the number of counts per pixel in the sky flat spectra will be too low to allow the fibre transmission variations to be calculated from these spectra. As when creating the dome flat, the thermal background and dark current have to be removed from the sky flats. This is achieved by taking two sky flat observations, one of the bright sky and one of the faint sky. Subtracting the faint sky observation from the bright sky observation then removes the thermal background and dark current and still leaves a sky signal.
PARAMETERS
ILLUSTRATION
task name | DR kwords read | QuickLook initial value | other kwords read | kwords updated | output files |
bkgsub | bkgfile | lampoff_runno_cr.fits | bkgdone | dflat_runno.fits | |
bkgcorr | perform | ||||
tramline | spectra positions in database |
PARAMETERS
ILLUSTRATION
task name | DR kwords read | QuickLook initial value | other kwords read | kwords updated | output files |
bkgsub | bkgfile | lampoff_runno_cr.fits | bkgdone | ||
bkgcorr | perform | ||||
extract | extrfile | dflat_runno.fits | |||
wvcal | wvfile | " " | ctype,crval,crpix,cdelt,wvcldone | ||
dowvcal | omit | ||||
wvalign | dowvalgn | omit | wvaldone | ||
normspec | |||||
combspec | |||||
fitfunc | |||||
divspec | _norm.fits |
PARAMETERS
ILLUSTRATION
task name | DR kwords read | QuickLook initial value | other kwords read | kwords updated | output files |
bkgsub | bkgfile | faintskyflat_runno_cr.fits | bkgdone | ||
bkgcorr | perform | ||||
extract | extrfile | dflat_runno.fits | |||
divspec | _norm.fits | ||||
sumint | |||||
normint | write normalised throughput values from _norm.fits to database |
Find the final flat field from a dome flat and a sky flat observation
PARAMETERS
ILLUSTRATION
task name | DR kwords read | QuickLook initial value | other kwords read | kwords updated | output files |
multspec | domerunnoskyrunno_flat |
PARAMETERS
ILLUSTRATION
task name | DR kwords read | QuickLook initial value | other kwords read | kwords updated | output files |
extract | extrfile | dflat_runno.fits | |||
identify | write wavelength calibration for arc_runno_cr.fits to the database |
dotarget reduces target object data. This script takes the ME FITS file output by docirpass and outputs an x,y,lambda data cube.
PARAMETERS
ILLUSTRATION
task name | DR kwords read | QuickLook initial value | other kwords read | kwords updated | output files | |
bkgsub | bkgfile | " " | bkgdone | |||
bkgcorr | perform | |||||
extract | extrfile | " " | ||||
wvcal | wvfile | " " | ctype,crval,crpix,cdelt,wvcldone | |||
dowvcal | omit | |||||
wvalign | dowvalgn | omit | wvaldone | |||
flatfield | flatfile | " " | flatdone | |||
flatcorr | omit | |||||
fluxcal | calfile | " " | caldone | |||
doflcal | omit | |||||
mkcube | x,y,lambda datacube |
If bkgcorr = perform, bkgsub subtracts the file given
in bkgfile.
bkgcorr checks if the following conditions are true:
- bkgfile exists
- bkgfile and input file have the same number of extensions.
- bkgfile and input file have the same value of bunit.
- If bunit=counts then background file and object file have the
same samptime value in the corresponding science extensions.
If any of these conditions are false then an error message is printed and
bkgdone=skipped.
extract extracts the individual
spectra. The parameters to use to extract the spectra are found from
extrfile. extract looks for the parameters from this
file in the database given in dbfile.
extract checks if the following conditions are true:
- the parameters found using extrfile are in the database dbfile
If this condition is false then an error message is printed and extrdone=skipped.
- If the input file is a ME FITS file then it only contains one science extension.
If this condition is false then a warning if printed and the spectra are
extracted from [sci,1] only.
If dowvcal = perform then wvcal wavelength calibrates the
individual spectra. The parameters to use for wavelength calibration are
determined from the file wvfile. wvcal looks for the
parameters from this file in the database given in dbfile.
wvcal checks if the following conditions are true:
- extrdone = performed.
- the parameters found using wvfile are in the database dbfile
If either of these conditions are false then an error message is printed and wvcldone=skipped.
If dowvalgn = perform then wvalign aligns all the extracted
spectra to have the same wavelength at each pixel. Note: may already have
done this to extract the spectra.
wvalign checks if the following conditions are true:
- wvdone = performed.
If this condition is false then an error message is printed and
wvaldone = skipped.
If flatcorr = perform then flatfield flatfields all the
extracted spectra using the flatfield spectra in flatfile.
flatfield checks if the following conditions are true:
-extrdone = performed
-the number of extracted spectra in the input and flatfield files are
the same.
If either of these conditions are false then an error message is printed and
wvaldone = skipped.
-the values of wvcldone and wvaldone in the input and
flatfield file are the same.
If this condition is false then a warning message is printed.
If doflcal = perform then the extracted spectra are flux
calibrated using the flux calibration parameters derived from calfile.
fluxcal looks in the database file dbfile for these
flux calibration parameters.
fluxcal checks if the following conditions are true:
-extrdone = performed
-the flux calibration parameters derived from calfile exist
in the database.
-the values of ctype1 in the input file and
in the flux calibration parameters are the same (ie wavelength units or pixels).
task name | description | language | useful existing tasks | equivalent NICMOS flag |
---|---|---|---|---|
assemble | PixCel assemble quadrants into one image after each read | PixCel program | ||
unfit | PixCel remove column shift problem | PixCel program | ||
stack_read | assemble individual FITS files output by PixCel for one observation (eg several loops) into a data cube. Add error, data quality and time arrays. | |||
zerosub | subtract zeroth read from each read | cl script | imarith | zoffcorr |
badmask | populate data quality array | cl script? | maskcorr | |
mkerr | populate error array for each science image | C program? | noiscalc | |
mktime | populate time array for each science image | C program? | ||
lincorr | do linearization correction step for each pixel | C program | nlincorr | |
ctrate | turn each science array into cts/s | C program? | unitcorr | |
cosmic | reject cosmic rays using intermediate reads and make final image | C program | cridcalc | |
bkgsub | subtract background frame from object frame | cl script | imarith | |
tramline | find positions of spectra on detector | C program | ||
extract | extract spectra | C program | ||
wvcal | wavelength calibrate each extracted spectrum | cl script | ||
wvalign | resample and align wavelength calibrated to have the same wavelength at each pixel number. | |||
normspec | normalise a list of spectra by dividing each spectrum by its mean | cl script | imarith | |
combspec | combine 1D spectra by finding the mean or median of all the input spectra at each pixel | cl script | ||
fitfunc | fit high order function to a spectrum | cl script | ||
divspec | divide list of spectra in one image by a single spectrum, or by a list of spectra from the same or another image. | cl script | imarith | |
sumint | find total intensity for each spectrum in input list | cl script | ||
normint | for a list of total intensities, find the mean total intensity, and normalize each intensity by this mean | cl script | imarith | |
multspec | multiply 1D spectra by either a constant or a list of constants (one for each spectrum) | cl script | imarith | |
flatfield | divide each extracted spectrum by corresponding final flat field spectrum | cl script | imarith | |
fluxcal | flux calibrate each extracted spectrum | cl script |
stack_read | |
---|---|
filelist | list of FITS files |
outfile | output file |
stack_read takes all the FITS files in filelist and places them in a single multi-extension FITS file. The output FITS file is automatically named rootname_1strun#_lastrun#.fits, where rootname is FITS_file and 1strun# and lastrun# are the lowest and highest nos in run_nos. stack_readplaces one input FITS file in the science image of each FITS extension. The input FITS file with the lowest run no is placed in the highest extension, the next lowest run no in the next to highest extension etc. For CIRPASS data this means that the final read of an observation will be in the 1st extension. stack_read also adds (but does not populate) error, data quality and time arrays to each FITS extension. |
zerosub | |
---|---|
cubename | name of data cube |
FITS header keywords | |
reads | ZOFFCORR |
updates | ZOFFDONE |
CIRPASS does not have a shutter and so to begin an observation the array is reset and then directly read (so the integration time for this read is zero). This read is called the zeroth read and contains the bias level for that observation. Subtracting the zeroth read from all the other science images in the FITS data cube therefore removes the bias level from the images. |
badmask | |
---|---|
cubename | name of data cube |
FITS header keywords | |
reads | MASKFILE, MASKCORR |
updates | MASKDONE |
Populates the data quality arrays in the FITS data cube. Reads the positions of bad pixels from the bad pixel mask file and, for each FITS extension, sets the value of that pixel in the data quality array. |
mkerr | |
---|---|
cubename | name of data cube |
FITS header keywords | |
reads | NOISFILE, NOISCALC | updates | NOISDONE |
Populates the error arrays in the FITS data cube. For each extension, uses values of gain and read noise read from the FITS header and the value of each pixel in the science image to calculate the error for that pixel. |
mktime | |
---|---|
cubename | name of data cube |
FITS header keywords | |
reads | INTTIME, TIMECALC |
updates | TIMEDONE |
Populates the time arrays in the FITS data cube. For each extension, finds the value of the integration time for that science image from the FITS header, and places that in the time array. |
lincorr | |
---|---|
cubename | name of datacube |
FITS header keywords | |
reads | LINFILE, LINCORR |
updates | LINDONE |
The increase of counts with time in the CIRPASS data may be non-linear above a certain count level. The amount of non-linearity may be different for each pixel. If non-linearity is a problem then we will produce a linearity correction file which will contains the required coefficients for each pixel to correct the non-linear behaviour. This task will read this file and do the correction. |
ctrate | |
---|---|
cubename | name of datacube |
FITS header keywords | |
reads | INTTIME |
reads | UNITCORR |
updates | UNITDONE |
Convert each science image from counts to counts/s. |
cosmic | |
---|---|
cubename | name of zeroth read subtracted data cube |
threshold | threshold value for cosmic ray rejection |
FITS header keywords | |
reads | CRIDCALC |
updates | CRIDDONE |
This task removes cosmic rays from the FITS data cube containing several imsets and produces an image which contains one imset. The cosmic rays are rejected by comparing the values in a pixel at each read and rejecting those reads where the pixel values are > threshold*sigma. sigma is calculated from the error array. The output science image is in units of counts/s. |
bkgsub | |
---|---|
objname | name of cosmic-ray subtracted object observation |
bkgname | name of cosmic-ray subtracted background observation |
Subtract the background observation from the object observation. Each observation should consist of a single imset. The output error array is formed from combining the error arrays of the two input images. The output data quality array is formed by merging the data quality arrays of the two input images. The output time array is a copy of the input object time array. |
tramline | |
---|---|
imname | name of image to use to trace spectra |
template | output template file containing positions of spectra |
Create the spectral position template that will be used to extract the object spectra. The image used to trace the spectra should have a high signal-to-noise (eg a dome flat). The positions of the spectra are output into a template file, which is then used as input to the task extract. Pixels which have a data quality flag set in the data quality array are not used in the spectral tracing. |
extract | |
---|---|
imname | name of 2D image containing object spectra |
algo | algorithm to use to extract spectra |
FITS header keywords | |
template | name of file output by tramline task |
Extract 1D spectra from a 2D image. The positions of the spectra are given in the template file output by the tramline task. The name of this file is given in the image header. |
wvcal | |
---|---|
spec | list of 1D spectra to be wavelength calibrated |
wvfile | file containing wavelength calibration to be applied to each spectrum |
Wavelength calibrate a list of 1D spectra using a wavelength calibration file. |
wvalign | |
---|---|
speclist | list of 1D wavelength calibrated spectra to be aligned |
Align all the wavelength calibrated spectra in a file so that the same wavelengths lie at the same pixel numbers. Will probably have to resample the spectra to have a linear dispersion value first. Then find the wavelength at a given pixel for each spectrum (from the header information), find the mean wavelength at this pixel, and shift each spectrum to have the mean wavelength at this pixel. |
normspec | |
---|---|
imname | name of file containing 1D spectra to be normalised |
Normalize each 1D spectrum by its mean. |
combspec | |
---|---|
speclist | list of 1D spectra to be combined |
Combine a list of 1D spectra by finding the mean or median at each pixel. The input list should specify which apertures of each spectrum should be combined. If no aperture list is given then it is assumed that all the apertures in a file should be combined. |
fitfunc | |
---|---|
specname | Spectrum to be fit |
functype | Type of function to be fit |
funcorder | Order of function to be fit |
Fit a high order function to a spectrum |
divspec | |
---|---|
speclist | list 1D spectra |
specone | name of file containing single spectrum to be divided into all spectra in spec1 |
Divide each spectrum in a list of 1D spectra by a single 1D spectrum. |
sumint | |
---|---|
speclist | list of 1D spectra to be summed |
Find the total intensity in each 1D spectrum from a list of 1D spectra. |
normint | |
---|---|
intfile | file containing list of total intensities |
Find the mean of a list of numbers. |
multspec | |
---|---|
speclist | list of 1D spectra |
mult | constant or list of constants to multiply spectra by |
Multiply a each 1D spectrum in speclist by either a constant or a list of constants. If there is one constant then all the spectra in the list will be multiplied by this constant. If there is more than one constant then each spectrum in the list will be paired with the constant in the same position in the constant list, and multiplied by it. If there are fewer constants than spectra then, when the end of the constant list is reached, the final constant in the list will be used to multiply the remaining spectra. |
flatfield | |
---|---|
speclist | list of 1D spectra |
flatspec | list of 1D flat field spectra |
Flatfield the extracted spectra by dividing each extracted 1D spectrum by the corresponding 1D final flat field spectrum. |
fluxcal | |
---|---|
speclist | list of 1D spectra to be flux calibrated |
fluxspec | flux calibration spectrum to be applied to each 1D spectrum. |
Flux calibrate a list of 1D spectra using a flux calibration spectrum. The units of the flux calibration spectrum are fluxunits/counts and the units of the input 1D spectra are counts. Multiplying the input 1D spectrum by the flux calibration spectrum therefore gives an output spectrum calibrated in fluxunits. |