Data reduction package

Current Worries

- Is it possible to have a single database file that stores all the calibration information (eg spectra positions, flux calibration, wavelength calibration), or do I need a different database file for each different type of calibration data ?
- I think that, because the standard star is observed through a single fibre, and then this calibration is applied to all fibres, you should wavelength calibrate before flat fielding and flux calibration. This means that you should wavelength calibrate the dome flats before finding the normalised flat field spectra. But I haven't managed to totally convince myself (and others) yet.

Overview

The data reduction package contains the tasks required to reduce the calibration and target data. It is assumed that PixCel outputs a FITS file for each read, which contains the four quadrants assembled into one image. PixCel can read an external FITS header fragment which means that the FITS files output from PixCel will have all the required header keywords (though not all with the correct values). The final data product for the target data is an x,y,lambda data cube, which can be visualised as a cube where the z-axis is wavelength and each plane is a picture of what the IFU observed at that wavelength.

The first version of the data reduction package will deal separately with the different types of data (eg dome flats, sky flats, arc lamps, flux standards and target observations. For each type of data there are one or more pipeline scripts to reduce the data. Each pipeline script runs a series of IRAF tasks. Each IRAF task can also run stand-alone.

Some tasks are always called by the pipeline scripts, others are called depending upon the value of FITS header keywords. This enables 'QuickLook' data reduction at the telescope to run a certain subset of the possible tasks in each pipeline. The initial values of the FITS header keywords which drive the data reduction are set by the user at the start of an observation through the instrument control software. There is a task to allow the user to easily change the values of these keywords is required.

Below is an illustration of the order of the pipeline tasks that would be run to produce a QuickLook data reduction (ie not wavelength or flux calibrated) from a beamswitched pair.

Make illustration for here

Pipeline tasks

Below is a description of each of the main pipeline scripts in the CIRPASS data reduction package. For each pipeline script there is a description and a figure showing which of the IRAF tasks are used. There is a link from each IRAF task to its description.

Pipeline task summary

Pipeline task nameBrief description
docirpassbasic processsing required by all observations
findspecpostrace the spectra and write positions to database file. extract the spectra.
dodomeflatmake final dome flats
doskyflatfind normalised throughput values for each fibre
dofinalflatmake final flat field from final dome flat and normalised throughput values
doarcfind wavelength calibration from arc observation
dotargetproduce x,y,lambda data cube for target observation (could be object, sky, flux std)

docirpass

Most types of images need to undergo the same basic processing so these tasks are grouped together in a pipeline called docirpass.

PARAMETERS

Root name = ""
Root name of observation
run_no = ""
Observation run no
Date = "" (yyyymmdd)
Date of observation

OUTPUT FILES
rootname_runno_raw.fits
rootname_runno_ima.#.fits

ILLUSTRATION

In this and the following illustrations, tasks in yellow boxes denote those that will be used for QuickLook data reduction.

task name DR kwords read QuickLook initial value other kwords read kwords updated output files
stack_read



_raw.fits
zerosubzoffcorrperform
zoffdone,samptime
badmaskmaskfile" "
maskdone
maskcorromit


mkerrnoisfile" "
noisdone
noiscalcomit


mktimetimecalcperform
timedone
lincorrlinfile" "
lindone
lincorromit


ctrateunitcorrperformsamptimeunitdone,bunit
cosmiccridcalcperform
criddone_cr.fits

DESCRIPTION

The inputs to docirpass are a root name, and run number and a date. The raw data files output by PixCel have the format rootname_runno_readno.fits. The raw data directory is in /data1/cirpass/date. docirpass can output a MEF FITS file containing the stacked raw data, this is called rootname_runno_raw.fits. docirpass always outputs a MEF FITS file containing the reduced data, this is called rootname_runno_ima.#.fits where # is a number which is automatically incremented to allow several reduced versions of the same data.

docirpass checks if the _raw file already exists. If it does then no stacking is carried out and the existing _raw file is used. If _raw does not exist then stack_read is called to create a _raw file from the raw data. The _raw file is a multi-extension FITS (MEF) file where each read is one extension with extension name SCI. Associated with each science extension are a variance extension, a data quality extension and a time extension. These four extensions are known as an imset. stack_read creates these extensions but does not populate them. The structure of the _raw file is for the first read to be in the last science extension and the final read to be in the first science extension. If an observation contained 2 reads then the structure of the resulting _raw file would be
primary header
SCI 1 - final read
VAR 1
DQ 1
TIME 1
SCI 2 - first read
VAR 2
DQ 2
TIME 2

docirpass reads the data reduction keyword values from the primary header of the _raw file.

If zoffcorr = perform, zerosub subtracts the zeroth read in the _raw file off all the other science extensions, and sets zoffdone = performed.

If maskcorr = perform, badmask reads the bad pixel mask file maskfile, populates the data quality image of each imset and sets maskdone to performed.

If noiscalc = perform, mkerr reads a file noisfile containing the read noise value at each pixel, populates the variance extension of each imset and sets noisdone = performed.

If lincorr = perform, lincorr reads the non-linearity correction file linfile, corrects each pixel for non-linearity and set lindone = performed.

If unitcorr = perform, ctrate divides each science image by the exposure time and sets unitdone = performed.

If cridcalc = perform, cosmic performs cosmic ray rejection.
cosmic checks if the following conditions are true:
- bunit=counts/s
- nextend/4>=2
If either of these conditions are false then an error message is printed and criddone=skipped.

Flat fields

The contributors to the flat field are the pixel-pixel sensitivity variation, differences in fibre transmission profiles with wavelength, and differences in the total fibre throughput. The pixel-pixel sensitivity variations can be removed by a flat field made by uniformly illuminating the detector not through the fibres, or by illuminating the detector by moving the fibre slit along the detector in the spatial direction. Here, however, we are assuming that differences in fibre transmission profiles are important. In this case the method described below to remove these differences also removes differences due to pixel-pixel sensitivity variations, and so the pixel-pixel flat is not required. If the fibre transmission profiles are different then a dome flat can be used to remove these differences.

The differences in total fibre throughput are found by taking sky flats. The illumination of the IFU by the sky is assumed to be flat. However the number of counts per pixel in the sky flat spectra will be too low to allow the fibre transmission variations to be calculated from these spectra. As when creating the dome flat, the thermal background and dark current have to be removed from the sky flats. This is achieved by taking two sky flat observations, one of the bright sky and one of the faint sky. Subtracting the faint sky observation from the bright sky observation then removes the thermal background and dark current and still leaves a sky signal.

findspecpos

Uses dome flat observations to find the positions of the spectra on the detector.

PARAMETERS

lampon
name of ME FITS file containing lamp on observations processed through basic_proc
lampoff
name of ME FITS file containing lamp off observations processed through basic_proc

ILLUSTRATION

task name DR kwords read QuickLook initial value other kwords read kwords updated output files
bkgsubbkgfilelampoff_runno_cr.fits
bkgdonedflat_runno.fits
bkgcorrperform


tramline



spectra positions in database

dodomeflat

PARAMETERS

lampon
name of ME FITS file containing lamp on observations processed through basic_proc
lampoff
name of ME FITS file containing lamp off observations processed through basic_proc

ILLUSTRATION

task name DR kwords read QuickLook initial value other kwords read kwords updated output files
bkgsubbkgfilelampoff_runno_cr.fits
bkgdone
bkgcorrperform


extractextrfiledflat_runno.fits


wvcalwvfile" "
ctype,crval,crpix,cdelt,wvcldone
dowvcalomit


wvaligndowvalgnomit
wvaldone
normspec




combspec




fitfunc




divspec



_norm.fits

doskyflat

Find the normalised fibre throughput values from a bright and a faint sky flat observation.

PARAMETERS

lampon
name of ME FITS file containing bright sky observations processed through basic_proc
lampoff
name of ME FITS file containing faint sky observations processed through basic_proc

ILLUSTRATION

task name DR kwords read QuickLook initial value other kwords read kwords updated output files
bkgsubbkgfilefaintskyflat_runno_cr.fits
bkgdone
bkgcorrperform


extractextrfiledflat_runno.fits


divspec



_norm.fits
sumint




normint



write normalised throughput values from _norm.fits to database

dofinalflat

Find the final flat field from a dome flat and a sky flat observation

PARAMETERS

domeflat
Name of dome flat processed through dodomeflat (_norm.fits)
skyflat
Name of sky flat processed through doskyflat

ILLUSTRATION

task name DR kwords read QuickLook initial value other kwords read kwords updated output files
multspec



domerunnoskyrunno_flat

Wavelength calibration

doarc

Find the wavelength calibration from an arc lamp observation processed through docirpass.

PARAMETERS

FITS_file
Name of ME FITS file containing arc observations processed through bsaicproc

ILLUSTRATION

task name DR kwords read QuickLook initial value other kwords read kwords updated output files
extractextrfiledflat_runno.fits


identify



write wavelength calibration for arc_runno_cr.fits to the database

Target data (object, sky, standard stars)

dotarget

dotarget reduces target object data. This script takes the ME FITS file output by docirpass and outputs an x,y,lambda data cube.

PARAMETERS

FITS_file = ""
name of ME FITS file(s)

ILLUSTRATION

task name DR kwords read QuickLook initial value other kwords read kwords updated output files
bkgsubbkgfile" "
bkgdone
bkgcorrperform


extractextrfile" "


wvcalwvfile" "
ctype,crval,crpix,cdelt,wvcldone
dowvcalomit


wvaligndowvalgnomit
wvaldone
flatfieldflatfile" "
flatdone
flatcorromit


fluxcalcalfile" "
caldone

doflcalomit


mkcube



x,y,lambda datacube

If bkgcorr = perform, bkgsub subtracts the file given in bkgfile.
bkgcorr checks if the following conditions are true:
- bkgfile exists
- bkgfile and input file have the same number of extensions.
- bkgfile and input file have the same value of bunit.
- If bunit=counts then background file and object file have the same samptime value in the corresponding science extensions.
If any of these conditions are false then an error message is printed and bkgdone=skipped.

extract extracts the individual spectra. The parameters to use to extract the spectra are found from extrfile. extract looks for the parameters from this file in the database given in dbfile.
extract checks if the following conditions are true:
- the parameters found using extrfile are in the database dbfile
If this condition is false then an error message is printed and extrdone=skipped.
- If the input file is a ME FITS file then it only contains one science extension.
If this condition is false then a warning if printed and the spectra are extracted from [sci,1] only.

If dowvcal = perform then wvcal wavelength calibrates the individual spectra. The parameters to use for wavelength calibration are determined from the file wvfile. wvcal looks for the parameters from this file in the database given in dbfile.
wvcal checks if the following conditions are true:
- extrdone = performed.
- the parameters found using wvfile are in the database dbfile
If either of these conditions are false then an error message is printed and wvcldone=skipped.

If dowvalgn = perform then wvalign aligns all the extracted spectra to have the same wavelength at each pixel. Note: may already have done this to extract the spectra.
wvalign checks if the following conditions are true:
- wvdone = performed.
If this condition is false then an error message is printed and wvaldone = skipped.

If flatcorr = perform then flatfield flatfields all the extracted spectra using the flatfield spectra in flatfile.
flatfield checks if the following conditions are true:
-extrdone = performed
-the number of extracted spectra in the input and flatfield files are the same.
If either of these conditions are false then an error message is printed and wvaldone = skipped.
-the values of wvcldone and wvaldone in the input and flatfield file are the same.
If this condition is false then a warning message is printed.

If doflcal = perform then the extracted spectra are flux calibrated using the flux calibration parameters derived from calfile. fluxcal looks in the database file dbfile for these flux calibration parameters.
fluxcal checks if the following conditions are true:
-extrdone = performed
-the flux calibration parameters derived from calfile exist in the database.
-the values of ctype1 in the input file and in the flux calibration parameters are the same (ie wavelength units or pixels).

Task Summary

task name description language useful existing tasks equivalent NICMOS flag
assemblePixCel assemble quadrants into one image after each readPixCel program
unfitPixCel remove column shift problemPixCel program
stack_read assemble individual FITS files output by PixCel for one observation (eg several loops) into a data cube. Add error, data quality and time arrays.
zerosubsubtract zeroth read from each readcl scriptimarithzoffcorr
badmaskpopulate data quality arraycl script?maskcorr
mkerrpopulate error array for each science imageC program?noiscalc
mktimepopulate time array for each science imageC program?
lincorrdo linearization correction step for each pixelC programnlincorr
ctrateturn each science array into cts/sC program?unitcorr
cosmicreject cosmic rays using intermediate reads and make final imageC programcridcalc
bkgsubsubtract background frame from object framecl scriptimarith
tramlinefind positions of spectra on detectorC program
extractextract spectraC program
wvcalwavelength calibrate each extracted spectrumcl script
wvalignresample and align wavelength calibrated to have the same wavelength at each pixel number.
normspecnormalise a list of spectra by dividing each spectrum by its meancl scriptimarith
combspeccombine 1D spectra by finding the mean or median of all the input spectra at each pixelcl script
fitfuncfit high order function to a spectrumcl script
divspecdivide list of spectra in one image by a single spectrum, or by a list of spectra from the same or another image.cl scriptimarith
sumintfind total intensity for each spectrum in input listcl script
normintfor a list of total intensities, find the mean total intensity, and normalize each intensity by this meancl scriptimarith
multspecmultiply 1D spectra by either a constant or a list of constants (one for each spectrum)cl scriptimarith
flatfielddivide each extracted spectrum by corresponding final flat field spectrumcl scriptimarith
fluxcalflux calibrate each extracted spectrumcl script

Task details

Below is a brief description of each task, the input parameters required, and the FITS header keywords that are read and/or updated by the task.
stack_read
filelist list of FITS files
outfile output file

stack_read takes all the FITS files in filelist and places them in a single multi-extension FITS file. The output FITS file is automatically named rootname_1strun#_lastrun#.fits, where rootname is FITS_file and 1strun# and lastrun# are the lowest and highest nos in run_nos. stack_readplaces one input FITS file in the science image of each FITS extension. The input FITS file with the lowest run no is placed in the highest extension, the next lowest run no in the next to highest extension etc. For CIRPASS data this means that the final read of an observation will be in the 1st extension. stack_read also adds (but does not populate) error, data quality and time arrays to each FITS extension.

zerosub
cubename name of data cube
FITS header keywords
readsZOFFCORR
updatesZOFFDONE
CIRPASS does not have a shutter and so to begin an observation the array is reset and then directly read (so the integration time for this read is zero). This read is called the zeroth read and contains the bias level for that observation. Subtracting the zeroth read from all the other science images in the FITS data cube therefore removes the bias level from the images.

badmask
cubename name of data cube
FITS header keywords
readsMASKFILE, MASKCORR
updatesMASKDONE
Populates the data quality arrays in the FITS data cube. Reads the positions of bad pixels from the bad pixel mask file and, for each FITS extension, sets the value of that pixel in the data quality array.

mkerr
cubename name of data cube
FITS header keywords
readsNOISFILE, NOISCALC
updatesNOISDONE
Populates the error arrays in the FITS data cube. For each extension, uses values of gain and read noise read from the FITS header and the value of each pixel in the science image to calculate the error for that pixel.

mktime
cubename name of data cube
FITS header keywords
readsINTTIME, TIMECALC
updatesTIMEDONE
Populates the time arrays in the FITS data cube. For each extension, finds the value of the integration time for that science image from the FITS header, and places that in the time array.

lincorr
cubename name of datacube
FITS header keywords
readsLINFILE, LINCORR
updatesLINDONE
The increase of counts with time in the CIRPASS data may be non-linear above a certain count level. The amount of non-linearity may be different for each pixel. If non-linearity is a problem then we will produce a linearity correction file which will contains the required coefficients for each pixel to correct the non-linear behaviour. This task will read this file and do the correction.

ctrate
cubename name of datacube
FITS header keywords
readsINTTIME
readsUNITCORR
updatesUNITDONE
Convert each science image from counts to counts/s.

cosmic
cubename name of zeroth read subtracted data cube
threshold threshold value for cosmic ray rejection
FITS header keywords
readsCRIDCALC
updatesCRIDDONE
This task removes cosmic rays from the FITS data cube containing several imsets and produces an image which contains one imset. The cosmic rays are rejected by comparing the values in a pixel at each read and rejecting those reads where the pixel values are > threshold*sigma. sigma is calculated from the error array. The output science image is in units of counts/s.

bkgsub
objname name of cosmic-ray subtracted object observation
bkgname name of cosmic-ray subtracted background observation
Subtract the background observation from the object observation. Each observation should consist of a single imset. The output error array is formed from combining the error arrays of the two input images. The output data quality array is formed by merging the data quality arrays of the two input images. The output time array is a copy of the input object time array.

tramline
imname name of image to use to trace spectra
template output template file containing positions of spectra
Create the spectral position template that will be used to extract the object spectra. The image used to trace the spectra should have a high signal-to-noise (eg a dome flat). The positions of the spectra are output into a template file, which is then used as input to the task extract. Pixels which have a data quality flag set in the data quality array are not used in the spectral tracing.

extract
imname name of 2D image containing object spectra
algo algorithm to use to extract spectra
FITS header keywords
template name of file output by tramline task
Extract 1D spectra from a 2D image. The positions of the spectra are given in the template file output by the tramline task. The name of this file is given in the image header.

wvcal
spec list of 1D spectra to be wavelength calibrated
wvfile file containing wavelength calibration to be applied to each spectrum
Wavelength calibrate a list of 1D spectra using a wavelength calibration file.

wvalign
speclist list of 1D wavelength calibrated spectra to be aligned
Align all the wavelength calibrated spectra in a file so that the same wavelengths lie at the same pixel numbers. Will probably have to resample the spectra to have a linear dispersion value first. Then find the wavelength at a given pixel for each spectrum (from the header information), find the mean wavelength at this pixel, and shift each spectrum to have the mean wavelength at this pixel.

normspec
imname name of file containing 1D spectra to be normalised
Normalize each 1D spectrum by its mean.

combspec
speclist list of 1D spectra to be combined
Combine a list of 1D spectra by finding the mean or median at each pixel. The input list should specify which apertures of each spectrum should be combined. If no aperture list is given then it is assumed that all the apertures in a file should be combined.

fitfunc
specname Spectrum to be fit
functype Type of function to be fit
funcorder Order of function to be fit
Fit a high order function to a spectrum

divspec
speclist list 1D spectra
specone name of file containing single spectrum to be divided into all spectra in spec1
Divide each spectrum in a list of 1D spectra by a single 1D spectrum.

sumint
speclist list of 1D spectra to be summed
Find the total intensity in each 1D spectrum from a list of 1D spectra.

normint
intfile file containing list of total intensities
Find the mean of a list of numbers.

multspec
speclist list of 1D spectra
mult constant or list of constants to multiply spectra by
Multiply a each 1D spectrum in speclist by either a constant or a list of constants. If there is one constant then all the spectra in the list will be multiplied by this constant. If there is more than one constant then each spectrum in the list will be paired with the constant in the same position in the constant list, and multiplied by it. If there are fewer constants than spectra then, when the end of the constant list is reached, the final constant in the list will be used to multiply the remaining spectra.

flatfield
speclist list of 1D spectra
flatspec list of 1D flat field spectra
Flatfield the extracted spectra by dividing each extracted 1D spectrum by the corresponding 1D final flat field spectrum.

fluxcal
speclist list of 1D spectra to be flux calibrated
fluxspec flux calibration spectrum to be applied to each 1D spectrum.
Flux calibrate a list of 1D spectra using a flux calibration spectrum. The units of the flux calibration spectrum are fluxunits/counts and the units of the input 1D spectra are counts. Multiplying the input 1D spectrum by the flux calibration spectrum therefore gives an output spectrum calibrated in fluxunits.


Rachel Johnson <raj@ast.cam.ac.uk>
Last modified: Tue Aug 22 23:31:04 2000