Sky Subtraction With UFTI Data

Document number VDF-TRE-IOA-00008-0002

Jim Lewis


In preparation for the flow of WFCAM data we are examining several ways of removing background emission from IR data. The sky background can be affected by several processes:

In the absence of any real WFCAM data, we cannot say how that data will be affected (if at all) by any of these. The first two are very instrument and telescope specific. UKIRT's current imager (UFTI) does suffer from both of these and it can be instructive to look at various means of removing them from UFTI data.

The third is site specific and if it exists will be the most problematic. The irony is that the combined temporal/spatial variation of the sky background over the WFCAM field of view is the one effect that we are unable to test. UFTI has such a small field of view that it only covers 1% of the total area of a single WFCAM detector and hence we are unable to measure the variation over the spatial scales that we need.

In a previous paper we reduced a set of UFTI service data. For sky removal we simply combined all images from a given jitter sequence and subtracted the normalised result. Although the jitter-combined images showed no signs of real background residuals, we did not examine the sky subtraction on individual frames in any detail. With this in mind we have revisited this issue using a variant of our normal defringing algorithm.

Data Reduction

The data used in the following tests were the nights 20030923, 20031008 and 20031120 on which whole nights of service data were taken to be used for flat field and sky subtraction algorithm testing. 20030923 is a night of data taken in the H band only. The other two were both nights of K band observations. The nights each consist of at least four sets of exposures of test regions, with as many as 50 jitter positions in some cases.

For the sky subtraction tests we first need to get the data to the point where sky subtraction is necessary. The following steps were used in getting the data to this level:

The main structures in the background in the H band are two sets of well defined fringes. One set is fairly broad and feature mainly in the lower two quadrants. The other set appear as cocentric rings centred in the upper right hand quadrant.

H sky from Group 6 20030923

The K band sky exposures also have the cocentric ring structure (now centred somewhere in the lower right hand quadrant). What is different here from the H band is the huge number of slightly defocused sources, which are caused by thermal emission from dust on the cryostat window.

K sky from Group 99 20031003

In a previous paper I demonstrated that removing this sort of structure in non-crowded regions could be accomplished very satisfactorily by a simple subtraction of the "mean" sky frame from a particular jitter sequence, from each frame in that jitter. Difficulties can arise, however, for several reason:

We need to look if it is possible to use sky data from one part of the night to correct for background features in another part of the night. If so, this would solve both above problems,

To address this and the question of the variation of the background components over the course of the night, I first combined all of the mean sky frames for a particular night into one master background frame (this was done with a suitable weighting scheme to compensate for the fact that some of the mean sky frames had much better signal/noise than others). I then scaled this master sky frame to each of the mean sky frames. This was done using our standard fringe scaling algorithm which attempts to robustly minimise the mean absolute deviation of the background of a defringed image. Once the scale factor was found it was a simple matter of scaling the master fringe frame by that factor and subtracting it off. Any residual features should then be obvious.

Results For H Band

Considering the H data first (night 20030923), below are the four mean sky frames before and after defringing with the master sky file.

H Sky Group 6

H Sky Group 69

H Sky Group 177

H Sky Group 236

In all four cases the two main fringe patterns are removed completely. Two significant residuals remain however. The first is a set of fringes that appear sky6 and sky177 and which run nearly vertical. Although these appear to be rather strong it must be pointed out that the maximum peak to trough intensity of the original fringes was between 3 to 4% of sky in these frames. These residual fringes are about 0.2% of sky at the very worst. They are only visible in sky frames 6 and 177, which have a much higher signal to noise ratio that the other two. (Sky 6 and 177 have a total exposure time of 1800s. Sky 69 has 735s and Sky 236 has 125s) In the lower exposure observations these fringes are lost in the sky noise. That these residuals appear to shift across the frame shows some sort of phase difference between them and the main fringe patterns.

The question here is whether it is worth taking any more heroic steps to try and remove these completely, perhaps with some sort of independent component modelling as mentioned above. We have to remind ourselves that what we are looking at here are two frames with a total 1800s of exposure time each. What we are trying to achieve is an optimal way to remove background features from single exposures. Features visible in high signal to noise x-y stacked images (ie. no shifts between images) such as these will be much less prominent in single images and are not generally detectable above sky noise. Furthermore, in stacking the jitter sequence to create astronomical images the implict offsets used cause the fine-grained fringe pattern to disappear completely since the offsets essentially randomize its systematic spatial contribution. Providing the systematics are small compared to the rms photon noise this has negligible impact on the quality of the resulting stacked frames. In this this case, trying to do a second order fringe removal may in fact just increase the sky noise as the scaling factors will have very larger than normal uncertainties.

To illustrate these points let's step back and look at a single frame that has been badly affected by fringing.

H observation 16 after dark correction and flat fielding, but before defringing

H observation 16 after defringing with a sky formed from all observations in group 6 (to which this observation belongs)

H observation 16 after defringing with a sky formed from all observations in the night

The first attempt at defringing removes the background features completely. (It is reassuring to note that the scale factor that the defringing algorithm found to be optimal for this observation was 0.9988). This tells us that using a sky estimate that is derived from the target observations themselves is probably the way to go if we have that option. The second attempt at defringing using a nightly mean sky estimate does still leave small residuals. In this particular frame, the peak to trough difference is about 1.2 counts out of a a mean background of 1037.0, or roughly 0.1%. If this frame were to be used with a set of similarly corrected images to form a dither any such residual structure would be washed out during the dithering stage. Given that there may be situations where you cannot use observations of crowded regions to define a mean sky frame, this shows that it is still possible to get perfectly acceptable results using a nightly mean sky.

The other significant residual is a sort of rippling effect that can be seen in all four defringed mean sky frames which is nearly horizontal. This is a result of using a single dark frame to estimate the dark current and reset anomaly. Below is dark frame f20030923_00235. The ripples are clearly seen and are obviously the cause of the effect seen in the sky frames. This will only be sorted out when a proper dark subtraction is done using many averaged dark frames.

Dark frame for group 235

A dark frame created by averaging nine 40 second dark exposures

Results for K Band

Considering now the K band data of 20031008, below are the four mean sky files from that night before and after defringing with the master sky

K Sky Group 99

K Sky Group 153

K Sky Group 297

K Sky Group 382

Using the defringing algorithm to remove both fringes and thermal emission features leave some residual structure on most of the frames. This is a demonstration of variations in these features that are out of phase with each other. This is not at all surprising as they have completely different physical causes. Sky 9 shows a positive residual dust feature in the upper right hand quadrant, that is lying on a negative fringe residual. The opposite is true of Sky 297. Although again these look bad, the worst peak to trough residual fringe in any of these is less that 0.25% of sky and affect only the very high signal to noise frames in a significant way. In the event that a user wanted to do a second order correction to remove more of the residual features, the fact that the features seem to divide so neatly into just two components would ensure that probably only one further correction would be needed.

Let's step back again and look at a single image as we did in the previous section. Below are images of a K band image that has been badly affected by dust and fringing.

K observation 99 after dark correction and flat fielding, but before defringing

K observation 99 after defringing with a sky formed from all observations in group 99 (to which this observation belongs)

K observation 99 after defringing with a sky formed from all observations in the night

Again, using the nightly mean sky estimate does cause some small residual fringes to be visible in the third image. These are barely detectable however and amount to less than 0.08% of sky. This again tells us that in cases where we cannot use the local observations for a sky estimate, that a nightly mean sky can be used for very little cost. It is interesting that although there appears to be residuals in the fringes, this is not the case for the dust emission. This is likely to be because the dust emission is a much greater contribution to the total background emission to be subtracted. If there is some sort of temporal variation in either effect, then the algorithm is most likely to remove the dust better than the fringes.

Changes In Dust Emission Over Time

Over time dust on the cryostat changes position and the amount varies depending on many factors. Below are two K sky frames -- one from the 20031008 data and one from the 20031120 data.

K Sky Group 99 from 20031008 and K Sky Group 41 from 20031120

The images clearly show that while a large number of the dust particles seen on 20031008 are still present a month and a half later, there are now many new particles, particularly in the lower left hand corner of the frame. The fringe rings seem to be affected very little by the passage of time. Given that the defringing algorithm is more sensitive to the dust emission in K than the rings, this argues (certainly for K) in favour of forming mean sky frames on a nightly basis, rather than trying to carry them over from previous nights.


We have looked at data both in H and K using UFTI. It was important to look at both wavebands as H contains background emission due soley to sky effects and K contains both sky and dust emission, which are part of two different physical processes. From the discussion above I draw the following conclusions:
  1. The best way to rid the observations of both fringe and thermal emission is to use the target data itself to form the mean background frame. Although both of these effects do vary with time, it appears that neither does so to sufficient extent during the time-scale of a single set of observations (here between 30-50 minutes) to cause any residual emission to be present after the correction.
  2. There may be situations where it is impossible to use the data itself to form a mean sky frame. Observations of crowded regions or large extended objects mean that any sky estimate would be contaminated by astronomical objects. It has been suggested that observations of offset sky regions could be used to work around this conwtraint. These tests also prove that a single mean sky frame taken from all exposures during the night can be used to this end. This method does leave some residual background in the individual frames, but at such a low level as to be barely detectable (they only become obvious when they are stacked without jitter offsets, which is something we don't ever do). This would also be smeared out most likely at the jitter stage resulting only in a negligible increase in the background noise.
  3. If nightly sky frames are to be used, it is good practice to try and ensure that mean frames are not carried over from night to night unless it can't be avoided.

Last modified: Thu Feb 26 10:44:22 2004