Notes from UKIDSS meeting 25th November 2002

Updated after CASU-WFAU liaison meeting 9th December 2002 (+NCH, IB and SJW)

Updated after detailed feedback from Steve Warren 23rd January 2003

Updated with new timescales 18th March.


Overall philosophy
------------------

Summit    -  causal pipeline: DQC feedback, first pass science products

Standard  -  nightly pipeline: DQC check, astrometric and initial 
             photometric calibration, standard catalogue parameters, 
             final 2D science images

Further   -  nightly pipeline: PSF estimation and general profile fitting

Database  -  asynchronous processing: stacking, difference imaging,
driven       temporal, matching and colocated list driven stuff 
             (inc. external list drivers), contiguous tiled regions and 
             seamless overlap calibrated catalogues, optimal detection and 
             parameterisation of eg. LSBs (ie. bigger than PSFs)

(note basic --> standard since not such negative connotations and
   advanced --> Database driven since some of the processing not that
                advanced per se)


2D instrumental signature removal
---------------------------------

effects to deal with may include: persistence (if present), cross-talk, 
dark stability and non-linearity, flatfield stability, defringing,
sky variation, telescope/dust thermal emission

non-linearity, reset-correction (de-biassing) dealt with in data 
aquisition system

Issues: 

variable seeing on short timescales, high proportion of bad pixels
should also consider stacking cf. interleaving for shallow surveys ?

may need to do PSF fitting on individual components of interleave 
irrespective of above 

for shallow surveys can we use 16 bit unsigned products rather than -32 bit
reals ?  can do for much of pipeline anyway


Test data:

need to aquire test data from WFCAM to assess persistence, reset anomalies,
dark stability, flat stability, bad pixel problems etc... asap

what other UKIRT?WFCAM test data do we need ? - more flats (dome cf. 
twilight sky cf. dark sky) dark stability, WFCAM test data - cross-talk
image persistence, reset anomaly, dark stability

Standard 2D processing at summit and in Cambridge to be ready for Version 1. 
with some tuning during commissioning and science verification.


Standard products
-----------------

Additional image parameters: 

Kron radius:  r_k equal to 1st moment of radial distribution

Petrosian radius:  r_p defined in terms of annular surface brightness
                   parameter \eta

Kron flux: summation out to k*r_k can be elliptical or circular ?
(similar to the total flux which it could replace)

Petrosian flux: summation out to k*r_p only seen circular, is elliptical 
worth having ?

Feedback suggests we stick to circular for ease of comparison (and simplicity) and at least pending JHK simulations (or real data) use the SDSS values for r_k, r_p and eta

FWHM alternative: image FWHM summation out to k*FWHM, since for standard
profiles is at least as good as above two

finely? sampled circular aperture fluxes of fixed sizes - similar to what 
we already have except more  

suggest:

Pk, core/2, core/sqrt(2), core, sqrt(2)*core, 2*core, 2*sqrt(2)*core, 4*core, 4*sqrt(2)*core,
8*core, 8*sqrt(2)*core, 16*core, 16*sqrt(2)*core, 32*core

where core = ~ for stellar images for site ie. ~0.5 arcsec

Note that if we set core = 0.5 arcsec then the above sequence leaving out 
Pk, would read apertures of diameter:

0.5, 0.7, 1.0, 1.4, 2.0, 2.8, 4.0, 5.7, 8.0, 11.3, 16.0, 22.6, 32.0 arcsec


which makes for a pleasing spacing.

[see Staunton etal. p 498 for SDSS apertures - problem with them is they
are too coarse a grid (eg. x2.5 steps in radius) and extend to far too 
large a radius given the likely 2D processing constraints on WFCAM data]

Note that for circular apertures all of Kron, Petrosian and FWHM are  
subsumed within circular aperture fluxes if interpolate from fine enough
grid, as are SDSS apertures (different seeing etc.. implies will need to
interpolate/correct to really match the SDSS ones anyway)

How fine ? what spacing ? how big a radius ?  above are suggested set.

finer grid means can always pick optimal aperture for stellar images

aperture corrections for stellar images are useful adjunct (already computed)

Note that standard background estimation ultimately defines max. radius
as may 2D processing and is expected to be <~30 arcsec.


couldn't see anything extra in Sextractor list worth having not covered 
in above and current lists

Image morphological classification: is spurion, stellar, non-stellar enough
together with the pseudo N(0,1) statistic ?, could use Bayesian priors
when calibrated + other knowledge eg. globular cluster..........

to produce P(star), P(galaxy) etc.....

and is mainly based on the curve-of-growth defined by the aperture fluxes, profile fit results.
Image sizes and ellipciticy, are useful extra information thrown into the image classification pot.
The Bayesian prior knowledge will be needed to produce sensible P(star), P(galaxy) using a standard Bayesian classification scheme.


need to add in errors on certain parameters and processing flag a la
SDSS one eg. set bit for hit edge etc...  

see updated parameter list.


Standard products to be ready for Version 1. with some tuning during 
commissioning and science verification.


Further products
----------------

PSF generation methods, oversampled spatially varying PSFs good news, 
how to form them cf. SDSS, HST, 2MASS ? 
may need to use components of interleave ? and for other possible 2Ds fits
such as Sersic.

oversampled real PSFs vital for correct fitting of undersampled images

Options for PSF fitting are:
 1) Profile fit without centroid adjustment using standard multiple components
 2) Profile fit with centroid adjustment using standard multiple components
 3) DAOPHOT/CLEAN algorithm (Full Monty) - multiple sources with full iterative
    solution, adding and removing components as necessary (CPU intensive) and
    obviously won't work for galaxies ie. prbably useless for generic higher
    latutidue fields

do we additionally need more general profile fits eg. de Vaucouleur, 
exponential, Plummer ???

After much discussion with Sloan people and others we have decided on attempting Sersic profile fits both 1D (from the aperture fluxes) and a goal of 2D direct from the images, using the above PSF information to eliminate the seeing dependence of the result. Note that exponential and de-Vaucouleur profiles are special cases of Sersic profile.


Aligning with DB versioning: further processing Version 1. and 2. would
include options 1,2 for PSF work.  Option 3 would be developed for
Version 3.  1D/2D Sersic fits would be developed for Version 2.


Database-driven products   
------------------------

Organised by DB but run using general purpose selectable agorithmic toolkit

asynchronous operations such as
list driven photometry, vital to make same measurements across all bands

master list created how? from union of catalogues in each band ? + external
lists ? stacking images first then generating catalogue in general not so 
good since s:n will be worse for interesting images, should be RA, Dec list

list drives software processing via WCS, can use object masks to improve 
standard background estimation.  External (no-WFCAM) lists can also be 
incorporated.

Version 1. just WFCAM data
Version 2. externally derived lists
Version 3. cross-photometry on external datasets

stacking, complex problem, not necessarily single optimum solution eg.
image detection cf. resolution of higher s:n images, Subaru involvement,
JPE's e-science project, loads of possible methods, inc. re-running 
image detection alogrithms

[At the moment both the stacking and mosaicing prototypes use the simplest possible method - ie. nearest pixel, which avoids ending up with correlated spatial noise and gives very reliable parameter estimates for detected images. We need to investigate alternative methods thoroughly since it is not clear what method gives the best results from the point of view of deriving image parameters. There is a recent TERAPIX report (somewhere) on this very problem. One of its conclusions was that nearest pixel gives very good parameter estimates cf. to original (not too surprisingly). It also discusses other alternatives. We need to do something similar for WFCAM since the different sampling/resolution may well lead to different conclusions.]


Version 1. stacker and mosaicer, straighforward algorithms with ability to 
deal correctly with, scaling, confidence maps, spurion rejection, and
accurate WCS alignment

Version 2. optimal use of varying PSF information for image detection 
problem and for image resolution problem.

Version 3. complete image restoration using all data contemporaneoulsy

LSBs, or more generic image detection methodology, not discussed at 
UKIDSS meeting, is it worth pursuing for NIR arrays ?

Version 2. re-detections using fixed large detection filters having 
previously removed standard image detections.  Assess feasibility of
LSB work in NIR mosaic data.

Version 3. await results for Version 2. before proceeding.

detection of transient phenomena, adpative kernel matching, difference 
imaging continuum subtraction, inc. re-running image detection alogrithms

Version 1.  ability to do continuum subtraction for narrow band (eg. H2) work
Version 2.  more sophisticated difference imaging using adaptive kernel
            matching
Version 3.  the rest

--------------------------------------------------------------------------

Timescales for Versions 1-3 are:

Version 1.  ready for WFCAM commissioning eg. end of 2003
Version 2.  ready for one year after survey operations begin eg. notionally
            end of 2004
Version 3.  rollout sometime later though development can be concurrent

--------------------------------------------------------------------------


Other Issues
------------

Setting up standard photometric calibration fields regularly spaced in RA
need to ensure this gets done early on and that standards in all broadbands
used in a night are taken at regular (1hr?) intervals.  STH to coordinate.

Skyprobe local extinction monitor -v- ad hoc mishmash of AG measures etc..


MJI, DWE, STH, JRL


-------------------------------------------------------------------------

	
ACTIONS
-------

1.  CASU to produce updated CDR document taking account of recent       
    discussions - needs timeline for various components                 ONGOING
2.  WFAU to produce updated document ditto, both by end of           
    December                                                            DONE
3.  MJI to contact MMC about acquiring recent WFCAM test data,             
    (received test data and report)                                     DONE
4.  STH Dome flats -v- twilight flats -v- dark sky flats, test data    
    still needed, STH to try again in December - nada.  AA to
    attempt with UFTI on 15th January                                   ONGOING
5.  JRL to get prototype WFCAM FITS header for images and forward to    
    WFAU for comments                                                   ONGOING
6.  JRL to put together example data and catalogues from CIRSI data    
    for use in tests of ingestion, stacking/mosaicing at WFAU           ONGOING
7.  SJW to contact MMC about commissioning duration and plans,          
    particularly with regard to setting up photometric standard fields  ONGOING
8.  STH assuming CASU processes the data who is going to do colour  
    equations produce the standards and (keep updating them) do meso    STH
    slew tests                                                          VOLUNTEERED
9.  MJI to ask MMC about plans for DAS 5s exposures, 16 bit alternative 
    data format                                                         DEBATING
10. SJW to set up simulations to test parameter generation software,    
    sensitivity to seeing etc...                                        ONGOING
11. SJW to provide/select a night of UFTI/UIST data to compare 2D       
    processing                                                          ONGOING
12. WFAU to do more tests to find bottleneck in network connectivity 
    between Cambridge and Edinburgh.  Current large scale transfer
    achieved speed at 2.4 Mbit/s is hopeless                            ONGOING
13. CASU to forward prototype stacking and mosaicing software to WFAU for 
    tests.                                                              DONE



Last modified: Wed Mar 19 10:52:34 2003