Notes from UKIDSS meeting 25th November 2002

Updated after CASU-WFAU liaison meeting 9th December 2002 (+NCH, IB and SJW)

Overall philosophy

Summit    -  causal pipeline: DQC feedback, first pass science products

Standard  -  nightly pipeline: DQC check, astrometric and initial 
             photometric calibration, standard catalogue parameters, 
             final 2D science images

Further   -  nightly pipeline: PSF estimation and general profile fitting

Database  -  asynchronous processing: stacking, difference imaging,
driven       temporal, matching and colocated list driven stuff 
             (inc. external list drivers), contiguous tiled regions and 
             seamless overlap calibrated catalogues, optimal detection and 
             parameterisation of eg. LSBs (ie. bigger than PSFs)

(note basic --> standard since not such negative connotations and
   advanced --> Database driven since some of the processing not that
                advanced per se)

2D instrumental signature removal

effects to deal with may include: persistence (if present), cross-talk, 
dark stability and non-linearity, flatfield stability, defringing,
sky variation, telescope/dust thermal emission

non-linearity, reset-correction (de-biassing) dealt with in data 
aquisition system


variable seeing on short timescales, high proportion of bad pixels
should also consider stacking cf. interleaving for shallow surveys ?

may need to do PSF fitting on individual components of interleave 
irrespective of above 

for shallow surveys can we use 16 bit unsigned products rather than -32 bit
reals ?  can do for much of pipeline anyway

test data:

need to aquire test data from WFCAM to assess persistence, reset anomalies,
dark stability, flat stability, bad pixel problems etc... asap

what other UKIRT?WFCAM test data do we need ? - more flats (dome cf. 
twilight sky cf. dark sky) dark stability, WFCAM test data - cross-talk
image persistence, reset anomaly, dark stability

Standard 2D processing at summit and in Cambridge to be ready for Version 1. 
with some tuning during commissioning and science verification.

Standard products

additional image parameters: 

Kron radius:  r_k equal to 1st moment of radial distribution

Petrosian radius:  r_p defined in terms of annular surface brightness
                   parameter \eta

Kron flux: summation out to k*r_k can be elliptical or circular ?
(similar to the total flux which it could replace)

Petrosian flux: summation out to k*r_p only seen circular, is elliptical 
worth having ?

FWHM alternative: image FWHM summation out to k*FWHM, since for standard
profiles is at least as good as above two

finely? sampled circular aperture fluxes of fixed sizes - similar to what 
we already have except more  

suggest: Pk, core/2, core, sqrt(2)*core, 2*core, 2*sqrt(2)*core, 4*core, 
4*sqrt(2)*core, 8*core, 8*sqrt(2)*core, 16*core, 16*sqrt(2)*core, 32*core) 
where core = ~ for stellar images for site

Note that if we set core = 0.5 arcsec then the above sequence leaving out 
Pk, would read apertures of diameter:

0.5, 1.0, 1.4, 2.0, 2.8, 4.0, 5.7, 8.0, 11.3, 16.0, 22.6, 32.0  arcsec

[see Staunton etal. p 498 for SDSS apertures - problem with them is they
are too coarse a grid (eg. x2.5 steps in radius) and extend to far too 
large a radius given the likely 2D processing constraints on WFCAM data]

Note that for circular apertures all of Kron, Petrosian and FWHM are all 
subsumed within circular fluxes if interpolate from fine enough grid as
are SDSS aperture (different seeing etc.. implies will need to interpolate/
correct to really match the SDSS ones anyway)
how fine ? what spacing ? how big a radius ?  above are suggested set.

finer grid means can always pick optimal aperture for stellar images

aperture corrections for stellar images are useful adjunct (already computed)

Note that standard background estimation ultimately defines max. radius
as may 2D processing and is expected to be <~30 arcsec.

couldn't see anything extra in Sextractor list worth having not covered 
in above and current lists

image morphological classification: is spurion, stellar, non-stellar enough
together with the pseudo N(0,1) statistic ?, could use Bayesian priors
when calibrated + other knowledge eg. globular cluster..........

profile fit results are useful extra information for image classification

need to add in errors on certain parameters and processing flag a la
SDSS one eg. set bit for hit edge etc...

Standard products to be ready for Version 1. with some tuning during 
commissioning and science verification.

Further products

PSF generation methods, oversampled spatially varying PSFs good news, 
how to form them cf. SDSS, HST, 2MASS ? 
may need to use components of interleave ?

oversampled real PSFs vital for correct fitting of undersampled images

Options for PSF fitting are:
 1) Profile fit without centroid adjustment using standard multiple components
 2) Profile fit with centroid adjustment using standard multiple components
 3) DAOPHOT/CLEAN algorithm (Full Monty) - multiple sources with full iterative
    solution, adding and removing components as necessary (CPU intensive) and
    obviously won't work for galaxies ie. prbably useless for generic higher
    latutidue fields

do we additionally need more general profile fits eg. de Vaucouleur, 
exponential, Plummer ???

Aligning with DB versioning: further processing Version 1. would include 
options 1,2 for PSF work.  Option 3 would be developed for Version 2.

Database-driven products   

Organised by DB but run using general purpose selectable agorithmic toolkit

asynchronous operations such as
list driven photometry, vital to make same measurements across all bands

master list created how? from union of catalogues in each band ? + external
lists ? stacking images first then generating catalogue in general not so 
good since s:n will be worse for interesting images, should be RA, Dec list

list drives software processing via WCS, can use object masks to improve 
standard background estimation.  External (no-WFCAM) lists can also be 

Version 1. just WFCAM data
Version 2. externally derived lists and cross-photometry on external datasets

stacking, complex problem, not necessarily single optimum solution eg.
image detection cf. resolution of higher s:n images, Subaru involvement,
JPE's e-science project, loads of possible methods, inc. re-running 
image detection alogrithms

Version 1. stacker and mosaicer, straighforward algorithms with ability to 
deal correctly with, scaling, confidence maps, spurion rejection, and
accurate WCS alignment

Version 2. optimal use of varying PSF information for image detection 
problem and for image resolution problem.

Version 3. complete image restoration using all data contemporaneoulsy

LSBs, or more generic image detection methodology, not discussed at 
UKIDSS meeting, is it worth pursuing for NIR arrays ?

Version 1. re-detections using fixed large detection filters having 
previously removed standard image detections.  Assess feasibility of
LSB work in NIR mosaic data.

Version 2. await results for Version 1. before proceeding.

detection of transient phenomena, adpative kernel matching, difference 
imaging continuum subtraction, inc. re-running image detection alogrithms

Version 1.  ability to do continuum subtraction for narrow band (eg. H2) work
Version 2.  the rest.


Timescales for Versions 1-3 are:

Version 1.  ready for WFCAM commissioning eg. last quarter of 2003
Version 2.  one year after survey operations begin eg. notionally end of 2004
Version 3.  rollout sometime later though development can be concurrent


Other Issues

Setting up standard photometric calibration fields regularly spaced in RA
need to ensure this gets done early on and that standards in all broadbands
used in a night are taken at regular (1hr?) intervals.  STH to coordinate.

Skyprobe local extinction monitor -v- ad hoc mishmash of AG measures etc..




1.  CASU to produce updated CDR document taking account of recent       
    discussions - needs timeline for various components                 ONGOING
2.  WFAU to produce updated document ditto, both by end of           
    December                                                            ONGOING
3.  MJI to contact MMC about acquiring recent WFCAM test data,             
    (received test data and report)                                     DONE
4.  STH Dome flats -v- twilight flats -v- dark sky flats, test data    
    still needed, STH to try again in December - nada.  AA to
    attempt with UFTI on 15th January                                   ONGOING
5.  JRL to get prototype WFCAM FITS header for images and forward to    
    WFAU for comments                                                   ONGOING
6.  JRL to put together example data and catalogues from CIRSI data    
    for use in tests of ingestion, stacking/mosaicing at WFAU           ONGOING
7.  SJW to contact MMC about commissioning duration and plans,          
    particularly with regard to setting up photometric standard fields  ONGOING
8.  STH assuming CASU processes the data who is going to do colour  
    equations produce the standards and (keep updating them) do meso    STH
    slew tests - STH??? +??? decide                                     VOLUNTEERED
9.  MJI to ask MMC about plans for DAS 5s exposures, 16 bit alternative 
    data format                                                         DEBATING
10. SJW to set up simulations to test parameter generation software,    
    sensitivity to seeing etc...                                        ONGOING
11. SJW to provide/select a night of UFTI/UIST data to compare 2D       
    processing                                                          ONGOING
12. WFAU to do more tests to find bottleneck in network connectivity 
    between Cambridge and Edinburgh.  Current large scale transfer
    achieved speed at 2.4 Mbit/s is hopeless                            ONGOING
13. CASU to forward prototype stacking and mosaicing software to WFAU for 
    tests.                                                              DONE

Last modified: Thu Jan 23 14:57:27 2003