Notes from UKIDSS meeting 25th November 2002 Updated after CASU-WFAU liaison meeting 9th December 2002 (+NCH, IB and SJW)
Updated with new timescales 18th March.
Overall philosophy ------------------ Summit - causal pipeline: DQC feedback, first pass science products Standard - nightly pipeline: DQC check, astrometric and initial photometric calibration, standard catalogue parameters, final 2D science images Further - nightly pipeline: PSF estimation and general profile fitting Database - asynchronous processing: stacking, difference imaging, driven temporal, matching and colocated list driven stuff (inc. external list drivers), contiguous tiled regions and seamless overlap calibrated catalogues, optimal detection and parameterisation of eg. LSBs (ie. bigger than PSFs) (note basic --> standard since not such negative connotations and advanced --> Database driven since some of the processing not that advanced per se) 2D instrumental signature removal --------------------------------- effects to deal with may include: persistence (if present), cross-talk, dark stability and non-linearity, flatfield stability, defringing, sky variation, telescope/dust thermal emission non-linearity, reset-correction (de-biassing) dealt with in data aquisition system Issues: variable seeing on short timescales, high proportion of bad pixels should also consider stacking cf. interleaving for shallow surveys ? may need to do PSF fitting on individual components of interleave irrespective of above for shallow surveys can we use 16 bit unsigned products rather than -32 bit reals ? can do for much of pipeline anyway Test data: need to aquire test data from WFCAM to assess persistence, reset anomalies, dark stability, flat stability, bad pixel problems etc... asap what other UKIRT?WFCAM test data do we need ? - more flats (dome cf. twilight sky cf. dark sky) dark stability, WFCAM test data - cross-talk image persistence, reset anomaly, dark stability Standard 2D processing at summit and in Cambridge to be ready for Version 1. with some tuning during commissioning and science verification. Standard products ----------------- Additional image parameters: Kron radius: r_k equal to 1st moment of radial distribution Petrosian radius: r_p defined in terms of annular surface brightness parameter \eta Kron flux: summation out to k*r_k can be elliptical or circular ? (similar to the total flux which it could replace) Petrosian flux: summation out to k*r_p only seen circular, is elliptical worth having ?
FWHM alternative: image FWHM summation out to k*FWHM, since for standard profiles is at least as good as above two finely? sampled circular aperture fluxes of fixed sizes - similar to what we already have except more
Pk, core/2, core/sqrt(2), core, sqrt(2)*core, 2*core,
2*sqrt(2)*core, 4*core, 4*sqrt(2)*core,
8*core, 8*sqrt(2)*core, 16*core, 16*sqrt(2)*core, 32*core
where core = ~
for stellar images for site ie. ~0.5 arcsec Note that if we set core = 0.5 arcsec then the above sequence leaving out Pk, would read apertures of diameter:
which makes for a pleasing spacing. [see Staunton etal. p 498 for SDSS apertures - problem with them is they are too coarse a grid (eg. x2.5 steps in radius) and extend to far too large a radius given the likely 2D processing constraints on WFCAM data] Note that for circular apertures all of Kron, Petrosian and FWHM are subsumed within circular aperture fluxes if interpolate from fine enough grid, as are SDSS apertures (different seeing etc.. implies will need to interpolate/correct to really match the SDSS ones anyway) How fine ? what spacing ? how big a radius ? above are suggested set. finer grid means can always pick optimal aperture for stellar images aperture corrections for stellar images are useful adjunct (already computed) Note that standard background estimation ultimately defines max. radius as may 2D processing and is expected to be <~30 arcsec. couldn't see anything extra in Sextractor list worth having not covered in above and current lists Image morphological classification: is spurion, stellar, non-stellar enough together with the pseudo N(0,1) statistic ?, could use Bayesian priors when calibrated + other knowledge eg. globular cluster..........
and is mainly based on the curve-of-growth defined by the aperture fluxes,
profile fit results.
Image sizes and ellipciticy, are useful extra information thrown into the image classification pot.
The Bayesian prior knowledge will be needed to produce sensible P(star), P(galaxy) using a standard Bayesian classification scheme.
need to add in errors on certain parameters and processing flag a la SDSS one eg. set bit for hit edge etc...
Standard products to be ready for Version 1. with some tuning during commissioning and science verification. Further products ---------------- PSF generation methods, oversampled spatially varying PSFs good news, how to form them cf. SDSS, HST, 2MASS ? may need to use components of interleave ? and for other possible 2Ds fits such as Sersic. oversampled real PSFs vital for correct fitting of undersampled images Options for PSF fitting are: 1) Profile fit without centroid adjustment using standard multiple components 2) Profile fit with centroid adjustment using standard multiple components 3) DAOPHOT/CLEAN algorithm (Full Monty) - multiple sources with full iterative solution, adding and removing components as necessary (CPU intensive) and obviously won't work for galaxies ie. prbably useless for generic higher latutidue fields do we additionally need more general profile fits eg. de Vaucouleur, exponential, Plummer ???
Aligning with DB versioning: further processing Version 1. and 2. would include options 1,2 for PSF work. Option 3 would be developed for Version 3. 1D/2D Sersic fits would be developed for Version 2. Database-driven products ------------------------ Organised by DB but run using general purpose selectable agorithmic toolkit asynchronous operations such as list driven photometry, vital to make same measurements across all bands master list created how? from union of catalogues in each band ? + external lists ? stacking images first then generating catalogue in general not so good since s:n will be worse for interesting images, should be RA, Dec list list drives software processing via WCS, can use object masks to improve standard background estimation. External (no-WFCAM) lists can also be incorporated. Version 1. just WFCAM data Version 2. externally derived lists Version 3. cross-photometry on external datasets stacking, complex problem, not necessarily single optimum solution eg. image detection cf. resolution of higher s:n images, Subaru involvement, JPE's e-science project, loads of possible methods, inc. re-running image detection alogrithms
Version 1. stacker and mosaicer, straighforward algorithms with ability to deal correctly with, scaling, confidence maps, spurion rejection, and accurate WCS alignment Version 2. optimal use of varying PSF information for image detection problem and for image resolution problem. Version 3. complete image restoration using all data contemporaneoulsy LSBs, or more generic image detection methodology, not discussed at UKIDSS meeting, is it worth pursuing for NIR arrays ? Version 2. re-detections using fixed large detection filters having previously removed standard image detections. Assess feasibility of LSB work in NIR mosaic data. Version 3. await results for Version 2. before proceeding. detection of transient phenomena, adpative kernel matching, difference imaging continuum subtraction, inc. re-running image detection alogrithms Version 1. ability to do continuum subtraction for narrow band (eg. H2) work Version 2. more sophisticated difference imaging using adaptive kernel matching Version 3. the rest -------------------------------------------------------------------------- Timescales for Versions 1-3 are: Version 1. ready for WFCAM commissioning eg. end of 2003 Version 2. ready for one year after survey operations begin eg. notionally end of 2004 Version 3. rollout sometime later though development can be concurrent -------------------------------------------------------------------------- Other Issues ------------ Setting up standard photometric calibration fields regularly spaced in RA need to ensure this gets done early on and that standards in all broadbands used in a night are taken at regular (1hr?) intervals. STH to coordinate. Skyprobe local extinction monitor -v- ad hoc mishmash of AG measures etc.. MJI, DWE, STH, JRL ------------------------------------------------------------------------- ACTIONS ------- 1. CASU to produce updated CDR document taking account of recent discussions - needs timeline for various components ONGOING 2. WFAU to produce updated document ditto, both by end of December DONE 3. MJI to contact MMC about acquiring recent WFCAM test data, (received test data and report) DONE 4. STH Dome flats -v- twilight flats -v- dark sky flats, test data still needed, STH to try again in December - nada. AA to attempt with UFTI on 15th January ONGOING 5. JRL to get prototype WFCAM FITS header for images and forward to WFAU for comments ONGOING 6. JRL to put together example data and catalogues from CIRSI data for use in tests of ingestion, stacking/mosaicing at WFAU ONGOING 7. SJW to contact MMC about commissioning duration and plans, particularly with regard to setting up photometric standard fields ONGOING 8. STH assuming CASU processes the data who is going to do colour equations produce the standards and (keep updating them) do meso STH slew tests VOLUNTEERED 9. MJI to ask MMC about plans for DAS 5s exposures, 16 bit alternative data format DEBATING 10. SJW to set up simulations to test parameter generation software, sensitivity to seeing etc... ONGOING 11. SJW to provide/select a night of UFTI/UIST data to compare 2D processing ONGOING 12. WFAU to do more tests to find bottleneck in network connectivity between Cambridge and Edinburgh. Current large scale transfer achieved speed at 2.4 Mbit/s is hopeless ONGOING 13. CASU to forward prototype stacking and mosaicing software to WFAU for tests. DONE
Last modified: Wed Mar 19 10:52:34 2003