Difference between revisions of "Calibration Server"

From Gaia Science Alerts Working Group
Jump to navigationJump to search
(22 intermediate revisions by 3 users not shown)
Line 1: Line 1:
The main address at the moment is:
+
The main address of the server at the moment is:
http://gaia020.ast.cam.ac.uk:5000/
+
http://gsaweb.ast.cam.ac.uk/followup
  
 +
== Authorization ==
  
== Autorization ==
+
* In order to access the server you need a so called "hashtag" which is an authorization key. In order to get one you need to ask Lukasz Wyrzykowski and give him also
 +
the lon,lat of your observatory and list of filters you have.
 +
* If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests.
  
 +
== Preparing your data ==
 +
In order to obtain the sextractor file with [Ra, Dec] coordinates required by the photometric server, the following steps have to be performed.
  
* In order to access the server you need a so called "hashtag" which is an autorization key. In order to get one you need to ask Lukasz Wyrzykowski and report to him also
+
* Obtain the Equatorial coordinates for you photometry fits file by using [http://nova.astrometry.net/ Astrometry.net].
the lon,lat of your observatory and list of filters you have
+
* Download the Astrometry.net output file (''new-image.fits'' by default) from the site and use it as the sextractor input.
 +
* Run sextractor:
 +
sex new-image.fits -c /path_to/default.sex
 +
* make sure SExtractor outputs those : MAG_AUTO, MAGERR_AUTO, ALPHA_J2000, DELTA_J2000 and your configuration file should state that you want the output to be ASCII file with header (CATALOG_TYPE    ASCII_HEAD)
 +
* Upload the SExtractor output catalogue (by default ''test.cat'') to the Cambridge Photometry Calibration Server.
  
 +
== Calibration of your photometric data ==
  
 +
The main url for calibration of your sextractor catalogs is
 +
http://gsaweb.ast.cam.ac.uk/followup
  
== Headline text ==
+
You first need to login with your hashtag in order to be able to upload the data and to view the observations. Without login you can still see the lightcurves of the events.
 +
 
 +
==== Data format ====
 +
The allowed formats for the sextractor catalogs are plain ascii and FITS. We also read FITS_LDAC. The fits should have all necessary columns defined.
 +
The columns which are required (specified in ''default.params'':
 +
MAG_AUTO
 +
MAGERR_AUTO
 +
ALPHA_J2000
 +
DELTA_J2000
 +
 
 +
If outputing as ASCII make sure the header is also outputed. Set it in ''default.sex'' as:
 +
CATALOG_TYPE    ASCII_HEAD
 +
 
 +
If using different photometric software than SExtractor, then please add the following header to the ASCII output:
 +
#  1 MAG_AUTO        Kron-like elliptical aperture magnitude        [mag]
 +
#  2 MAGERR_AUTO    RMS error for AUTO magnitude                    [mag]
 +
#  3 ALPHA_J2000    Right ascension of barycenter (J2000)          [deg]
 +
#  4 DELTA_J2000    Declination of barycenter (J2000)              [deg]
 +
and make sure the columns are as described in the header (i.e. 1st is mag, 2nd is error, and so on). Also, note the separator in the data file should be space, not tab.
 +
 
 +
==== Uploading your observations ====
 +
Before uploading, please make sure your event you have observed is on the list of alerts. You can see all alerts or only those for which there is any data already.
 +
If your event is not on the list you will need to add it manually, see below.
 +
If your event is there, then please note its ivorn (ID), because you will need to provide it while uploading the data - this helps identify the right object on your image.
 +
 
 +
The upload form is straightforward. You need to provide the IVORN (the unique id of the alert), MJD of observations (note that MJD=JD-2400000.5), exposure time in seconds, optional comment on the datapoint, Sextractor catalog: browse the file here from your computer, matching radius: how accurate is your astrometry with respect to e.g. SDSS (typically 2'' is good, but you might want to try larger radii if calibration is poor). Force Filter = no (default) indicates that the filter will be determined automatically for your data - the best matching one will be used. If you know which filter is the closest to your then you can force it by choosing from the list.
 +
Dry Run: if ticked, it means this is just a test submission and nothing gets stored. You can then repeat the insertion as soon as you are happy with it. Then, switch off Dry Run and your data will get stored. Note, you will not be able to submit the same data point twice if not in Dry Run.
 +
 
 +
==== New event ====
 +
It is important that you can upload the data for the alerts which are in the database (e.g. which are listed here http://gsaweb.ast.cam.ac.uk/followup/list_of_alerts ).
 +
If the alert is not in our DB, you can insert it http://gsaweb.ast.cam.ac.uk/followup/newevent/.
 +
IVORN is the unique Virtual Observatory event name, e.g. ivo://nvo.caltech/voeventnet/catot#1306271320054132125. If it is unknown the format should be, e.g. ivo://asassn/ASASSN-13ax, for ASAS-SN survey object 13ax.
 +
 
 +
Ra,Dec are given in decimal degrees.
 +
 
 +
URL: link to the transient's page, ideally to individual event, but if not available, then to the main page of the feeding survey, e.g. http://ogle.astrouw.edu.pl/ogle4/transients/
 +
 
 +
== Automated submission ==
 +
You can also do that from the command line using HTTP POST protocol
 +
curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog_with_filename;filename=test.cat"
 +
      -F "hashtag=XXXX" -F "MJD=2" -F expTime=1 -F noPlot=1
 +
      -F forceFilter=no -F dryRun=1 -F outputFormat=json "http://gsaweb.ast.cam.ac.uk/followup/cgi/upload"
 +
Parameters of the  http://gsaweb.ast.cam.ac.uk/followup/cgi/upload
 +
* matchDist -- matching radius in arcseconds
 +
* hashtag -- your authorization key
 +
* MJD -- mjd of the observations
 +
* expTime -- exposure time
 +
* comment -- comments
 +
* EventID -- the ivorn of the alert
 +
* dryRun -- the value of 1 allows you to check the results of the calibration without inserting anything into our DB
 +
* forceFilter -- "no" means that the calibration will be fully automated. But You can also specify APASS/V if you want to calibrate using a particular survey/filter
 +
* sexCat -- that's the Sextractor  catalog you are trying to submit
 +
* outputFormat -- at the moment the only allowed values are json and html
 +
 
 +
 
 +
== Review of the data in the Database ==
 +
 
 +
You can view the list of followup_data which are in the DB http://gsaweb.ast.cam.ac.uk/followup/list_of_followups
 +
 
 +
You can view the list of alerts which are in the DB http://gsaweb.ast.cam.ac.uk/followup/list_of_alerts
 +
, and the subset of them which have followup data
 +
http://gsaweb.ast.cam.ac.uk/followup/list_of_alerts?observed_only=1.
 +
From these pages you can also see the light curves for those alerts which do have followup data.
 +
Here is the example of the light curve:
 +
http://gsaweb.ast.cam.ac.uk/followup/get_alert_lc?alert_id=2999
 +
 
 +
== Source code ==
 +
 
 +
The calibration server is written in python using Flask framework. If you'd like to help/implement something, ask Sergey Koposov for the access to the git repository
 +
on http://bitbucket.org

Revision as of 16:33, 29 November 2016

The main address of the server at the moment is: http://gsaweb.ast.cam.ac.uk/followup

Authorization

  • In order to access the server you need a so called "hashtag" which is an authorization key. In order to get one you need to ask Lukasz Wyrzykowski and give him also

the lon,lat of your observatory and list of filters you have.

  • If you have the hashtag, you can use the login form of the server to authorize yourself. Or if you are doing some scripting, you will need to specify hashtag option in the GET, POST requests.

Preparing your data

In order to obtain the sextractor file with [Ra, Dec] coordinates required by the photometric server, the following steps have to be performed.

  • Obtain the Equatorial coordinates for you photometry fits file by using Astrometry.net.
  • Download the Astrometry.net output file (new-image.fits by default) from the site and use it as the sextractor input.
  • Run sextractor:
sex new-image.fits -c /path_to/default.sex
  • make sure SExtractor outputs those : MAG_AUTO, MAGERR_AUTO, ALPHA_J2000, DELTA_J2000 and your configuration file should state that you want the output to be ASCII file with header (CATALOG_TYPE ASCII_HEAD)
  • Upload the SExtractor output catalogue (by default test.cat) to the Cambridge Photometry Calibration Server.

Calibration of your photometric data

The main url for calibration of your sextractor catalogs is http://gsaweb.ast.cam.ac.uk/followup

You first need to login with your hashtag in order to be able to upload the data and to view the observations. Without login you can still see the lightcurves of the events.

Data format

The allowed formats for the sextractor catalogs are plain ascii and FITS. We also read FITS_LDAC. The fits should have all necessary columns defined. The columns which are required (specified in default.params:

MAG_AUTO
MAGERR_AUTO
ALPHA_J2000
DELTA_J2000

If outputing as ASCII make sure the header is also outputed. Set it in default.sex as:

CATALOG_TYPE     ASCII_HEAD

If using different photometric software than SExtractor, then please add the following header to the ASCII output:

#   1 MAG_AUTO        Kron-like elliptical aperture magnitude         [mag]
#   2 MAGERR_AUTO     RMS error for AUTO magnitude                    [mag]
#   3 ALPHA_J2000     Right ascension of barycenter (J2000)           [deg]
#   4 DELTA_J2000     Declination of barycenter (J2000)               [deg]

and make sure the columns are as described in the header (i.e. 1st is mag, 2nd is error, and so on). Also, note the separator in the data file should be space, not tab.

Uploading your observations

Before uploading, please make sure your event you have observed is on the list of alerts. You can see all alerts or only those for which there is any data already. If your event is not on the list you will need to add it manually, see below. If your event is there, then please note its ivorn (ID), because you will need to provide it while uploading the data - this helps identify the right object on your image.

The upload form is straightforward. You need to provide the IVORN (the unique id of the alert), MJD of observations (note that MJD=JD-2400000.5), exposure time in seconds, optional comment on the datapoint, Sextractor catalog: browse the file here from your computer, matching radius: how accurate is your astrometry with respect to e.g. SDSS (typically 2 is good, but you might want to try larger radii if calibration is poor). Force Filter = no (default) indicates that the filter will be determined automatically for your data - the best matching one will be used. If you know which filter is the closest to your then you can force it by choosing from the list. Dry Run: if ticked, it means this is just a test submission and nothing gets stored. You can then repeat the insertion as soon as you are happy with it. Then, switch off Dry Run and your data will get stored. Note, you will not be able to submit the same data point twice if not in Dry Run.

New event

It is important that you can upload the data for the alerts which are in the database (e.g. which are listed here http://gsaweb.ast.cam.ac.uk/followup/list_of_alerts ). If the alert is not in our DB, you can insert it http://gsaweb.ast.cam.ac.uk/followup/newevent/. IVORN is the unique Virtual Observatory event name, e.g. ivo://nvo.caltech/voeventnet/catot#1306271320054132125. If it is unknown the format should be, e.g. ivo://asassn/ASASSN-13ax, for ASAS-SN survey object 13ax.

Ra,Dec are given in decimal degrees.

URL: link to the transient's page, ideally to individual event, but if not available, then to the main page of the feeding survey, e.g. http://ogle.astrouw.edu.pl/ogle4/transients/

Automated submission

You can also do that from the command line using HTTP POST protocol

curl -F matchDist=2 -F EventID='ivo://110610' -F sexCat="@path_to_your_sex_catalog_with_filename;filename=test.cat" 
     -F "hashtag=XXXX" -F "MJD=2" -F expTime=1 -F noPlot=1
     -F forceFilter=no -F dryRun=1 -F outputFormat=json "http://gsaweb.ast.cam.ac.uk/followup/cgi/upload"

Parameters of the http://gsaweb.ast.cam.ac.uk/followup/cgi/upload

  • matchDist -- matching radius in arcseconds
  • hashtag -- your authorization key
  • MJD -- mjd of the observations
  • expTime -- exposure time
  • comment -- comments
  • EventID -- the ivorn of the alert
  • dryRun -- the value of 1 allows you to check the results of the calibration without inserting anything into our DB
  • forceFilter -- "no" means that the calibration will be fully automated. But You can also specify APASS/V if you want to calibrate using a particular survey/filter
  • sexCat -- that's the Sextractor catalog you are trying to submit
  • outputFormat -- at the moment the only allowed values are json and html


Review of the data in the Database

You can view the list of followup_data which are in the DB http://gsaweb.ast.cam.ac.uk/followup/list_of_followups

You can view the list of alerts which are in the DB http://gsaweb.ast.cam.ac.uk/followup/list_of_alerts , and the subset of them which have followup data http://gsaweb.ast.cam.ac.uk/followup/list_of_alerts?observed_only=1. From these pages you can also see the light curves for those alerts which do have followup data. Here is the example of the light curve: http://gsaweb.ast.cam.ac.uk/followup/get_alert_lc?alert_id=2999

Source code

The calibration server is written in python using Flask framework. If you'd like to help/implement something, ask Sergey Koposov for the access to the git repository on http://bitbucket.org