[ << ] [ >> ]           [Top] [Contents] [Index] [ ? ]

6. Data Reduction and End of Observing

6.1 ATNF Data Format

Raw data output from ATNF correlators is written in RPFITS format. More information on RPFITS can be found at http://www.atnf.csiro.au/computing/software/rpfits.html.

6.2 DFB normalization

As of June 2015, it has become known DFB4 (and DFB3) spectral–line data lack the correct normalization and have not had the Tsys scaling applied. To convert the individual raw spectrum R(p,t,c) for polarization p, time t and spectral channel c, it should be scaled as follows (courtesy John Reynolds);

S(p,t,c) = Tsys(p,t)/Av(p,t) * R(p,t,c)

to obtain a calibrated spectrum S(p,t,c) where Tsys(p,t) is the Tsys scale-factor and Av(p,t) is the normalising term. The values of Tsys(p,t) are stored in the RPFITS header and are available in ASAP (see below), using the <scantable>.get_tsys() or <scantable>.get_tsysspectrum() commands. A suggested method to correct the data within ASAP, is provided below. The script outputs corrected SDFITS files which can be reloaded into LIVEDATA or ASAP (see below.)

import glob
from asap import *
import numpy as np

types = ('*.rpf')
for t in types:

for f in files:



 scns = sc.getscannos()
 for scn in scns:


  for nif in range(0,sc.nif()):
   for pol in range(0,sc.npol()):
    for cyc in range(0,sc.ncycle()):
     nspec = ( sc.get_tsysspectrum(cyc) / np.max( sc.get_spectrum(cyc) ) ) * sc.get_spectrum(cyc)

Applying the above scaling should render spectra calibrated in SEFD (system equivalent flux density, i.e. Janskys not Kelvins).

For polarization data, p=AA* or p=BB*, the normalising term Av(p,t) is simply the mean power of the (real) spectrum R(p,t,c), averaged over all spectral channels c. For the cross-polar terms p=AB* the correction is the geometric mean of the two parallel hands;

Av(AB*,t) = sqrt( Av(AA*,t) * Av(BB*,t))

6.3 Opacity calculations

Opacity is not an issue for low-frequency (< 15 GHz) observations. However, for observations of the 22.2GHz water line and Ammonia 23.6GHz (1,1), 23.7GHz (2,2) and 23.8GHz (3,3) lines, opacity has to be taken into account. Normally, skydips are performed to determine the opacity using a simple model which assumes the system temperature will increase linearly with increasing airmass (decreasing elevation). Within ASAP (see below), there is a ’skydip’ routine which allows the user to input a set of scans (assuming system temperature is being recorded) and determine the Zenithal opacity. See here for more information.

Annex 2 of ITU-R Recommendation P.676-8 (10/2009) allows calculation of the Zenithal attenuation (opacity) due to atmospheric oxygen and water using ground weather-station data. For more information showing examples of opacities calculated using skydips and weather-data, please see the above link. A web form allowing calculation of Zenithal Opacites (and other meterological data) from the above ITU-R Recommendation can be found here.

6.4 Spectral–line Calibration

Traditionally, spectral–line observing teams have devised their own means to calibrate their data. A suggested procedure using the SPOT mode of the Telescope Control Software (TCS), which is used to obtain scaling factors using point sources can be found here. In addition to calibration, the above utility can be used for high-frequency ($\sim$ 22-24GHz) observations to determine pointing offsets using bright water maser and ammonia sources.


Detailed information on the specifics of LIVEDATA and GRIDZILLA as related to the HIPASS and ZOA HI Surveys can be found in Barnes et al. (2001), MNRAS, 322 486. The GRIDZILLA GUI contains a HELP menu item and the different sections of the LIVEDATA GUI contains widgets, which when clicked on, present the user with a popup containing information relevant to that section.

With LIVEDATA and GRIDZILLA it is possible to perform batch processing via scripts. Examples of batch scripts can be found here.

6.6 ASAP

The ATNF Spectral Analysis Package (ASAP) is a new software package to reduce single-dish, single-pointing spectral line observations. At this stage it is tuned towards data from ATNF instruments and reads RPFITS, SDFITS and is able to output SDFITS, ASCII and CLASS format for processing within GILDAS and SPECX. A Cookbook, Tutorials and further information on ASAP is located here.

In addition to those below, more examples of data reduction using ASAP can be found here. This list will be added to in time.

6.6.1 Example ASAP batch script: POSITION SWITCHING

In this example, observations at 22GHz are done in Position Switching Mode, where the source and a referenced position (free of line emission) are observed in sequence. The script below is saved into a file (called proc.py) and run from within ASAP by typing execfile('proc.py') or python -i proc.py from the command-line. It reads in and processes all *RCW49*.rpf files in the current directory.

import glob
from asap import *

fnames = glob.glob('*RCW49*.rpf')

vec = []
for f in fnames:

quotients = []
for scan in vec:
        scan.average_pol() #comment out this line to display both pols

for q in quotients:
        msk = q.create_mask([-100,100])

av = average_time(quotients)
iav = av.average_pol()

plotter.set_legend(mode=-1) # No legend


Figure 6.1: 22GHz Stokes I spectrum of RCW49.

6.6.2 Example ASAP batch script: Beam Switching (MX Mode)

In Beam–Switching or MX mode, each beam of the 20CM or 6GHz Multibeam receivers is placed in turn on the source of interest. When not on-source, the other beams are still aquiring data and so are used as reference data. In this example (taken from the P502 Methanol Multibeam Survey), the RPFITS file contains two IFS; the first (IF0) contains the Methanol maser line at 6668MHz and the second (IF1) contains Excited OH at 6035MHz. The IF selected depends on the index used in selection.set_ifs(). This example only shows data associated with IF0. The script file is shown below.

scans = scantable("2009-03-20_1025_MMB-MX-G300.96.rpf",average=False)
selection = selector()

q = scans.mx_quotient()
msk = q.create_mask([-55,-30])

av = q.average_beam()
iav = q.average_pol()



Figure 6.2: Stokes I 6668MHz Methanol spectrum for 300.96+1.14 .


It is possible to process Parkes data using MIRIAD. Initially, data is processed using LIVEDATA and GRIDZILLA, where the latter produces a FITS file which can be loaded into MIRIAD using the FITS utility. As an example, here we load in a GRIDZILLA–created FITS file and use the MIRIAD utility MBSPECT to display the profile of a galaxy, various parameters, plus create both an ASCII file of the spectra and a colour postscript file. The basic script below allows one to copy and past the code into a file (making it executable) for displaying a large number of sources quickly; it assumes the BASENAME utility is available on your system.

if [ $# -lt 1 ]; then
 echo "Please supply a FITS file"
 exit 0
BASE=`basename ${FILE} .fits` #change suffix if different!

if [ -d "${BASE}.mir" ]; then #remove existing image if present
 rm -rf ${BASE}.mir

fits in=${BASE}.fits op=xyin out=${BASE}.mir
puthd in=${BASE}.mir/restfreq value=1.42040572 #insert rest freq of line
mbspect in=${BASE}.mir xaxis=optical hann=2 order=-3 ${OPT} 
              ${MSK} device=/xs log=${BASE}.spec
mbspect in=${BASE}.mir xaxis=optical hann=2 order=-3 ${OPT} 
              ${MSK} device=${BASE}.ps/cps

Running this script produces the following output and image.

#FR   Clipped rms:    0.0066 Jy
#FN   ( 654 out of 1024 channels)
#MC   xaxis: optical
#MX   Maximum:      0.050 Jy   at   5065.440 km/s
#MN   Minimum:     -0.028 Jy   at   6141.707 km/s
#NP   Number of spectral points:  1024
#SN1  Peak S/N ratio =       7.57
#CL   Clipping inside range (    0.000,    0.000) Jy
      Moment:         0               1               2
#MM                  9.660        5215.471         101.712
#SN2  Mean S/N ratio =       0.83
      Robust moments: 0               1               2
#MR                  9.744        5197.634         104.134
      units:        Jy km/s         km/s            km/s


Figure 6.3: Spectrum of a galaxy using the MBSPECT utility of MIRIAD

For information on keywords used in the above script, please refer to the MIRIAD user guide.

6.7.1 Gaussian fits within ASAP

Although MIRIAD has utilities to perform gaussian-fits to spectra, you can import the ASCII file created from MBSPECT (using log=file) into ASAP. For example, place the following code into the asapuserfuncs.py file (located in your $HOME/.asap directory).

def readxy(filename=None):
     f = file(filename,'r')
     x = []
     y = []
     lines = f.readlines()
     for l in lines:
         vals = l.split()
     return x,y

Now when you run ASAP, you read in the ASCII file (with comments removed and only two columns) and fit a two–component gaussian and output a Postscript file of the plot as follows.

x,y = readxy('fred')
f = fitter()
f._p.set_axes('xlabel','velocity km/s (LSR-K)')

6.8 Source finding programs

ASAP has the functionality to identify spectral lines within a data cube using LINEFINDER. Similarly, the DUCHAMP utility can also search for spectral-line sources, produce source lists, spectra and moment maps. Users are encouraged to view the online documentation for these.

6.9 Pulsar data reduction

For reducing Pulsar data, please refer to the ATNF Pulsar page.

6.10 Other Packages

A list of supported and non-supported data-reduction software can be found at the ATNF website.

6.11 Australia Telescope Online Archive

Although DVDs of data can be supplied to observers, spectral–line (non-pulsar) data is automatically uploaded into the Australia Telescope Online Archive (ATOA). Using your OPAL account, you can download data directly from ATOA. There is usally a gap of a few days between taking your data and seeing it in ATOA.

6.12 Observer Report

At the end of your run, we request you fill in an Observers Report to help us make the Observatory a better place to work and play. The form is found here. The completed form is distributed with your name visible to a selected list of Observatory staff whom are listed via a link on the above form.

[ << ] [ >> ]           [Top] [Contents] [Index] [ ? ]
Last updated by Stacy Mader on October 14, 2017