1. at the end of each subscan display the recorded data, including a gauss fit
2. at the end of a scan determine the pointing corrections for azimuth and elevation
3. if the backends have been calibrated before, the data shall be plotted as temperatures
4. process all receivers

Before a spectral-line on-the-fly (OTF) scan a valid CAL COLD
must have been taken. 

This CAL COLD scan will be used to calibrate the OTF scan.

During a standard OTF scan:

1    the first subscan is taken on an off-source reference position (REF).
2    one or several OTF subscans are taken.
3    normally another subscan is taken at a reference position.

     Optionally: continue with step 2.

     Normally the last subscan is a REF;
     but in general the sequence can also end with an OTF subscan.

The processing of spectral-line on-the-fly observations shall:

     after the first subscan (REF): do nothing

     after an OTF subscan that is not the last subscan: do nothing

     after the n-th REF subscan (n > 1):
     process all OTF subscans taken between the (n-1)-th and n-th
     REF subscan, using both the (n-1)-th and the n-th REF subscan.

     at the end of the OTF scan, if the last subscan was not a REF:
     process all OTF subscans taken since the last REF subscan,
     using (only) the last REF subscan.

     after any group of OTF subscans has been procesed: 
     generate plots (TBD)


Notes: 
------

Only "standard" OTF scans in the sense explained above shall be
processed correctly by ODP.

ODP requires that REF and OTF data are taken in the same scan.


Desiderata for future iterations:
---------------------------------

Support frequency-switched OTF.

Could consider using CAL COLD scans taken after the OTF scan and
processing of non-standard OTF scans.

1. data-producer: the current telescope control software produces so-called raw data and header files: header files with specification of the observation and one data file for each backend (see also subsections dataTransfer...). All files are finally stored on mrt-lx1.
2. post-processing tasks (data-consumer): After raw header and data files are produced, other task will process these files. Possible tasks are:
   1. calibration of heterodyne data (odpCal)
   2. fits converters (odpAbbaToFits)
   3. data analysis of pointing, focus, skidip (odpPointing, ...)
   4. data plots (odpPlot)

1. 4MHz backend:
   1. the backend data is transferred directly from the backend processor vbe4m to the file server mrt-ux1.
   2. The 4MHz backend processor also send UTC for all data dumps to the backend process beorga on iramea.
   3. The antenna control software reserves space in the raw data file and adds DAPs for the specific UTC time given.
   4. When this raw data file is transferred at the end of a subscan to the MRT file server, the actual data is written into the raw data file
2. ABBA:
   1. Raw data files are only written for ONE channel in wobbler and skydip mode.
   2. In the fast-scanning mode, data from the continuum backend is recorded every 250ms in order to have DAPs.
   3. At the end of a subscan:
      1. the raw data files are transferred
      2. data for all channels is retrieved from ABBA
      3. FITS files are generated from rowdata and ABBA files

1. The data-producer (see also subsection on "architectureOverview") will put raw data of a new subscan into a directory "/mrt-lx1/mrt/data/"project"/r"date".
2. After this, it will put a link to the header file of the new subscan in all subdirectories that of /mrt-lx1/mrt/data/Pipe/raw/pODP.
3. Each data processing task has its own subdirectory (e.g. /mrt-lx1/mrt/data/Pipe/raw/cal) and monitors this directory for new links.
4. If a task detects a new link, it does what it has to do and then removes the link.

1. data processing tasks can also "forward" links to other tasks, e.g. after a calibration the link can be forwarded to a plot task.
2. the raw data producer can code (e.g. in the file name) what class of observation we have: heterodyne, hera, abba, ... . This can help the consumer to decide if the link has to be processed or can be ignored.
3. The current pipeline concept only foresees consumer that monitor themself their directory. In the future, we also foresee to have consumers that can be woken up or that are started each time a jobs is available.

raw
   pODP
   abbaToFits

1. define SIC variables
   1. the observing procedure (odpProcedure)
   2. the project(odpProject)
   3. scan, subscan number(odpScan, odpSubscan)
   4. end of scan,0: no, 1:yes (odpEndOfScan)
   5. sky (0) or reference (>0) (odpSkyReference)
   6. calibration scan (1) or no (0) (odpCalibration)
   7. scan number of last calibration or -1 (odpCalibrationLast)
   8. backend number (1,2,3,4,or 7) (odpBackend)
   9. the directory where the raw data is (odpDirectory)
   10. the filename of the backend file (odpFileName)
2. call a script with the name of the observing procedure
3. wait for the end of the script WITH a TIMEOUT

after the n-th REF subscan (n>1):
---------------------------------

   otfcal is called with:
   1 a pre-script that assigns the approprate values to the
     following required SIC GLOBAL variables (all integers)
          odp#backendCode         ! backend 2 3 4 (5) or 7      
          odp#calScan             ! CAL scan number
          odp#flyScan             ! OTF scan number
          odp#refSubscan1         ! REF subscan before OTF subscans
          odp#flySubscanList[1]   ! OTF subscan
              ...                 ! more OTF subscans (otional)
          odp#refSubscan2         ! REF subscan after OTF
   2 a script that processes the OTF data as explained in the
     requirements/specifications and according to the values
     of the variables listed above.
   3 a "post"-script. unused. 

at the end of the OTF scan, if the last subscan was not a REF:
--------------------------------------------------------------

   otfcal is called as above, except that:
   1      odp#refSubscan2 = 0     ! must be set to 0 (zero)


generate plots (TBD)
--------------------

1. will transfer the subscan data from the Abba backend
2. and execute the abbaToFits program to generate a Fits file in the directory /mrt-lx1/mrt/data/"project"/fits.

1. 4 MHz backend: the design is implemented. Currently, the 4MHz processor writes data to files on the file-server via NFS. This data is merged with the pseudo raw data files after the transfer to the file server as part of the data transfer software.(source: /mrt-ux1/usr/local/bin/b4merge.icn)
2. Abba: the process abbaToFits.py transfers ABBA data files via FTP and then executes a merger program abbaToFits. (sources in /mrt-lx1/mrt/data/src): TO BE CHECKED
3. Hera: TO BE CHECKED

1. it checks if new links are in /mrt-lx1/mrt/data/Pipe/raw/pODP
2. a script with the name of the observing procedure is executed
3. the link it removed

"odp#prescriptOTF.otf" 
----------------------

Template script showing how to assign values to the required
variables.  In this template the values are taken from the SIC
parameters &1 to &9; in a real case they should be set directly.

Up to 99 OTF subscans can be specified: 
odp#flySubscanList[1] = i
...
odp#flySubscanList[99] = j


"odp#processOTF.otf" 
--------------------

Process the OTF subscans as explained above:

For each OTF subscan in odp#flySubscanList (for each element of
odp#flySubscanList that is > 0) the scripts odp#prescript1OTF.otf and
odp#process1OTF.otf are called.


Scripts, Used Scripts, and Helper Scripts:
-----------------------------------------

odp#defineOTF.otf

    ensures the correct definition of the SIC GLOBAL variables

odp#examineOTF.otf

    makes a neat list of the SIC GLOBAL variables and their values

odp#plot1OTF.class
    (pre-prototype, unused)

    makes a plot

odp#prescript1OTF.otf

    sets values for processing of 1 OTF subscan

odp#prescriptOTF.otf

    sets values for processing of OTF subscans from one CRO cycle

odp#process1OTF.otf

    processes 1 OTF subscan
    the maximum number of OTF "dumps" is set to 1999 in this script;
    maybe this should be determined by another variable ...
    NOTE: THIS IS THE (ONLY) SCRIPT THAT DOES "REAL WORK".
          SHOULD BE CHECKED AND TESTED ON MORE DATA.

odp#processOTF.otf

    organizes processing of OTF subscans fromone CRO cycle


Implementation Notes:
---------------------

These scripts assume that a SIC logical "ODP:" is defined pointing to
the directory containing the scripts.

All names of these scripts, as well as the GLOBAL SIC variables they
use, start with the string "odp#".

Some overhead could be avoided by processing the CAL in
odp#processOTF.otf.  In the current version the CAL is processed in
odp#process1OTF.otf for each OTF subscan (and each backend). For
development and test purposes this has the advantage that all the
"real work" is encapsulated in one script.


Note: 
-----

The prototypes of these scripts are on mrt-ux1 in:
/users/astro/ungerech/ncs30m/onlineDataProcessing

1. separate calibration and further processing (data plot, pointing, ...)
2. start and stop OTFCAL for each subscan
3. check if calibration of data could also be done with calibrations done after an observation