4.) The automatic processing modes: AOT/Batch Processing
PIA also offers the possibility of processing data in automatic or semi-automatic mode. This allows for an easy and fast obtainment of reduced data, including the possibility of choosing among several alternative reduction modes. Start and end of the automatic processing are user-defined, and intermediate products can also be fully interactively accessed after the processing sequence is finished.Note that the buffers are by default not deleted during the sequence. To enable the processing of a large number of ERD files, an option exists to delete every individual ERD buffer after having processed its contents to the SRD level.
Under Sequences in the top PIA window, a button for AOT/Batch processing offers various alternatives:
- General : valid for all kinds of data, including non-AOT data, as calibration measurements initiated by the Calibration Uplink System. It also includes special processing (AOT dedicated) recognized automatically, as explained further below.
- -> ERD per RPID -> SCP: this is a special processing sequence for very long raster measurements, which separate the ERD measurement into pieces per raster point ID, process them individually up to the SCP level and merge them on that level. In this way accomodating too much data into the IDL memory is avoided. For the user the only difference is that after processing he will not find the measurement in the ERD and SRD stages.
- Absolute Photometry (AOT P05/P25): the absolute photometry mode, which, in addition to the source and the FCS measurements, can include a dark current measurement and/or a straylight measurement, require special treatment by the combination of results. While basically the processing is the same as in the general AOT/Batch processing, the straylight measurement is used for subtracting straylight (and including dark current) from the FCS calibration measurement, and the dark current measurement is used for a more accurate dark current subtraction from the source (if a straylight measurement is missed in the sequence, the DC measurement is also used for subtracting dark current from the FCS measurement).
- Sparse Map (AOTs P17/18/19 / P37/38/39): In this case the calibration measurements performed at the beginning and the end of such a sequence are taken (in the way defined by the user, see explanation further below) for calibrating all observations lying in between.
- Serendipity: Special sequence designed for the batch processing of serendipity observations.
When
starting any of the options but the second one (which runs with pre-defined
ERD begin and SCP end level) a first level definitions widget pops up,
as given in Figure 1.
Start and end level of processing can be chosen. In addition the AOT
recognition mode can be switched on, allowing for proper processing in
the case of absolute photometry and/or sparse mapping within the several
files / measurements which will be sent together to batch processing.
From this menu the choice of getting a final summary table can be made.
This choice only makes sense if the end level corresponds to the AAP level,
and for non raster measurements.
After these very first choices a second decision panel will pop up,
which contains several definitions and choices to be made by the user.
Figure 1: The batch processing top decision widget.
Figure 2 shows the sequences selection menu. It is divided into several sections:
- input section, from which the user decides to start with ERD FITS files or with measurements in buffer. This choice is only active if the top level decision was to start from ERD. In any other case it is mandatory to start with measurements in buffer.
- The ERD buffers can be kept or deleted after having being processed. This option is useful when working with many measurements, which could occupy too much of the internal memory.
- Selection section: selections as given from the different main level interactive windows can be also obtained by the time of starting, if the corresponding buttons are pressed on.
- Corrections section: a list of the corrections to be applied to the data with its corresponding level of reduction can be obtained by clicking the buttons for the different reduction levels given here. All corrections can be activated / deactivated. The parameters taken for the various corrections are the default ones, or the modified ones, in case the user has accessed the menus for modifying them within this PIA/IDL session (on each level under Testing in the main window). Buttons for modifying while running this procedure are also included, as shown in Figure 3, which is the complete menu (if the user has chosen all three levels).
- Processing section: here special choices for data reduction can be made. The defaults are selected by start.
- Saving section: PIA internal files can be saved at the time of processing in batch mode.
- Responsivity section: For deriving in-band powers several options are offered:
- showing the various alternatives (for taking averages or medians of FCS1 or FCS2) asking for an interactive choice,
- using the signal average from the FCS1 for deriving the actual responsivities, or
- using the signal average from the FCS2 for deriving the actual responsivities, or
- using the average of FCS1 and FCS2 (by chopped calibration measurements) for obtaining the actual responsivities, or
- using default responsivities, thus ignoring the calibration measurement which may be present.
- interpolating linearly the responsivities obtained,
- using the average of both, or
- using only the last one.
- Detector choice: this makes a pre-selection of files / measurements to be used in the sequence.
FIGURE 2: The batch processing main widget
FIGURE 3: The batch corrections menu
The several choices which can be performed do not avoid certain automatic decisions taken by this mode, which make out of it an "intelligent" pipeline, capable of handling with all types of observations at the same run. The recognition of the AOT mode for proper handling of the absolute photometry observations (in all their possible variations) or of the sparse mapping observations are examples (as mentioned at the beginning of the section). The peculiarities of the "ramps pattern" data reduction of chopped observations are also taken into account. A choice of having the signal linearization performed on the SCP level (which should be the default) will not be followed by "ramps pattern" reduced chopped observations (if this option is taken). Signal linearization and consequently previous reset interval correction and dark current subtraction will be performed on the SRD level in this case.
| Date | Author | Description |
|---|---|---|
| 09/07/1997 | Carlos GABRIEL (ESA/VILSPA-SAI) | First Version (V6.4) |
| 13/02/1998 | Carlos GABRIEL (ESA/VILSPA-SAI) | Update (V7.0) |
| 24/08/1999 | Carlos GABRIEL (ESA/VILSPA-SAI) | Update (V8.0) |
| 02/11/1999 | Carlos GABRIEL (ESA/VILSPA-SAI) | Update (V8.1) |
| 17/03/2000 | Carlos GABRIEL (ESA/VILSPA-SAI) | Update (V8.2) |
| 27/06/2000 | Carlos GABRIEL (ESA/VILSPA-SAI) | Update (V9.0) |