2.) Learning Chapters

2.1) Basic Description of the PHT Data Reduction

This chapter gives a survey of the data reduction steps which can be performed by the PHT Interactive Analysis and of how they are implemented in the PIA software. While the Guided Tour presents a step by step sample session, the present chapter aims at giving some insight into the steps which need to be performed.

---

List of the Data Reduction Steps

The data reduction consists basically of three parts, namely:

• removal of instrumental effects
• flux calibration
• combine several data sets (images/spectra/photometric & polarimetric tables).

In general, the data in a PHT observation consist of a measurement on the internal Fine Calibration Source(s) (FCS) and one or more measurements of the astronomical target. Normally, an FCS measurement is reduced in the same manner as the astronomical data up to the signal-per-chopper-plateau level and then used to compute responsivities which are applied to the astronomical data during the power calibration step. The responsivity computation is the final step in the reduction of FCS measurements.

In detail, the data reduction will consist of some of the following steps:

1. Convert read-out digital data into analog data
2. Correct for non-linearity
3. Correct for cosmic particle hits (glitches) on read-out level (single threshold ramp deglitching) OR
4. Correct for cosmic particle hits (glitches) on read-out level (two threshold voltage differences deglitching)
5. Determine signal (in V/s) per integration ramp OR
6. Determine "signal pattern" (in V/s) (for chopped measurements)
7. Correct for cosmic particle hits on the signal level (signal deglitching)
8. Analyse/edit data to minimise the effect of drifts in detector responsivity
9. Determine signal per chopper plateau OR
10. Calibrate "signal pattern" (for chopped measurements)
11. Correct for reset interval effect
12. Perform dark current subtraction
13. Linearize signals
14. Subtract straylight (for FCS measurements)
15. Calculate actual responsivity (for FCS measurements)
16. Perform vignetting correction
17. Subtract background within a measurement (for chopped measurements)
18. Correct for signal losses caused by chopping between different fluxes (chopper frequency dependence)(for chopped measurements)
19. Correct (graphically) for long term responsivity changes
20. Convert signal (V/s) into in-band power (W)
21. Subtract a measurement containing the background (externally) (for staring measurements)
22. Calculate flux density and surface brightness
23. Correct for pointing offset by PHT-S observations
24. Perform AOT-specific processing (e.g., multi-filter, multi-aperture photometry, spectrophotometry, polarimetry, or mapping)

In PIA appropriate steps are combined within the following procedures, often corresponding to a single main display or interface in PIA:

Read Edited Raw Data (ERD) files:

Process ERD:

Includes steps 2-6. Step 5 and 6 are alternatives for reducing the ERD to Signal per Ramp Data (SRD).

Process SRD:

Includes steps 7-11. Step 10 and 11 reduce alternatively the SRD to Signal per Chopper Plateau Data (SCP).

Process SCP:

Includes steps 12-20. SCP data are reduced to the Standard Processed Data (SPD) in step 20.

Process SPD:

Steps 21 and 22. Step 22 reduces SPD to the Astrophysical Applications data (AAP).

Processing AAP:

Step 23.

---

Description of the data reduction steps

The data reduction steps as realized within PIA are listed and described below. A detailed guide to applying them by using PIA is given in Processing Data with PIA - A Guided Tour. A description of physical effects affecting the data can be found in the IDUM . A detailed discussion of the algorithms used is given in chapter 4, Processing Algorithms.

Reading ERD files

PIA automatically converts all digital data into analog data and stores the data in the internal buffers. A buffer in PIA refers to a data structure in memory which can be loaded and operated on or written out to a file. Parameters from several conversion tables and a calibration file are used to convert the data into an analog form.

Processing ERD

Within this procedure the following steps can be performed:

• Correct for non-linearity:

 

 
 
 

The PHT detector output suffers from some non-linearity. Two effects contribute: a non-linear response of the Cold Readout Electronics (CRE) (relevant to all PHT detectors) and de-biasing effects (only for the low-biased detectors P3, C100 and C200). A linearization correction may be applied within PIA. It uses special calibration files and corrects for both non-linear effects at the same time.
 

• Correct for effects of cosmic particle hits using the single threshold deglitching per ramp:

 

 
 
 

A cosmic radiation event ("glitch") generally results in a jump of two consecutive read-out values, i.e. in a step within the "ramp". The signal to be derived is therefore affected. The disturbance is usually very short and the slope of the ramp after the glitch is similar to the slope before it. In this case, it is not necessary to discard the whole ramp, because PIA is able to recognise the glitches/jumps and to correct affected ramps using the single threshold ramp deglitching procedure. This level of deglitching is sometimes also referred to as read-out deglitching. Ramp deglitching can be performed interactively so that the user can tune the parameters of the procedure to optimise the performance for his/her data. The algorithm works on a per ramp basis and therefore requires a sufficiently large number of read-outs per ramp to be able to use a statistical analysis.
 

• Correct for effects of cosmic particle hits using the two thresholds deglitching method:

 

 
 
 

In many cases a cosmic hit produces -in addition to the disturbance described above- a momentaneous change in the detector response, with some relaxation time. In this case a distribution of subsequent voltage differences between read-out pairs shows a dacaying tail, until the level before the glitch is reached. Taking this into account an algorithm was proposed (Meyer et Haas) based on a search for statistically significant deviant points within the distribution of read-out pairs voltage differences using one threshold (eg 3 sigma) and the discard of these and all subsequent points until the corresponding voltage differences are within much more constrained limits (eg 1 sigma) to the average of the distribution. This algorithm takes as statistical sample a full chopper plateau or raster point (all signals which should be the same), thus is not limited by the length of a ramp.
 

• Determine signal:

 

 
 
 

The output of each of the integrating PHT detectors consists of the read-out value as a function of time. A number of non-destructive integrating read-outs (NDRs) are followed by a destructive read-out at which the integration plateau is reset, in principle, to zero. Each sequence of NDRs defines a ramp (voltage as a function of time); the steeper the ramp, the higher the signal (and the power on the detector) is. Thus the signal values are derived by fitting a 1st order polynomial to each ramp. Please take into account that the possibility of fitting higher order polynomials in PIA is only for calibration inverstigation purposes and should not be performed in normal data reduction, since there is no corresponding calibration.
 

• Determine signal pattern by chopped measurements:

 

 
 
 

Chopped measurements are affected by a continuous flux change, which introduces a typical signal pattern within a chopper plateau. By the method of signal pattern determination (default for P1, P2, P3, C100 and C200 chopped measurements from PIA V8.0 on) the differential voltages between two consecutive read-outs of all the off-source chopper plateaux and all the on-source chopper plateaux are computed together, so that only two sequences (off and on) of a number N of signals in V/s is derived (by default 4 per chopper position). The temporal relative positions of the differential signals within the chopper plateaux are preserved, so that actually only a compression of all off source measured signals into one off source chopper plateau of N signals is obtained. The same with the on source positions. The characteristic temporal behaviour is preserved but with a very much reduced noise level. This makes possible a better calibration of the flux losses introduced by the continuous flux changes performed in this mode.

Some editing of the data is generally necessary because some of the read-outs are disturbed. The default read-out selection is to discard the first and last read-out values of each ramp because they may seriously suffer from electronic effects and should usually be discarded. Additionaly, the first n read-outs in every long integration ramp can be disturbed. By default 10-15% of each ramp are discarded in order to minimise the effects of these ramp disturbances without affecting the signal-to-noise ratio more than necessary. For a careful analysis the user should inspect (some of) the ramps and should optimise the selection of read-out values before deriving signals.

Additionally the user may want to have a look at the detector temperature curve which is automatically provided by PIA and which should be reasonably flat - if it is not, the signal values may be influenced by temperature fluctuations.

The user should also look at the dynamic range of the read-out values. The maximum dynamic range lies between about -0.85 and +1.09 [V] depending on the detector. Saturated read-outs will be discarded by PIA, while a poor usage of the dynamic range (e.g. -0.8 to -0.7 [V] leads to higher noise in the results. This cannot be corrected for once the measurement has been performed, but should be noted by the user.

After this processing step the data consist of the signal as a function of time.

Processing SRD

The next step is to combine all the signal values into a representative signal for each chopper plateau (note that a staring observation is considered a single chopper plateau). PIA can display information on which chopper plateaux go with which signals.

Within this procedure the following steps are performed:

 
• Deglitching on signal-per-ramp level:

 

 
 
 

Some cosmic particle hits on a PHT detector are followed by a temporary increase in the responsivity, thus affecting a series of read-outs and therefore one or more signals. On a statistical basis PIA can also discard such signals. This signal deglitching can be done interactively so that the parameters for the algorithm which recognises affected signals can be tuned by the user.
 

• Detector transients handling:

 

 
 
 

The detectors show transient drift/memory effects on timescales of the order of a few seconds up to a few minutes (in the worst cases). Thus the true signal values corresponding to the incoming flux are not obtained immediately, rather they are approached asymptotically after some time. Fortunately, in many cases this behaviour is a monotonic rise; however, a considerable number of pathological cases show an overshoot followed by a slow decay.

To account properly for detector drifts is one of the most difficult tasks of the data reduction. PIA provides different ways of handling this problem:

• stability analysis and drift recognitionprocedure:

based on a trend analysis of the signal values, it discards signals from the beginning of the measurement until a stable region is found or the number of signals is below a certain limit. The parameters for this analysis can be optimised by the user. The results of the analysis are kept in text files which can be inspected.
• transients modelling:

several functions and fitting routines are offered, trying to correct for drift effects using physical and empirical models. Again, the user has the choice of function and parameters within a full graphical interface while default-optimised values for any ISOPHOT detector are proposed, which reflect the experience of the ISOPHOT scientists. The user has also the possibility of determining if the fits results are used for overwriting the average signals per chopper plateau (as described in the next point), taking into account the fit quality.
• Determine signal per chopper plateau:

 

 
 
 

For every chopper plateau in a measurement averages and medians of the corresponding signals are calculated. The results are combined with other reduced variables in the signal-per-chopper-plateau (SCP) buffer.
 

•  Calibrate signal pattern (only for chopped measurements):

 

 
 
 

Instead of calculating a simple average per compressed chopper plateau (one on- and one off-source) the signal difference between mean on- and off-values (different determination depending on the detector used) is used for computing a signal difference, corrected according to calibration tables. This difference is used for recompute a mean on and a mean off signal which is passed to the SCP level. Please note that two corrections must be performed before: the reset interval effect correction and the dark current subtraction. (Both can be performed on the SRD and on the SCP level by PIA, we recommend however to apply them on the SCP level by staring measurements or by chopped measurements which are processed in the traditional way [not by the "signal pattern" recognition]. They are described under the SCP corrections)
 

A signal selection (editing) is generally necessary because the first ramp of every chopper plateau is disturbed. While the chopper is changing its position, the reset is interrupted. Therefore, such a reset shows a behaviour different from the 'normal' ones, affecting the signal value. By default this first signal is deselected. The PIA user has the option of deselecting any signals to cope with other effects which could affect the signal per chopper plateau, such as initial signal drift, etc.

Processing SCP

The next step is to perform the signal to optical power conversion, in the case of calibration (FCS) measurements.

Since the actual responsivity should usually be used for the power derivation, we describe the processing of FCS calibration measurements first, including the data corrections which can be applied.
 

• Correct for reset interval effect:

 

 
 
 

It has been found that the signals obtained from the CRE output are partly dependent on the used reset interval (ramp integration time) for sampling. This correction normalizes to the output one would have obtained if measuring with an integration time of 1/4 of a second. This correction must be applied (since PIA V7.0) because the whole calibration has now been determined accordingly. Although it can be also applied on the SRD level, we recommend to apply it on the SCP level, prior to the dark current subtraction (except for the case of "signal pattern" analysis by chopped measurements).
 

• Perform dark current subtraction:

 

 
 
 

The signal per ramp contains a more or less constant component coming from the dark current in the detectors. Dark current values per pixel and detector are contained in calibration files. They can be used for subtracting this component from every measurement. This correction is also more efficiently applied here than to the former level of reduction (SRD).

Since PIA V7.0 the dependence of the dark current on the orbital position of ISO is included within the calibration data and the corresponding algorithms. Several interpolations are possible, they are based on one value per signal, per chopper plateau, per raster or per AOT.
 

• Perform signal linearization:

 

 
 
 

A dependence of the detectors response on the illumination level has been established. Since all measurements are affected by this effect (also the ones used for establishing the Responsivity, the FCS measurements) the linearization of the detector response is better handled by a linearization of all signals, as recorded by the detectors. This correction can be applied using PIA also both on the SRD and the SCP level, but except for chopped measurements reduced using the "ramp pattern" procedure it should be better performed on the SCP level.

• Straylight subtraction:

 

 
 
 

When the optical path to the detector is directed to one of the internal Fine Calibration Sources (FCSs), the telescope field of view is pointing to the sky. This yields a certain amount of straylight on the detector, which affects the signal measured from the FCS source. It is very difficult to correct for this effect. Nevertheless, PIA offers a way to at least estimate the influence of such an undesired effect on the responsivity calculation. This correction is optional; it is most important when the sky is fairly bright.
 

• Responsivity Calculation:

 

 
 
 

The responsivity of the detector(s) is derived from the FCS calibration measurement. It is assumed that the responsivity from one FCS measurement can be applied to all the data in a given AOT, which is a reasonable assumption if the AOT is not too long and the flux changes within the AOT are not too drastic. For the responsivity computation, the mean signals of the FCS measurements are compared to the (in principle) known optical power on the detector corresponding to the electrical power applied to the FCS. The relationship between the optical power and the electrical power is contained in calibration files.

For astronomical data at the SCP Level the processing is described below, including the relevant corrections:

• Vignetting correction:

 

 
 
 

When the chopper throw is large, all PHT detectors can be telescope- vignetted. In the case of the PHT-P detectors the effect depends on the aperture used. The correction for this effect is taken from calibration files.
 

• Internal background subtraction:

 

 
 
 

"Internal" background subtraction refers to the calculation of the source signal from an on-source/off-source chopped measurement. This can also be done after the data have been calibrated to in-band power.

The result of a background subtraction is a single value (e.g. mean source - background), which can be written into the SCP buffer as a subtracted pseudo-measurement with a single chopper plateau. This buffer must be selected for processing in order to use the subtracted data for further reduction.

Selection/de-selection of chopper plateaux is possible within PIA and can be used to control what data are included in the source-background calculation. Editing of the data at this level may be useful when anomalies affect one or several chopper plateaux; this can usually be determined by visual inspection of the (non-background-subtracted) SCP data.

In most cases external background subtraction (using a separate background measurement) should be done at the SPD level since the responsivity may have changed between execution of the source and reference observation. However, it is possible to do the subtraction on data reduced only to SCP level, which may be desirable in cases where the same (e.g. default) responsivity must be used to derive the optical power for both source and reference observations. Since external background subtraction is normally performed on data reduced to the SPD level, this is described in detail below.
 

• Chopper frequency dependence of a subtracted signal:

 

 
 
 

[This correction which should still be applied to chopped measurements which have not been "signal pattern" reduced, is now obsolete in the default processing, since the calibration of the signal pattern includes already the correction]

The subtracted value from an on-source/off-source chopped measurement depends in the first approximation on the chopping frequency used (and in a second order on the signal difference between both levels). Empirical values have been determined which correct for these signal losses. One should only make use of these correction factors in the case of subtracted chopped measurements (PIA enables the correction only in this case) and if no drift modelling has been applied to the data.
 

• Long term responsivity correction (baseline drift):

 

 
 
 

By very long PHT measurements a baseline drift is often observed, which can be related to slow changes in the detector responsivity. While signal drift modelling on a short timescale can be performed using diverse functions (see transients modelling), there is no easy approach for long term drifts . In some cases, eg mapping using the P32 mode, it is impossible due to the superimposed signal changes by chopping to diverse positions. A very user interactive approach was chosen for correcting in these cases, based on the judgement by eye-balling. The user has the possibility of defining several points within a measurement which should have a similar flux but show different signals. From these points factors are derived for correcting the measurement, interpolating them for every point in the chosen distribution.
 

• Power calibration:

 

 
 
 

To convert signal per chopper plateau in V/s to in-band power in W it is necessary to know the responsivity of the detector, which varies during a revolution. There are two possibilities of which responsivity to use when doing the calculation: the default detector responsivities as contained in general calibration files, or the actual responsivities, as calculated from the associated FCS measurement and described above. While this second alternative should always be better, default responsivities may need to be used in special cases, such as failed FCS measurements. In the case of PHT-S default responsivities are the only choice, since there are no FCS measurements incorporated into the observations.

The detector response varies for certain detectors with a clear tendency within an orbit due to the continuous radiation to which the detector is exposed. This fact is reflected in orbital position dependent default responsivities, which have been established and are used from PIA V7.0 on. A value per measurement or per AOT can be computed by interpolating within the numbers present in the calibration files.
 

• Using "external flatfield" corrections: this option, which can be disabled, accounts for the inhomogenous illumination of the array detector pixels by the sky. Furthermore, it contains also filter-to-filter transmission differences so that it is absolutely necessary, when computing in-band-power for a certain filter, to use the responsivities measured by a different filter.


From PIA V8.0 on, the power calibration leads directly to the AAP level, thus contain the optical corrections described under the next point. Nevertheless, SPD level data is produced and can be inspected by using the dedicated GUIs. Although it would not be necessary at all by pure PIA data processing, we keep in this way compatibility to the PHT OLP pipeline, which only knows SPD as intermediate data processing stage.

Processing SPD

Now we only have to apply the optical corrections (filter transmission, etc.) to obtain flux density and surface brightness from the in-band power. External background subtraction is normally performed on the SPD and internal background subtraction can also be done at this stage. Both are also possible under the AAP level.

• External background subtraction:

 

 
 
 

A background measurement from another observation can be subtracted from a source measurement. The result of this subtraction can be stored in the SPD buffer as a subtracted (pseudo-) measurement for further reduction. This subtracted measurement must be selected for processing if it is to be used for subsequent data reduction.
 

• Flux/Surface Brightness calibration:

 

 

When computing Jy or MJy/sr, PIA applies an optical correction to the in-band power (in W). This optical correction takes into account the known properties of filter, aperture and other optical elements (stored in Calibration Files).

Processing AAP

The AAP processing (Astronomical Analysis) is completely AOT dependant, except for the background subtraction (internal or external, which can be applied at this level, but was already explained under the former points) and for the pointing correction in the case of PHT-S observations:
 

Correction for pointing offset by PHT-S observations:

Due to the small pixel size of PHT-S (24"x24") and to the sharply peaked footprint shape, already a few arcsecond off-centre positioning may lead to both significant intensity changes as well as to spectral shape alterations. Footprint matrices for every PHT-S pixel can be used for correcting depending on the established offset (see section 3.3.10. Pointing correction for PHT-S).

It is at this stage that the specific data reduction according to the final observation objectives takes place. PIA offers special data reduction modes for every type of observation, as listed below:

• Multi-filter photometry:

 

 
 
 

The results from different measurements and detector pixels can be combined, yielding photometric tables and plots of flux density/surface brightness versus wavelength, depending on the choice of point or extended source photometry. The possibility of performing colour corrections is included, for which the user has a choice among a black-body spectrum of a user's chosen temperature or a power-law spectrum of a user's chosen index.
 

• Multi-aperture photometry:

 

 
 
 

Measurements using the same filter but different apertures can be combined, yielding curves of growth (in Jy and MJy/sr), both as tables and plots. In addition, colour corrections are possible here, as they were described for multi-filter photometry.
 

• Spectro-photometry:

 

 
 
 

A PHT-S spectrum of flux density/surface brighntess per raster point and chopper plateau can be analysed by fitting lines and continuum. Fit parameters can be varied and regions of the spectrum defined. Plots and tables of the spectra can be also obtained.
 

• Mapping:

 

 
 
 

Maps in RA/DEC can be made from the surface brightnesses (or fluxes) and corresponding astronomical coordinates of a mapping measurement. PIA allows flexibility in mapping procedures and parameters. E.g. the pixel size of the map, detector pixel selection, automatic flat fielding, two- and three-dimensional map display, profiles, map zooming, colour table selection, etc. can be varied.
 

• Polarimetry:

 

 
 
 

Measurements using different polarizers can be combined. Polarimetry tables and plots can be obtained. In addition, colour corrections are possible here, as they were described for multi-filter photometry.

---

 

Chapter history:

Date	Author	Description
13/05/1996	Martin Haas (MPIA) / Carlos Gabriel (ESA-SAI)	First Version
19/07/1996	Carlos Gabriel (ESA-SAI)	Revised
26/02/1997	Carlos Gabriel (ESA-SAI)	Updated
03/06/1997	Carlos Gabriel (ESA-SAI)	Update (V6.3)
09/07/1997	Carlos Gabriel (ESA-SAI)	Update (V6.4)
12/02/1998	Carlos Gabriel (ESA-SAI)	Update (V7.0)
15/08/1999	Carlos Gabriel (ESA-SAI)	Update (V8.0)
02/11/1999	Carlos Gabriel (ESA-SAI)	Update (V8.1)