Next: 4.4 Auto-Analysis Processing Steps Up: 4. Data Processing Previous: 4.2 Quality Check of

4.3 Derive-SPD Processing Steps

4.3.1 Introduction

The inputs to Derive-SPD (SPL) are the Compact Status History file (LSTA), science ERD files, the housekeeping ERD file (LWHK), the executed observation history files (EOHI and EOHA), and various calibration files. The science ERD files consist of LGER for grating scan data, LLER for FPL scan data, LSER for FPS scan data, and LIER for illuminator flash data.

The outputs of SPL are the standard processed data file (LSPD), the illuminator processed data file (LIPD), and the Glitch History file (LWGH). The LWGH file is for information only, and is not used during any further processing steps.

SPL is driven by the contents of the Compact Status History file (CSH) for the selected observation. The LWS CSH file is named LSTA. The LSTA file identifies the regions of data taken in an observation with the grating, FP, or illuminator.

4.3.2 Construction of ramps and discarding unusable readouts

The first stage of SPL reads in all records from the currently open science ERD file that correspond to one ramp of data for all ten detectors. The LSTA file specifies which of the science ERD files the data is read from.

The start of a ramp is indicated by a detector readout which has its most significant bit set. The expected number of readouts per ramp is then read in from the housekeeping ERD file (LWHK).

The time key of each readout is checked as it is read in to identify any periods of missing data and to adjust the ramp contents appropriately.

After the ramp has been read in, some of the readouts have to be discarded for the following reasons:

Readouts immediately following a reset pulse (the start of the ramp) are affected by the reset and must be discarded. The amount of data to discard for each detector is specified in the LCDT calibration file. It is currently 55ms for each detector reset.
The last point of each ramp is deleted. This is to equalise the length of all ramps, since the last ramp in each telemetry format is slightly longer than the other ramps.
Any readouts taken while the grating or FP are still moving should also be discarded. The LCDT file also contains the values for these discard times. The discard time for grating movements is 70ms and the discard time for the FP movement is 5ms.

The number of points discarded for the above reasons are written as keywords into the header of the LSPD file (see Section 7.2.5 for details).

4.3.3 Conversion of readouts to voltages

Before the raw detector readouts are converted into voltages, any invalid points which are outside the valid range for the analogue amplification chain are discarded (see more explanations about this in Section 5.8). The valid range is specified in the LCAL calibration file. Note that this is NOT the same as the saturation of the detector, which is corrected later in the processing chain.

The number of readouts discarded for this reason are written as keywords into the header of the LSPD file. See Section 7.2.5.

For each raw detector readout (in digital units; DN), the conversion to voltages is performed using the formula:

$\begin{displaymath} V = \frac{A (D - D_{off})}{G_{gain,det}} \end{displaymath}$

(4.1)

Where:

is the detector voltage at the input to the JF4 amplifier in volts.
is the readout to voltage conversion factor in volts per readout unit.
is the detector readout value.
$D_{off}$ is an offset to the detector readout value.
$G_{gain,det}$ is the analogue amplifier gain factor for the current gain level and detector. The gain level is in the range 0-7. The level is stored in bits 12-14 of the detector word.

The values for and $D_{off}$ can be found in calibration file LCVC (see Section 7.3.1 for details). The gain for the analogue amplification chain is read from calibration file LCGA (see Section 7.3.1) using the gain level (0-7) read from the detector word.

Finally the voltage as derived using Equation 4.1 is divided by the gain factor of the JF4 for the appropriate detector to reconstruct the voltage at the input of the JF4's. The JF4 gain factor can be found in calibration file LCJF (see Section 7.3.1).

4.3.4 Flag saturated ramps

In previous versions of the pipeline saturated points had to be removed at this stage. A saturated point is one where the voltage exceeds the threshold where the model of the detector behaviour breaks down. This model has now been replaced by the $\Delta V/\Delta t$ method, as described in Section 5.5. It is therefore no longer necessary to throw away saturated points. However, it was thought desirable to continue to flag any ramps which contain saturated points. This stage therefore checks all of the points in the ramp and if one or more points exceed the saturation threshold then the ramp is flagged as saturated in the LSPD status word (see Section 7.2.6). The saturation thresholds can be found in the LCDB calibration file (see Section 7.3.1).

The number of saturated points and the number of ramps containing one or more saturated points are written into the header of the LSPD file (see Section 7.2.5).

4.3.5 First level deglitching

4.3.5.1 Introduction to glitches and spikes

Glitches are caused by the effects of cosmic ray particles on the detectors (see Section 2.6). There is roughly one glitch every ten seconds per detector during the normal period of LWS operation. These energetic particles cause a sudden jump in the ramp voltage, due to a quantity of charge being dumped on the integrating amplifier. They also cause a change in the detector responsivity which affects the following ramps.

`Slow' glitches are glitches where the jump in voltage covers more than one readout value.

In addition to these `positive' glitches, `negative' glitches have also been found. These cause a sudden decrease in the ramp voltage, rather than an increase. They are thought to be due to hits on the FET. Negative glitches do not appear to affect the detector responsivity.

Before launch it was anticipated that `spikes' in the analogue amplification chain may also need to be located and removed. They cause a single detector readout to have a much larger value than normal. Subsequent readouts are unaffected and there is no effect on subsequent ramps. However, no real spikes were seen in the data when the satellite was in-orbit. The spike removal was switched off as all of the spikes detected were actually caused by the effects of glitches. A modified spike detection remained operational, but it would be more accurate to describe it as an `anomaly' detector, rather than a spike detector. The anomalies which were detected could be caused by real spikes, but they are more likely undetected glitches, or the effects following glitches above mentioned.

Statistics related to glitches and spikes are written into the header of the LSPD file. See Section 7.2.5 for details.

4.3.5.2 Detection method

The following list describes how glitches and spikes were detected. Note that glitch detection is only performed on the section of ramp after the discarding of data due to the reset pulse etc. Any glitches which occur in this discarded section of a ramp are not currently detected.

Perform point-by-point differentiation. This consists of finding the gradient in volts per ITK unit (ITK is the unit of time, defined in Section 7.2.1) between each point and the following point, and each point and the point two places away.
The mean and standard deviation of the first set of differentiated points are then calculated. The two largest values from the set are excluded from these calculations. This excludes any large jumps which may be caused by glitches.
Each point in the two sets of differentials is checked and those more than N standard deviations away from the mean are flagged as outliers. It is also recorded whether the point is an outlier above or below the mean. The value of N is specified in the LCD1 calibration file.
The outliers are searched to find the patterns expected from glitches or spikes. This is described in more detail in the following section.
If a glitch is detected by this step then the next three points are not checked for glitches. This is because it has been found that the effects of a glitch often caused a second, false glitch, to be detected shortly afterwards.
No spikes detection is done for the remainder of a ramp following a glitch. This is because it has been found that the effects of glitches caused lots of false spikes to be detected.
The heights of any glitches and spikes detected are estimated. The height of a spike is estimated by subtracting the voltage of the previous point from the voltage of the spiked point. The expected voltage increment due to the ramp slope is then subtracted from this value.
There is a special case for the first point in the ramp, since there will be no previous point. In this case the spike height is obtained by subtracting the voltage of the point following the spike from the voltage of the spiked point. The expected voltage increment due to the ramp slope is then ADDED to this height.
The height of a glitch is estimated by finding the difference between the point at the glitch location and a point 3 places ahead. This is to cope with slow glitches, or glitches that have noise. If the second point is beyond the end of the ramp then the last point in the ramp is used.
The expected nominal ramp increment over the time period between these two points is calculated and subtracted from the glitch height.
The heights are compared with the height of the ramp and any below a threshold height are rejected as described below. This is to reject genuine glitches and spikes which are insignificant with respect to the ramp. It also provides a method of rejecting spurious glitches and spikes.
For spikes the fractional height with respect to the height of the ramp is calculated. The height of the ramp is simply the voltage of the last point in the ramp minus the voltage of the first point in the ramp. Only those spikes with fractional heights above the threshold specified in the LCD1 calibration file are accepted.
For glitches the same procedure is performed, except that the glitch height is also subtracted from the height of the ramp. This should give the height of the ramp as if no glitch has occurred. There is a separate threshold level specified in the LCD1 file for the fractional heights of glitches.
Note that these calculations have assumed that there is only one spike or glitch in the ramp.

Note that the `location' of a glitch is understood to mean the point before the outlying point(s):

                            *-*-*   
                           /          
                        *-*
                          ^
                          Glitch location

Using test data it has been found that `slow' glitches are often detected only on the second point of the glitch:

                              *-*   
                             /     
                            *   <--------- Glitch detected here
                           /
                        *-*

In order to cope with this, the point at the glitch location is always discarded. For normal `fast' glitches this will mean that one possibly good point is thrown away.

4.3.5.3 Patterns expected from spikes and glitches

This section details the patterns which identify spikes and glitches. The following conventions are used:

OUT1 are the outlying points in the first differential array.
OUT2 are the outlying points in the second differential array.
A value of 1 indicates an outlier above N standard deviations from the mean.
A value of 1 indicates an outlier below N standard deviations from the mean.
An * indicates that the value at this location is not checked, so it does not matter if it contains an outlier or not.
The ramp point increases reading from left to right.

Glitches


  
   Positive glitch at point n in ramp
   ==================================
  
  
  
     ramp point    n-1   n              ramp point  n-1   n
  
           OUT1    *    1        OR           OUT1   *    1
           OUT2    1    *                     OUT2   *    1

The second of these checks tends to catch the `slow' glitches, which cover more than one point.

For the first point in the ramp only the second of these tests for positive glitches is done, as there are no previous points to check.


  
   Negative glitch at point n in ramp
   ==================================
  
     ramp point    n-1   n              ramp point  n-1   n
  
           OUT1    *   -1        OR           OUT1   *   -1
           OUT2   -1    *                     OUT2   *   -1

No check is done for negative glitches at the first point in the ramp. These will be recorded as a positive `spike'. It remains to be checked whether this is the correct thing to do.

If the number of points in a ramp is NPOINTS, then these checks for glitches can only be done for n=1 to NPOINTS2, as there are only NPOINTS2 values for the second differential. This means that if the last point of a ramp is an outlier, then it will be reported as a spike, rather than a glitch. There is no way of telling the difference between a spike at the last point and a glitch.

Spikes


  
   Positive spike at point n in ramp
   =================================
  
     ramp point    n-1   n         
  
           OUT1    1   -1     
           OUT2    *    *

There is a special case for the first point in the ramp, as there is no previous point to check. In this case if OUT1 is 1 (i.e. a negative outlier) then this is regarded as a positive spike at this point.


  
   Negative spike at point n in ramp
   =================================
  
     ramp point    n-1   n         
  
           OUT1   -1    1     
           OUT2    *    *

It is not possible to distinguish between a negative spike at the first point in the ramp and a positive glitch at this point. Therefore, no check for negative spikes is performed for the first point. A negative spike at the first point will be reported as a positive glitch.

4.3.5.4 Glitch handling

The glitches identified using the method described above are removed in the processing. The way in which this is done is controlled by the values in the LCD1 calibration file. Note that the removal of glitched data is done after all glitches have been identified. This means that glitches which occur during ramps discarded because of glitches in previous ramps are still identified.

For positive glitches all of the ramp following the glitch is discarded, plus the two subsequent ramps. The section of ramp before the glitch occurred is still used, provided that it has at least the minimum number of points required for slope fitting (this value is specified in a file and is currently set to 10).

Negative glitches are handled in the same way as positive glitches, except that no ramps are discarded following the glitched ramp.

The deglitching performed during illuminator flashes is slightly different from the above description. See Section 4.3.7 for details.

The LSPD file also contains the `undeglitched' data, i.e. the results when there is no discarding of data due to glitches. The photocurrent for ramps discarded following a glitch are still available in the LSPD file, but are flagged as `invalid' in the status word.

Information about each glitch detected, including the time, the glitch height and the detector number, is written into the Glitch History file (LWGH). This file is for informational purposes only. It is not used as an input for any further processing steps.

4.3.6 Extraction of ramp slopes and conversion to photocurrent

Starting with OLP Version 7 the method used for the ramp extraction is the ` $\Delta V/\Delta t$ ' method described below. A more detailed description can be found in Leeks 2000, [24].

For each ramp of each detector, the points which have not been discarded by previous stages are processed as follows:

The ITK time key for each point is converted into time in seconds relative to the very start of the ramp (the point where the reset occurred).
A second order polynomial is fitted to the set of time and voltage values for the ramp. This is done using a standard least-squares algorithm.
The initial and final voltages, and are then estimated as follows:

$\begin{displaymath} V_{initial} = \alpha + \beta{t_{initial}} + \gamma{t_{initial}^{2}} \end{displaymath}$ (4.2)

$\begin{displaymath} V_{final} = \alpha + \beta{t_{final}} + \gamma{t_{final}^{2}} \end{displaymath}$ (4.3)

Where:
- $t_{initial}$ is the time of the first point which has been fitted.
- $t_{final}$ is the time of the last point which has been fitted.
- $\alpha, \beta, \gamma$ are the coefficients of the second order fit.
The value for $\Delta{V}/\Delta{t}$ is then calculated using the following formula:

$\begin{displaymath} \Delta{V}/\Delta{t} = \frac{V_{final} - V_{initial}}{t_{final} - t_{initial}} \end{displaymath}$ (4.4)
The values of $\Delta{V}/\Delta{t}$ are then converted into photocurrents as follows:

$\begin{displaymath} I_{ph} = (\Delta{V}/\Delta{t}) . C_{JF4} \end{displaymath}$ (4.5)

Where $C_{JF4}$ is the capacitance of the JF4 for this detector, which is obtained from the LCJF file.

In addition to the detector photocurrent, the `rms of the detector ramp fit' is also calculated. This gives a measure of how well the points in the ramp were fitted by the second order polynomial. In previous versions of SPL this value was called the `uncertainty' in the photocurrent. However, this was an inaccurate description and this value should not be used as an uncertainty. The rms of the detector ramp fit is calculated as follows:

$\begin{displaymath} rms = C_{JF4} \sqrt{\frac{\sum [V(t_{i})-F(t_{i})]^2}{N_{points}}} \end{displaymath}$

(4.6)

where:

$V(t_{i})$ is the voltage at time $t_{i}$ .
$F(t_{i})$ is the value of the fit at time $t_{i}$ .
$N_{points}$ is the number of fitted points.
$C_{JF4}$ is the capacitance of the JF4 for this detector.

The calculated photocurrent and rms of the fit are now written into the LSPD file, together with a time key, grating and FP positions and other information. The time key assigned is the ITK time value of the start of the ramp.

If for any reason the photocurrent has not been calculated for this ramp then both the photocurrent and uncertainty will be set to zero. The most common reason for this is a glitch which has caused all of the data to be discarded. The status word should also indicated that this point is not valid (see Section 7.2.6).

4.3.7 Illuminator processing

For calibration purposes each observation includes two or more periods when the internal illuminators are used. The data from these `illuminator flashes' are identified by SPL, processed, and the results written into an LIPD file. This file is then used as an input to AAL.

Each illuminator flash consists of a `dark current' measurement (which is stricly speaking a dark signal measurement, see Section 4.4.2), followed by a sequence in which different illuminators are flashed at one or more different levels, followed by another dark current measurement. For grating scans, at least two of the illuminator flashes are `closed' flashes, where the FP is moved into the beam. This removes the source from the beam and means that the dark current measurement during the illuminator flash is a measure of the dark current/straylight. For FP scans all flashes will be `closed' flashes.

The processing of the ramps in illuminator flashes is identical to the processing of ramps of science data, as previously described. The only difference is in the handling of glitches. The LCD1 file contains a separate set of parameters which control the handling of glitches during illuminator flashes. The current setting of these parameters (Version 8 of the LCD1 file) means that the whole of a glitched ramp will be discarded, but no subsequent ramps are discarded.

After each ramp in the illuminator flash has been processed it is written into the LIPD file. The LIPD file is analogous to the LSPD file, except that it contains data from illuminator flashes rather than science data. The LIPD file contains the photocurrents for each ramp for each detector, plus auxiliary information such as the value of the illuminator commanded status word and the illuminator current.

Next: 4.4 Auto-Analysis Processing Steps Up: 4. Data Processing Previous: 4.2 Quality Check of

ISO Handbook Volume III (LWS), Version 2.1, SAI/1999-057/Dc