next up previous contents index
Next: 7.3 From SPD to Up: 7. Standard Product Generation Previous: 7.1 Introduction

Subsections


7.2 From ERD to SPD

Derive-SPDs' main input is Edited Raw Data (ERD), 24 Hz data containing detector readout (in bits) of the target's flux, of internal calibration exposures, of wavelength indicators, of dark currents and of housekeeping data. Within the ERD there is also information available from pointing or instrument status files. While processing, the pipeline software reads several Cal-G files.

For each detector, all data for one full reset interval are combined to a single current estimate at single wavelength and are then saved in the Standard Processed Data (SPD) file for that AOT. No attempt is made within Derive-SPD to average measurements of the same type (e.g. dark current measurements) on any time scale longer than one reset interval.

Each measurement for one reset interval of a given AOT is saved separately inside the same SPD product in chronological order of acquisition. The motivation for this is to offer to the observer the possibility of judging the stability and repeatability of SWS measurements. There is a one-to-one correspondence between AOTs and SPD files. That is, one AOT produces one SPD file.

A basic overview of the Derive-SPD processing is given in Figure 7.1, the ERD and SPD files are described in Sections A.2.1 and A.3.1, and each step in the processing is described in the following sections.

Figure 7.1: The Derive-SPD flow-diagram
\begin{figure} % latex2html id marker 5761\begin{center} \setlength {\unitleng... ...ar\put(12.0, 1.4){\makebox(0.8,0.4){dets}} \end{picture}\end{center}\end{figure}


7.2.1 Reading the ERD

The ERD data for the observation are read in. As discussed above, each AOT generates different types of data: astronomical measurements from the source, measurements of the currents for the grating and FP positions, measurements of the calibrators and dark currents. Each of these different types of data has a different physical meaning and serves a different purpose. Hence, the data must be flagged according to their type.

Each ERD data record is tagged with an Instrument Time Key (ITK) which points toward a unique record in the Compact Status History (CSH) file (see Section A.2.2). Cross-correlating the ERD ITK with the CSH, Executed Observation History per ICS (EOHI) and General Housekeeping (GEHK) files (both described in the ISO Handbook Volume I, [17]) allows a determination of the status of the SWS instrument at the time of the measurement and therefore of the type of the data recorded and the actual wavelength range covered.

The velocity of the spacecraft is obtained from the IRPH file. It is written into the header file and used later in the Derive-AAR processing stage.

Figure 7.2, top panel, shows 40 seconds of detector 16 ERD data from an SWS02 observation after reading in. This is the same SWS02 shown in Section 3.4.

Figure 7.2: Example data from various sections of the SPD processing. From top to bottom the first panel shows a portion of the raw ERD from detector 16 of an SWS02 observation. The second panel shows the same data after determination of data range, `midbit' subtraction, valid data selection and correction for RC constant. The next plot shows the ramps straightened by the pulse-shape correction. The final plot shows the SPD output. The top three plots are in bit values against time, whereas the SPD is in $ \mu $V/s against time.
\resizebox {10cm}{!}{\includegraphics{olp_mosaic.eps}}

7.2.2 Determination of the data range

This checks the data values and marks any out of limits. Values are OK when they are larger than 0 and less than 4095. Other data are flagged as out of limit. The values defining the ranges are kept in the calibration file Cal-G 4.

7.2.3 Subtraction of the `midbit' values

In SWS the detector readout voltages of between $ -$10 and $ +$10 V are converted into a bit number ranging from 0 to 4095. The `midbit' is the bit value that corresponds to 0 Volts. It is set to exactly half of 4095, i.e. 2047.5, for all detectors. This data is stored in the calibration file Cal-G 2A.

The precise value of `midbit' is not important. It only determines the global level of the samples and also, due to later steps in the pipeline (Section 7.2.5), the overall level of the signal. This overall level is corrected again when subtracting the dark current (Section 7.3.3).


7.2.4 Selection of valid samples

After each reset a number of samples should be ignored. They are too much affected by the reset pulse. This number is stored in the calibration file Cal-G 3. The values in the file are all set to 6 for OLP 10.

This is not formally a step in the DSPD processing as it is determined in each of the following steps what the valid samples are. It is presented here as henceforth all processing only affects the valid samples.

7.2.5 Correction of the integration ramp for the RC time constant

The amplifier chain contains a high pass filter. For a constant voltage gradient input the response of the amplifier is expected to asymptotically approach a constant value, i.e.

$\displaystyle {dV(t) \over dt} + {V(t) \over T } = {dV_o(t) \over dt }$ (7.1)

where $ V(t)$ is the output voltage at time $ t$, $ V_{o}$ the input voltage across the capacitor due to the accumulation of charges on the detector, and $ T$ the detector time constant. In order to later reconstruct the true photo-current, it is first necessary to linearize the ramps, i.e. determine the actual input voltages $ V_{o}(t)$ applied on the detector. Each detector has its own time constant, which is stored in the calibration file Cal-G 2.

Knowing the RC time constant $ T$, it is straightforward to integrate the above formula and derive the input voltage. The trapezium rule is used to calculate the integral.

$\displaystyle V_k^{corr} = V_k + \sum_{j=3}^k \frac{( V_j + V_{j-1} ) \cdot \Delta t} {2 \tau}$ (7.2)

The time constants stored in Cal-G 2, however, are expressed in terms of frequency, which relates to the $ \tau$ in Equation 7.2 as $ \tau = 1 / (2 \pi T )$.

The second plot in Figure 7.2 shows 40 seconds of detector 16 after the determination of data range, `midbit' subtraction and correction for RC constant. The main difference between the first and second plot is due to this correction.


7.2.6 Correction for the reset pulse effects

SWS defines the pulse-shape as the after-effect of the reset-pulse applied after each reset interval. It manifests itself as an exponential curve with typical decay times of 0.25 - 0.5 seconds and a typical amplitude of 20 - 50 $ \mu $V. The pulse-shape is an additive component to the integrated signal of the detector, while the impact on the observed ramp depends on the gain setting of the instrument. Significant corrections are to be expected for high gain observations only. The pipeline corrects for the pulse-shape on an observation basis. Consequently we do not rely on a fixed calibration file but use the data itself to do the calibration.

While the correction itself is a plain subtraction of the pulse-shape, the difficulty is to find the right pulse-shape. Per detector slopes are fitted to the readouts in each reset interval. The residuals contain the raw pulse-shape after-effects. To the residuals an exponential function is fitted, which determines the decay time for this detector in this observation. With this fixed decay time amplitudes of the exponential are determined, now for each reset interval individually. Subsequently the amplitudes are smoothed to reduce the noise and the resulting shapes are subtracted from the readouts.

More details of the method are explained in Lahuis 1998, [23] and Kester 2000, [20].

The third panel in Figure 7.2 shows 40 seconds of detector 16 after this processing step. Compared to the second panel the slopes have been straightened.

7.2.7 Removal of electrical cross-talk

Due to parasitic capacitance between neighbouring detectors any signal in one grating detector will `leak' to other detectors (primarily the adjacent ones) in the same array. This `leak' to the adjacent detector is of the order of 10%.

Assuming the detector response is a linear function of the intensity of the signal, a set of cross-talk matrices can be determined with constant correction factors. The readouts $ V$ of each individual detector $ j$ in the detector array may then be corrected by applying the following formula:

$\displaystyle V_{j}^{corr} = \sum_{i} C_{i,j} \cdot V_{i}$ (7.3)

The sum is over all detectors $ i$ that are within the same detector block as detector $ j$ (12 for the gratings and 2 for the FP). This cross-talk correction matrix, $ C_{ij}$, is held in the calibration file Cal-G 1. It was derived from data taken from charged particle radiation tests in the laboratory and verified in orbit.

There is little effect on the data at this processing step, so a plot is not shown.

7.2.8 Glitch detection

Glitches are sudden jumps in the output signal from a detector. How glitches are recognised in the data depends on which AOT the data is from. For SWS02, SWS06 and SWS07 the grating scanner does not move during a reset interval. Any sudden jump in the output of a detector during a reset interval is probably a glitch and can be recognised as such. For SWS01 however, the grating does move during a detector reset interval. It can scan across an entire line during a reset interval. If the line is strong enough there is a sudden increase in the signal level as the grating scans across it which might be mistaken for a glitch.

Therefore deglitching is handled differently in SWS01s compared to the other AOTs.

7.2.8.1 Glitch detection in general

Glitches are recognized in the differentiated outputs of each detector, $ \delta S_k = S_k - S_{k-1}$. For each detector, the median, $ m=$   median$ ( \delta S_k )$, and the median width, $ w =$   median$ ( \vert \delta S_k - m \vert )$, of the differentiated outputs are calculated. A threshold is defined as $ w_t = \max(\alpha \cdot w, w_{\min})$, where $ \alpha $ is a numerical factor and $ w_{\min}$ is a minimum width. Both of them are held in calibration file Cal-G 6. For OLP 10, $ \alpha $ equals 8 and $ w_{\min}$ equals 5. Samples are flagged as glitched when $ \vert \delta S_k - m \vert > w_t$.

As a large fraction of all glitches is spread over more than one sample, the samples preceding and following them are checked again. If they are larger than 40% of the threshold, they are also marked as glitches.

In this processing step glitches are only marked as such. During the slope fitting (Section 7.2.9) a model including slope and glitches is fitted to all samples within the reset interval.

If glitches are detected during a reset this is noted in the flag word. Also the actual number of samples contributing to the slope calculation is decreased by the amount of glitched points found. This number appears in the offset tag (SWSPOFFS) in the SPD and in the tint tag (SWAATINT) in the AAR.

The SWS Glitch History (SWGH) file contains the two largest, if any, glitches found in each reset interval. For a description of this file see Section A.3.2.

The accuracy of deglitching depends not only on how well a glitch can be identified, but also on the effects of the glitch with time (the so called glitch-tails, see also Section 9.2.4). Especially in band 4 and in the FP bands glitch tails are occurring frequently. They show as a sudden increase of the flux decaying in about 10 seconds.

Figure 7.3 shows the effect of the deglitching software, showing the ERD and the SPD processed with and without the deglitching software. Where the software has not been used the glitches in the ERD cause the slope in the SPD to vary rapidly in time. Where the software has been used it has identified where the glitches occur, allowing them to be taken into account when calculating the slopes. The resultant SPD is much smoother.

For a further discussion of glitches and their correction see Kester 1999, [19], and Wieprecht, Wiezorrek & Haser 2000, [43].

Figure 7.3: SPD processed with and without deglitching software. The top panel is the ERD (bit values against seconds), processed up until the glitch detection. Glitches can be seen where there are breaks in the sloping lines. The middle panel is SPD (in $ \mu $V/s) processed without deglitching. ERD ramps with glitches show very high SPD values. The bottom panel shows SPD processed with the deglitching software in place. The effect of the glitches has been removed, or substantially reduced. Glitch tails, however, are still present, but not at all glitches. The glitch at time 3576 e.g. has a conspicuous tail, while the even higher glitch at 3596 has none at all.
\resizebox {10cm}{!}{ \includegraphics{glitch_mosaic.eps}}


7.2.8.2 Glitch detection and correction in SWS01

Because in SWS01s the grating is moving during a detector reset interval special care has to be taken reducing the data in order not to confuse narrow lines with glitches. In order to correctly distinguish between glitches and unresolved lines, a source signal is estimated and subtracted from the ramp derivative before deglitching is carried out. This source signal estimate is derived using the knowledge that the source signal should be constant while the grating scanner is not moving (as each detector only sees light in one wavelength). After this step the glitch detection is the same as in the other bands.

In SWS01s glitches are not only detected but also corrected with the stepsize found at the detected glitch position. An example of this glitch recognition technique is shown in Figure 7.4.

For more information on how glitches are recognised in SWS01 data see Lahuis, Kester & Shipman 2001, [24].

Figure 7.4: Glitch detection in low resolution data with examples of a reset scanning an unresolved line (left) and a reset with a true glitch (right). From top to bottom we see:
- the input ramp.
- the uncorrected ramp derivative in black and the corrected ramp derivative in green.
- glitch detection in the pipeline using the corrected ramp derivative. In red the glitch detection limit and the samples identified as glitched are plotted.
- slope fits. In green the pipeline slope fit and the corrected ramp samples.
In the first example the pipeline deglitching using the corrected derivative correctly detects no glitches and fits a single slope. In the second example the glitch is detected and the derived slope is calculated from the corrected ramp.
\resizebox {7cm}{!}{\includegraphics{freddy_glitch1.eps}} \resizebox {7cm}{!}{\includegraphics{freddy_glitch2.eps}}

7.2.9 Extraction of the photo-currents and their uncertainties

The net effect of this processing step is to convert the data from a set of 24 samples per second per detector to 1 point per reset interval per detector. Because of two schemes if glitch detection, also slope fitting come in two variants, one for SWS01 and one for the general case.

7.2.9.1 Slope fitting in general

The current through the detector charges a capacitor. The voltage over the capacitor is measured non-destructively during a reset interval. When there is no glitch within the reset interval we can write the voltage, $ V_k$, as a linear function of the current, $ S$:

$\displaystyle V_k = S \cdot t_k + O$ (7.4)

where $ O$ is the offset caused by the charge at the zero-point of each ramp and the time $ t_k$ is counted in seconds since the start of the ramp. $ V_k$ and $ O$ are in bits and $ S$ is in bits/s.

When one or more glitches are present during the interval we add a term for each glitch. For a glitch at sample $ j$ we have:

$\displaystyle V_k = S \cdot t_k + O + p_j \cdot H(t_k - t_j) + \cdots$ (7.5)

where $ H(t)$ is the unit step function which step from 0 to 1 at t=0. $ p_j$ represents the glitch height at sample $ j$.

This model is fitted to the data $ D_k$ by minimising $ \chi^2$.

$\displaystyle \chi^2 = \sum_k ( V_k - D_k )^2$ (7.6)

where the sum is over all valid points within the reset interval, i.e. excluding 6 points after the reset pulse (the reset cutout length) and saturated points, if any. From this fit we find values for the parameters in Equation 7.5, the slope $ S$, the offset $ O$ and the glitch heights $ p_j$. Of these only the slope is kept. It represents the current through the detector.

The error in the signal is obtained using the standard method to calculate the standard deviation in a slope:

$\displaystyle \sigma_{S} = \sqrt{\frac{\chi ^{2}}{(N-K)}}\sqrt{C_{ss}}$ (7.7)

where $ N$ is the number of valid samples in the reset interval, $ K$ is the number of parameters to be extracted ($ K$ = 2 + number of glitches) and $ C_{ss}$ is the element corresponding to the slope, in the covariance matrix spanned by the model in Equation 7.5.

The first factor in Equation 7.7 is the standard deviation of a linear model fit, and the second factor is the amount that has to be assigned to the slope part of the model.

The standard deviation is directly estimated from the fit residuals. Obviously, the accuracy depends not only on the intensity of the source, but also on how well the ramps have been previously linearized and therefore on the measurement error of the RC time constants, the pulse-shape correction and at high gain also digitization plays its role.

For band 3 detectors the slopes are multiplied by $ -$1, as their bias is reversed.

7.2.9.2 Slope fitting for SWS01

Slope fitting in SWS01 is only different in the sense that all detected glitches are corrected in the glitch detection part. So there are no glitches to be fitted, no $ H$-functions. As the glitch heights are estimated from the difference of two consecutive points only, the fits and thus the slopes derived from them, are slightly less efficient in the case where no flux change occurs during the reset interval. It is a trade-off against grossly misfitting the slopes and glitches in the presence of changing flux. In Kester 1999, [19] a more precise method for doing the same is presented. It takes much more time, however.


7.2.10 Conversion to voltages

The electronics amplify the detector voltages with a preselected gain factor before the Analogue to Digital (AD) conversion. Each detector can be operated at three different gains of the amplifier chain. To convert the slopes to currents we have to undo the AD conversion and take into account the gains.

The current in $ \mu $V/s is related to the output of the AD converter, in bits/s, by

$\displaystyle S(\mu$V/s$\displaystyle ) = G_{AD} \cdot G_{pre} \cdot S($bits/s$\displaystyle )$ (7.8)

where $ G_{AD}$ is the AD-conversion factor (= 20 10$ ^6$ / 4095) and $ G_{pre}$ is the pre-amplifier gain which can be one of 225, 900 or 3600 for the grating and a value twice this high for the FP. They are kept in the calibration file Cal-G 5. The resulting signals are stored in the flux tag of the SPD (SWSPFLUX).

The same conversion is applied to the standard deviations of Equation 7.7. These are stored in the stdev tag of the SPD (SWSPSTDV). These standard deviations are passed unchanged to the AAR (SWAASTDV), see Section 7.4.

The fourth plot in Figure 7.2 shows 40 seconds of detector 16 after this processing step. The data is now in $ \mu $V/s against time. All that remains for Derive-SPD is to assign wavelengths to the individual data points.

7.2.11 Assigning wavelengths

All reset intervals of all detectors are assigned a wavelength within the limitations described below, even if the band was not requested or contains otherwise useless data. The wavelength is stored in the wave tag of the SPD (SWSPWAVE) or AAR (SWAAWAVE).


7.2.11.1 Grating wavelengths

Within an ERD we have measurements, 24 times per second, of the grating position both for the SW and the LW sections of the grating. The grating position is measured in LVDT, see Section 3.2.1 and Chapter 4.

The grating positions are averaged over the valid samples within each reset interval and for each detector separately as the valid samples may differ from detector to detector. This results for SWS01s in potentially different values for the grating position for each detector. In all other AOTs the scanner does not move during a reset interval.

Subsequently the averaged grating positions are converted into grating angles via the grating polynome. This is a 5th order polynome of which the (time dependent) parameters are stored in the calibration file Cal-G 16ET7.1. The grating angle, $ \theta$, is used to obtain the wavelength, $ \lambda$, via

$\displaystyle \lambda = \frac{D}{N} ( \sin(\theta + \beta) + \sin (\theta + \delta) )$ (7.9)

$ D$ is the grating constant. It is taken from calibration file Cal-G 16C. The angle $ \beta$ runs from the optical axis to the entrance slit. It is stored in the calibration file Cal-G 16A. The angle $ \delta$ runs from the optical axis to the detector pixel. It is stored in the calibration file Cal-G 16B.The spectral order $ N$ is determined by calculating all the orders that give wavelengths consistent with the transmission limits of the aperture filters in combination with the detection limits of the blocks (Table 2.1).

A valid wavelength can be assigned only when one order fits the criterion. The order found is incorporated in the flag, which accompanies every flux in the SPD (or AAR), see Table A.7. When no order fits the criterion, a wavelength of 0 is assigned and no order flag bit is set. When multiple orders are found, the wavelength is given its first order value while the order flag bits are all set, resulting in a value of 7.

The transmission limits are stored in the calibration file Cal-G 24. The detection limits of the detector blocks are stored in the calibration file Cal-G 23.

The SPD has two more tags related to the (grating) wavelengths: grating position (SWSPGPOS) and grating angle (SWSPGANG). The position and angle of the first sample of each reset interval is stored in there. As explained above, for SWS01s these tags do not convert directly into the wavelengths quoted for each detector because of the movement of the scanner.


7.2.11.2 Fabry-Pérot wavelengths

When determining the wavelength transmitted by the FPs, the grating wavelength must first be calculated as the LW grating section serves as order filter for the scanning FPs. Once the grating wavelength has been determined, calibration file Cal-G 12 is used to transform the nominal FP-position (an engineering parameter of no interest to the observer) into a FP-gap. This mechanical gap width is then corrected to get the optical gap width (using calibration file Cal-G 18). The wavelength which passes through the FP is the one of which a whole multiple fits exactly in two times the optical gap width.

The FP-position is stored in the SPD as SWSPFPOS and the FP-gap in SWSPFGAP. There is one more item in the SPD which is related to the FP, the FP-current (SWSPFCUR). It is computationally not related to the other FP items. Actually it is not used at all.

7.2.12 Writing the data

The data produced by the processing chain above is written out to the SPD and glitch list SWGH files. Each AOT results in one SPD file. For a definition of the SPD file see Section A.3.1. Note that the SPD file, like an ERD file, contains science data interleaved with dark current measurements, photometric checks etc.

An SPD record is produced for every reset interval. Complexities arise when the reset intervals for the SW and LW sections of the instrument are different, due to the instrumental setup required to obtain the required S/N ratios in the different halves of the instrument. In this case there will be one SPD record (containing data for all detectors) for each of the shorter reset intervals, with the SPD for the longer reset interval having its flag word set to `No data' for the records when it did not contain a reset (see Table A.7) and contains `0' in the flux -- this sometimes confuses plotting programmes. As an example, if the SW grating had a reset interval of 1 second and the LW one of 2 seconds, an SPD record would be produced every second. Every SPD record for the SW detectors (1-24) would contain valid data, while only every other record for the LW detectors would contain valid data, the others being marked as `No data'.

Figure 7.5 shows the final output of the Derive-SPD stage as detector output against wavelength.

Figure 7.5: A portion of the SPD data against wavelength for all detectors in band 3A. The detectors are shifted vertically with respect to each other for clarity. In this SWS02 scan not all detectors cover the same wavelength range or even the targeted line. In green the up scans are displayed and in red the down scans. The continuum is not at a constant level due to fringes - see Section 9.7.
\rotatebox {90}{\resizebox{!}{12cm}{\includegraphics{olp_spd_l.ps}}}

next up previous contents index
Next: 7.3 From SPD to Up: 7. Standard Product Generation Previous: 7.1 Introduction
ISO Handbook Volume V (SWS), Version 2.0.1, SAI/2000-008/Dc