The final version of the automated Offline Processing Software and associated Calibration files (OLP 10.0 and CALG 7.0) is now available through the on-the-fly-reprocessing facility of the ISO Data Archive.
The improvements affect all four ISO instruments and the Pointing information, as summarised below and described in more depth in the ISO Handbook V1.2, planned for release mid-2001.
By the end of 2001, all ISO telemetry will have been reprocessed with this version of the pipeline software, and the ISO Data Archive will have been repopulated with the latest version of the products.
During ISO's Active Archive Phase, effort will be invested to augment the archive content for many observations by ingestion of products which become available through interactive or specialised processing of subsets of the ISO data. So the archive will continue to be improved in terms of both its user friendliness, and the immediacy with which products can be taken and used.
The following changes have been made in the most recent OLP update:
Joris Blommaert (IDC Vilspa)
Diego Cesarsky (CAMNDC at MPE Garching)
Stephan Ott (IDC Vilspa)
Andy Pollock (IDC Vilspa)
OLP 10.0 and CALG 7.0 introduce the following upgrades and new features for ISOPHOT data processing:
calibration or software upgrade | affects AOT
------------------------------------------|--------------------------------
1) two-threshold deglitching for ramp | all AOTs, except PHT32
processing | for staring mode only
|
2) glitch removal for PHT32 observations | PHT32
|
3) signal processing parameters in case | all AOTs, except PHT32
of non-stable signals | for staring mode only
|
4) warning of residual drifts in FCS | all AOTs, except PHT40
calibration |
|
5) further consolidation of | all AOTs, except PHT40
flux dependent signal linearization |
and FCS characterization |
|
6) point spread function correction | all AOTs, except PHT40
factors |
|
7) orbit dependent default responsivities | all AOTs, except PHT40
|
8) application of default responsivities | all AOTs, except PHT40
if FCS power is outside calibrated |
range |
|
9) application of average responsivities | PHT03, PHT22, in raster mode
in mapping mode | PHT32
|
10) refinement of signal correction for | PHT03, in chopped mode
chopped P2 photometry measurements |
|
11) extension of PHT-S dynamic spectral | PHT40, in staring mode
response calibration to longest |
exposure times |
|
12) relative intensity profiles for | PHT04, in staring mode
multi-aperture measurements |
|
13) statistical flat-field correction for | PHT22, in raster mode
PHT-C raster maps | PHT32
|
14) on-source coordinates for rectangular | PHT03, PHT22, PHT40,
chopped measurements | in chopped mode
|
15) world coordinates in maps | PHT03, PHT22, in raster mode
| PHT32
Following the detection of a glitch along the sequence of read-outs of a ramp
the threshold is reduced giving a more efficient glitch detection, in particularfor glitch tails. For ramps with at least 32 read-outs the threshold is reduced
from 4.0 sigmas to 1.0 sigma until the end of the ramp or when the difference
is below 1.0 sigma.
Due to the fast chopper sweeps applied in PHT32 measurements, usually each chopper plateau comprises only a small number of ramps and each ramp a small number of read-outs. Hence the OLP standard deglitching algorithm was not adequate and practically no deglitching was performed for PHT32 observations. Similar to the processing for single pointing chopped measurements, the difference signals between adjacent read-outs are used rather than the slopes of the complete ramps and the signal per chopper plateau is determined by using the robust bi-weight mean method.
In case of the Mann test finding a non-stabilized signal at the end of the measurement, the best parameters for deriving an average signal are: the last 7 data points or the last 8 seconds of data whichever is longer in time. So far only the last 7 data points were used, regardless whether they correspond to at least 8 seconds of data or not. Thus, it was possible, that an amount of data less than 8 seconds was averaged which is now corrected.
The usual exposure time of FCS measurements is 32s. This is sometimes too short to achieve a stable signal. Since the absolute signal of the FCS measurement determines the responsivity and hence the absolute flux calibration, any unstable FCS signal has direct impact on the absolute calibration accuracy. In case of unstable FCS signals, warning messages are written to the headers of the SPD and AAR products.
The linearization tables P%%SLINR were updated with regard to the following aspects:
Since linearization curves and FCS power files are linked together, both were updated consistently, i.e. also new CalG tables P%%ILLUM, P%%FCSPOW, P%%FLAT and PPFTOF were generated. Artificial features in multi-filter measurements flagged in the caveat "Filters with less reliable photometry" of the OLP 9.0 Scientific Validation Report are either removed or drastically reduced.
The point spread function correction factors of PHT-P apertures and PHT-C pixels were based on a two-mirror model with central obscuration of the primary. New psf factors (CalG files PPPSF and PCPSF) were computed taking a model for the secondary tripod into account. These updated psf factors were also used in the ISOPHOT flux calibration system (item 5).
The relative orbit dependence of the default responsivities was based on an OLP 7 processing of the FCS measurement data base. This processing did not include linearity correction, so that some scatter in the resulting responsivities was introduced by the lack of this correction impacting also on the relative orbit shape. For P3, C100 and C200, showing strong orbit dependence of the default responsivity, the FCS measurement data set was reprocessed using the correct OLP 10 calibration. Linear least median square fits were applied to the data and bootstrapping on 500 data sets per detector and time dependence was performed to derive the uncertainties. The time dependent CalG files PPRESP_01, PPRESP_02, PC1RESP_01, PC1RESP_02, PC2RESP_01, and PC2RESP_02 were revised.
In particular for P1, tests showed that more accurate photometry results were achieved when using the default responsivity rather than the extrapolation of the FCS power curves. As soon as the FCS heating power is outside the calibrated range for the respective filter, default responsivities are used in calibration. This holds for staring, absolute, raster and chopped photometry. Information messages are written to the SPD product header.
The linear interpolation between two bracketting FCS measurements in mapping modes was found to be inadequate. The method does not correct for non-linear drifts. In addition, to the user it is not transparent which responsivity value can be assigned to a given data point. The responsivity is now obtained from the average of the two FCS measurements, provided that both are of good quality. If both are outside the calibrated FCS heating power range, default responsivities are used (see item 8). If only one good FCS measurement is available, this one will be taken. This is in particular important for sparse map measurements where the FCS settings can be different.
P2 chopped photometry uses now orbit dependent default responsivities rather than responsivites derived from the chopped FCS measurements. A more systematic chopped signal correction could be established, showing that:
CalG file PP2CHOPSIG was revised. Due to the new correction scheme, the asymmetry correction has been removed. This is the case for P1 chopped observations, too, for which no signal correction at all is applied.
There are no PHT-S calibration star measurements longer than 1024 seconds, in fact most exposures of calibration stars are less equal 512 seconds. The dynamic spectral response calibration was restricted to the first 1024 seconds. Good quality measurements of 1024 seconds up to 4096 seconds were used for self-calibration in that respect, that the flux was predicted from the first 512 seconds of the measurement, which is based on calibration standards, and transferred to the remaining exposure time. The consistency of the method was checked for the standard star measurements of 1024 seconds. Consequently the conversion factors in the CalG file PSDYNAMIC were updated and the weights in PSDYNWT adapted.
In ESA SP-455 (ISO Beyond Point Sources), page 41, T.G. Mueller described a method how to make scientific use out of multi-aperture measurements (PHT04), provided the source is as compact as 2 arcmin. The background signal is derived from the annulus formed by the 120 and 180 arcsec apertures of the on-source measurement. Therefore, only measurement sequences with these two apertures and at least two smaller ones are processed in this way. The background for all apertures has to be scaled with regard to the 180 arcsec aperture applying non-unity correction factors to the geometrical aperture area ratios. These were determined from consistency checks of many measurements using this method. A signal normalized to the one in the 180 arcsec aperture is constructed which is the same as the flux ratio of both apertures. In comparison with the profiles of true point sources available in the data base, the normalized psf fraction (see item 6), and the profiles of flat sky sequences, this allows to determine the extension of a source or the presence of a companion. The normalized fluxes are written to the header of the AAR product. In addition source and background signals are written to the header of the SPD product. The aperture correction factors for the background scaling have been hardcoded, no CalG file was generated. The numbers will be given in the PHT Handbook.
If the extent of the PHT22 or PHT32 raster map exceeds 20 raster points or plateaux, a statistical flat-field is determined from the ratio of the 10%tile of the surface brightness of each pixel to the average 10%tile for all pixels. These correction factors are applied to the PGAI products and are written to the product header.
So far pointing information in the product headers is related to the telescope pointing. In the case of rectangular chopped measurements the telescope pointing is, however, in between the on-source and the off-source position and the target position is not given. This is now provided by the new pointing keywords CONSCRA and CONSCDEC which are derived from the corrected telescope pointing information CINSTRA, CINSTDEC and CINSTROL and the chopper amplitude. The new keywords for the target position are given for all observing modes, hence their values are identical with the telescope pointing for all modes except rectangular chopped mode.
In PGAI products the world coordinate set CD1_1 to CD3_3 has been added, however with the restriction that axis 3 remains an index rather than a wavelength due to the inhomegeneous spacing of the ISOPHOT filter central wavelengths. For compatibility the old keywords CDELTi and CROTAi (i=1,2,3) are still provided.
For an update of the scientific validation status of individual AOTs or AOT submodes as well as the corresponding caveats, we refer to "Report on the PHT Scientific Validation for OLP 10.0".
Ulrich Klaas (PHTNDC at Max-Planck-Institut fuer Astronomie)
Phil Richards (UK ISO Data Centre at Rutherford Appleton Laboratory)
René Laureijs (IDC Vilspa)
In OLP versions prior to OLP 10 the deglitching could in some cases affect unresolved spectral features in AOT SWS01 data. Single points could be affected, but also parts of or complete lines were occasionally removed. These problems were most severe in the fast (speed 1 and 2) AOT SWS01 observations and caused complications for their analysis, e.g. the use of fast AOT SWS01 data to derive reliable line-fluxes.
This is because in AOT SWS01's the grating scanner was moving within an integration reset to be able to measure a low resolution spectrum. An integration ramp would thus consist of slopes from different spectral regions. When one of these came from a strong unresolved emission line it could be interpreted as being a glitch and corrected for in the slope fitting. This would then result in the loss of true signal.
In OLP 10 a correction has been implemented to solve these problems. Before the glitch detection (which has not changed) a correction is applied to remove source signal from the derivative of the integration ramp which is used to identify glitches. For details and examples please read the contribution of F. Lahuis to the "ISO Calibration Legacy Conference".
In OLP 10 false glitch detections are now highly suppressed and real glitches are still caught. In the few cases where there are still glitches detected where there should not have been one, they have little effect on the derived signal. OLP 10 AOT SWS01 data can therefore be used without any restrictions. The same applies to data reduced in the coming release (V3.0) of the SWS OSIA system.
Bands 2 and 4 suffer from memory effects, in that they respond slowly to flux changes. The other bands do not suffer, due either to their different detector material or different operating method. The main problem with the memory effects concerns dark current subtraction - the dark currents would change throughout an observation and this was not reflected in what was subtracted. Therefore band 2 data could be quite poor.
For band 2 it has been possible to fit a Fouks-Schubert model to the dark currents, effectively predicting the change of the dark current during an observation. This much better dark current estimate is subtracted leading to significant improvements in the AAR data. Residual effects of transients, especially at large flux steps, can still be found in band 2.
For band 4 we do not have a working model to correct for the transient effects and still rely on the `old' dark current subtraction. The effects of transients on each particular data set must be gauged by the user from comparing the up-scan with the down scan. For continuum flux the down-scan is most likely the better one because the detectors have had some time to get accustomed to the flux level they are seeing. Which immediately points to the weakness of this advice: When the (continuum) flux is changing rapidly or when passing over a strong emission line or even a glitch, the down scan might not be better than the upscan.
Extensive trend studies using data covering the full lifetime of the ISO satellite have been performed. These studies showed that the photometric stability of the SWS detectors was extremely high, and therefore a study was made to see if there could be a fundamental simplification in the application of the SWS flux calibration.
The new implementation separates three of the four fundamental steps involved in the flux calibration: flat-fielding; photometric gain correction; and signal to flux conversion. The fourth step, correction for the relative spectral response function, was considered separate from the beginning of the ISO mission.
The three steps were previously contained in one calibration file, CAL-G 13, which made identification of potential trouble difficult to recognize. Furthermore, two separate aspects of the CAL-G 13 and application were working to correct for flatfielding. This meant that after application of the calibration, the certainty that the average of the 12 detectors within an AOT band provided a better calibration was lost.
Three new calibration files have been introduced into the pipeline to account for these three photometric calibration steps. Application of CAL-G 43 corrects for flat fielding differences between detectors (per AOT band), CAL-G 41 corrects for short time variability based on observations of the internal stimulator (per detector block), and CAL-G 42 contains the signal to flux conversion (per AOT band). The photometric accuracies are listed in the SWS Handbook.
The RSRF files have been generated using the same models as were used for the flux calibration. Bands 1 and 2 use stellar models provided by Leen Decin (KUL, Leuven) exclusively for RSRF and Signal to Flux calibration. This means the consistency of the corrections is considerably improved.
Band 3D suffers from light leakage around 27 microns from light from 13 microns. This effect is most noticeable in targets with flux falling towards longer wavelengths, but it also affected the RSRF files producing incorrect continuum shapes for all targets around 27 microns. The new RSRF files do not suffer from this, so targets which do not suffer from light leakage (i.e. those in which the flux increases towards longer wavelengths) will have a better continuum shape at 27 microns.
Kieron Leech (IDC Vilspa)
Ekkehard Wieprecht (SWS NDC at ISOSDC, MPE, Garching)
Russel Shipman, Do Kester, Fred Lahuis (SWS NDC at DIDAC, Groningen)
Bart Vandenbussche (SWS NDC at DIDAC, KUL, Leuven)
for the SWS consortium.
Two calibration files, LCGR (grating Relative Spectral Responsivity Function (RSRF)) and hence the corresponding LCIR (Illuminator reference file), both used in auto-analysis, have been updated. They are now derived using the LWS01 AOT observations of Uranus. This gives two advantages over the OLP8 version which was based on a single observation. The AOT observations were taken with an oversampling of 4, the same oversampling used by the majority of observers. As the RSRF is now based on observations oversampled with a rate of 4, the effect of transients should be more similar between the RSRF and an individual observation. The previous RSRF based on a single observation used a much slower scanning speed as it was done with an oversampling of 8. The second advantage is that by combining the observations more scans are used in the RSRF determination giving a slight improvement in the S/N, however as the statistical error on the previous version was already <1%, this improvement is probably not noticeable except at the edges of the bands. The change means that the RSRF is now referenced to a different detector responsivity to OLP, hence the LCIR file has been updated to calibrate for the new responsivity reference.
2.1 Improvement to illuminator processing
The illuminator flash data are used in Auto-Analysis (AAL) to produce a responsivity correction ratio by comparing with the standard flash data in the LCIR file. The previous method treated all of the points within the flash equally. However, since the flash is divided up into a sequence of 'operations', treating each operation separately and then combining the results using a weighted average should produce a more accurate result. Therefore a separate ratio is now produced for each illuminator operation for each detector. The separate ratios are then combined using a weighted average to produce a single ratio and uncertainty for each detector.
However the new method is applied only to observations performed later than revolution 442, when a major change in the onboard illuminator operations was implemented (the number of integrations performed for each illuminator were increased from 8 to 24). For observations performed prior to revolution 442 the removal of points affected by glitches sometimes left just three or four points for an individual illuminator, making it almost impossible to apply the OLP 10 weighted-average method. Therefore, even in OLP 10, observations prior to revolution 442 are still processed using the "OLP 8" illuminator processing method.
2.2 Improvement to dark current subtraction at low signal levels
For grating observations (LWS01 and LWS02) the dark current is measured at the beginning of each illuminator flash.
For grating observations at low signal levels the flux values produced by AAL for some detectors can sometimes be negative for most of a scan. A predominantly negative flux indicates that the dark current value subtracted was too high. This may be due to inaccurate dark current measurements caused by glitches in the data. In these cases it would be better to subtract a nominal dark current value rather than the measured value. Therefore in OLP10 for grating observations each scan of each detector is checked to see if the nominal dark current produces a better result than the measured dark current. If this is the case then the nominal dark current is used instead of the measured dark current.
2.3 Change to content of flux uncertainty field in LSAN file
The AAL flux uncertainty includes the uncertainty in the RSRF. Since this is observation independent it is considered better to take this out of AAL. The RSRF term has thus been removed from the calculation of the AAL flux uncertainty. The LWS handbook will include the RSRF uncertainties so that the user can note its contribution in the context of the overall uncertainty.
2.4 Adoption of the Corrected Coordinates
Use is now made of coordinates that have been corrected for all known pointing system effects, including calibration of the optical system, differential aberration and guide star proper motion.
Cécile Gry (IDC, Vilspa)
Tanya Lim, Gerard Hutchinson, Andrew Harwood (LWS NDC at RAL, Didcot)
Andy Pollock (IDC Vilspa)