next up previous contents index
Next: 7.3 Database Architecture Up: 7. The ISO Data Previous: 7.1 Introduction

Subsections



7.2 IDA Contents

The IDA contains on magnetic disks all the levels of data products, from the reformatted telemetry up to the products fully processed by an automatic pipeline processing chain, as well as quick look browse products, together with ancillary information (engineering, uplink and downlink data). Essential parameters extracted from the data are part of the database allowing complex queries to be made. Also user provided reduced data, catalogues and atlases are archived (the so-called `Highly Processed Data Products' or HPDPs). They are well separated from the pipeline products and are queriable and retrievable in a user-friendly way.

The following paragraphs define in general terms what data and processing tools are archived. More details can be found in Appendix E.


7.2.1 Off-Line products

Every ISO observation was run through an automatic data-analysis pipeline called Off-Line Processing, or OLP, to produce three sets of data products. These correspond to increasing levels of data processing and were extracted from the OLP at successive stages of the pipeline process:

-
Edited Raw Data (ERD);
-
Science Processed Data (SPD);
-
Auto-Analysis Results (AAR);

The type and sophistication of analysis performed for each product set is instrument dependent. Detailed technical descriptions of the contents and formats of data product files and the various types of processing performed are given in the corresponding instrument chapters of the ISO Handbook.

In the ISO Data Archive, products are further classified according to the following scheme:


7.2.1.1 Scientific validation and accuracy

In addition to the functional validation, the pipeline was also subjected to a `scientific validation' to establish confidence in the processing and calibration of the products. The goals of this scientific validation included, inter alia:

The scientific validation of data per AOT and product level is summarized in the specific sections in the ISO Handbook instrument volumes.

Of the 30,000 standard observations in the ISO Data Archive, 89% were performed with modes for which the pipeline was scientifically validated. The remaining 11% (all LWS03 observations and some PHT modes) may still be used for scientific analysis. However, it is strongly recommended to look at the relevant ISO Handbook chapters in order to understand the limitations of the pipeline processing for those modes and consult the ISO Helpdesk as necessary.

For most of the cases, it is possible to improve the data quality of ISO observations by manual data reduction with the different Interactive Analysis (IA) software packages (see Section 1.4.2). Products derived in this way, of higher quality than the pipeline products, can be ingested in the ISO Data Archive in the form of HPDPs (see Section 7.2.3).


7.2.1.2 Quality control

All standard observations in the Routine Phase were individually inspected and an assessment of the quality of the data products was made. Technical flags were assigned to the ISO data products to assess their suitability for delivery to the observers. A detailed description of the flags is given in Appendix E.

The Quality control process was split into three different levels:

In addition, an extra level (QC level 4) was invoked when the quality assessment of the observation had to be performed by the proposer. In case of declared failure, the observation could be re-entered into the planning system for execution.

It should be noted that this quality information referred to whether the data were useful from the technical point of view; no attempt was made to judge whether the data served to a particular science goal. During the ISO Active Archive Phase (2002-2006), it is foreseen to enhance the quality information in the ISO Data Archive. ISO products Quality Control statistics are given in Figure 7.2

Figure 7.2: ISO pipeline products Quality Control statistics which shows the number of observations which passed through the Quality Control process (from Claes et al. 1998, [24]). An `OK' assessment means that the observation could be processed by the pipeline, partially (until Derive-ERD or Derive-SPD) or fully (until Auto-Analysis). `NL 4' refers to the number of QC L4 cases which were shown to be rescheduling cases. `No' or `N' cases points to pure rescheduling cases in QC L1 (total failure of an observation in QC L1).
Quality Control Statistics for the ISO Pipeline produ
cts


7.2.2 Browse products after OLP

The ISO Data Archive also provides browse products or quick-look data associated to each observation. These products enable users to make informed decisions as to what observations they want to download for detailed astronomical analysis. These products did not undergo scientific validation nor quality control and should therefore not be used to derive scientific conclusions.


7.2.3 Highly Processed Data Products

The Highly Processed Data Products (HPDPs) include DATA (images, spectra etc.), which have been processed beyond the pipeline and/or using new, refined algorithms and therefore have been improved to any degree compared to the legacy pipeline products, as well as any resulting CATALOGUES and ATLASES. In July 2003, the ISO Data Archive has been enhanced with the functionality for continuous ingestion of new data, catalogues and atlases, further to screening by the ISO Data Centre. A list of currently available Highly Processed Data Products is generated on-the-fly from the IDA and can be accessed through the ISO Data Center web page at:

         https://www.cosmos.esa.int/web/iso $\rightarrow$ List of Highly Processed Data Products

Data, catalogues and atlases can be easily queried from the IDA interface.

7.2.4 Other products

The ISO Data Archive also contains auxiliary data (e.g. spacecraft data, mission planning information and diagnostic data), ancillary data (such as space weather), calibration data and other observation specific information. These are described in more detail in Appendix E.


next up previous contents index
Next: 7.3 Database Architecture Up: 7. The ISO Data Previous: 7.1 Introduction
ISO Handbook Volume I (GEN), Version 2.0, SAI/2000-035/Dc