Next: 7.3 Database Architecture Up: 7. The ISO Data Previous: 7.1 Introduction

7.2 IDA Contents

The IDA contains on magnetic disks all the levels of data products, from the reformatted telemetry up to the products fully processed by an automatic pipeline processing chain, as well as quick look browse products, together with ancillary information (engineering, uplink and downlink data). Essential parameters extracted from the data are part of the database allowing complex queries to be made. Also user provided reduced data, catalogues and atlases are archived (the so-called `Highly Processed Data Products' or HPDPs). They are well separated from the pipeline products and are queriable and retrievable in a user-friendly way.

The following paragraphs define in general terms what data and processing tools are archived. More details can be found in Appendix E.

7.2.1 Off-Line products

Every ISO observation was run through an automatic data-analysis pipeline called Off-Line Processing, or OLP, to produce three sets of data products. These correspond to increasing levels of data processing and were extracted from the OLP at successive stages of the pipeline process:

-: Edited Raw Data (ERD);
-: Science Processed Data (SPD);
-: Auto-Analysis Results (AAR);

The type and sophistication of analysis performed for each product set is instrument dependent. Detailed technical descriptions of the contents and formats of data product files and the various types of processing performed are given in the corresponding instrument chapters of the ISO Handbook.

In the ISO Data Archive, products are further classified according to the following scheme:

Raw Data Products

These are essentially unpacked telemetry in which no data reduction has taken place and thus form the starting point for data analysis performed by OLP or the user's own system. PHT ERD is an example.
Basic Science Data Products

These data have been processed further to an intermediate level (with the use of calibration files where necessary), often containing physical units. They are therefore ready for assembly into structures of astronomical significance such as images or spectra by OLP or independently by observers themselves before scientific conclusions can be drawn. For example, the final result of the LWS pipeline is the LSAN file that contains a tabulation of flux in physical units measured during the observation.
Fully Auto-processed Data Products

These data include a set of coherent, instrument-dependent measurements of images or spectra designed to get as close as possible by automatic means to what could be produced by an astronomer using an interactive analysis system. Where possible, products follow FITS standard conventions to ease further analysis by standard high-level tools such as IDL or fv. They allow the user to carry out initial survey programmes or to make an assessment of the relevance of a given observation to their scientific work, although the limitations must be emphasised of such an automatic procedure which takes no account of the individual circumstances of an observation and involves no scientific judgement. Significantly better results are likely to be achieved using an interactive system. Examples of AAR products are, for the CAM pipeline, individual FITS images in the CMAP file, that are combined where appropriate into mosaics in the CMOS file; a tabulation of sources detected in the images in the CPSL file; and individual source spectra as appropriate in the CSSP file.

7.2.1.1 Scientific validation and accuracy

In addition to the functional validation, the pipeline was also subjected to a `scientific validation' to establish confidence in the processing and calibration of the products. The goals of this scientific validation included, inter alia:

Confirming that the applied processing had the expected effect(s) on the data
Establishing that sufficient processing had been applied to the data
Demonstrating that the processing was reliable, stable and repeatable
Establishing the level of accuracy of the calibration
Documenting any known caveats or concerns with the processing and/or calibration
Demonstrating that, with the information provided in the relevant ISO Handbook sections, user can work from the supplied products to produce publishable results.

The scientific validation of data per AOT and product level is summarized in the specific sections in the ISO Handbook instrument volumes.

Of the 30,000 standard observations in the ISO Data Archive, 89% were performed with modes for which the pipeline was scientifically validated. The remaining 11% (all LWS03 observations and some PHT modes) may still be used for scientific analysis. However, it is strongly recommended to look at the relevant ISO Handbook chapters in order to understand the limitations of the pipeline processing for those modes and consult the ISO Helpdesk as necessary.

For most of the cases, it is possible to improve the data quality of ISO observations by manual data reduction with the different Interactive Analysis (IA) software packages (see Section 1.4.2). Products derived in this way, of higher quality than the pipeline products, can be ingested in the ISO Data Archive in the form of HPDPs (see Section 7.2.3).

7.2.1.2 Quality control

All standard observations in the Routine Phase were individually inspected and an assessment of the quality of the data products was made. Technical flags were assigned to the ISO data products to assess their suitability for delivery to the observers. A detailed description of the flags is given in Appendix E.

The Quality control process was split into three different levels:

Level 1: This was a first quality assessment performed by the Off-line Processing operators according to some simple criteria. A Real Time report provided by the Instrument controllers, an Instrument report automatically generated by the RTA/QLA system and an output log from the Pipeline processing determined that the observation: a) Failed completely and therefore was not delivered and flagged as failed; b) Passed the level 1 assessment being delivered to the proposer and flagged as `Good'. c) Was passed to the next quality control level in case of doubt.
Level 2: This was a second quality assessment performed by the Instrument Dedicated Teams to identify the cause of the level 1 check failure and to decide whether the data could be released. The observation was also flagged as `failed' or `Good'. In the case of a `failed' observation the QC level 3 could be invoked for rescheduling.
Level 3: The third quality assessment was performed by the Science Team at the SOC. According to the level 1 report and the analysis by the Instrument Dedicated Teams, the Science Team decided whether a failed observation had to re-enter the Mission Planning system for rescheduling.

In addition, an extra level (QC level 4) was invoked when the quality assessment of the observation had to be performed by the proposer. In case of declared failure, the observation could be re-entered into the planning system for execution.

It should be noted that this quality information referred to whether the data were useful from the technical point of view; no attempt was made to judge whether the data served to a particular science goal. During the ISO Active Archive Phase (2002-2006), it is foreseen to enhance the quality information in the ISO Data Archive. ISO products Quality Control statistics are given in Figure 7.2

**Figure 7.2:** ISO pipeline products Quality Control statistics which shows the number of observations which passed through the Quality Control process (from Claes et al. 1998, [24]). An `OK' assessment means that the observation could be processed by the pipeline, partially (until Derive-ERD or Derive-SPD) or fully (until Auto-Analysis). `NL 4' refers to the number of QC L4 cases which were shown to be rescheduling cases. `No' or `N' cases points to pure rescheduling cases in QC L1 (total failure of an observation in QC L1).

7.2.2 Browse products after OLP

The ISO Data Archive also provides browse products or quick-look data associated to each observation. These products enable users to make informed decisions as to what observations they want to download for detailed astronomical analysis. These products did not undergo scientific validation nor quality control and should therefore not be used to derive scientific conclusions.

7.2.3 Highly Processed Data Products

The Highly Processed Data Products (HPDPs) include DATA (images, spectra etc.), which have been processed beyond the pipeline and/or using new, refined algorithms and therefore have been improved to any degree compared to the legacy pipeline products, as well as any resulting CATALOGUES and ATLASES. In July 2003, the ISO Data Archive has been enhanced with the functionality for continuous ingestion of new data, catalogues and atlases, further to screening by the ISO Data Centre. A list of currently available Highly Processed Data Products is generated on-the-fly from the IDA and can be accessed through the ISO Data Center web page at:

https://www.cosmos.esa.int/web/iso $\rightarrow$ List of Highly Processed Data Products

Data, catalogues and atlases can be easily queried from the IDA interface.

7.2.4 Other products

The ISO Data Archive also contains auxiliary data (e.g. spacecraft data, mission planning information and diagnostic data), ancillary data (such as space weather), calibration data and other observation specific information. These are described in more detail in Appendix E.

Next: 7.3 Database Architecture Up: 7. The ISO Data Previous: 7.1 Introduction

ISO Handbook Volume I (GEN), Version 2.0, SAI/2000-035/Dc