PIA recognizes the following types of data: Buffers, External FITS Product files, External FITS PIA files, Internal PIA files, Calibration Files (CAL-G, CAL-U, and CAL-Q), PIA Calibration Tables and PIA Configuration files. These files, data structures and their different flavours, will be described below.
The buffers are organized per processing level, and contain the data on a measurement basis, which is the PIA processing unit (thus differing to the SPG). A PHT-measurement is defined as a continuous observation of one sub-instrument configuration per raster point. Since the PHT sub-instruments have different data contents (given mainly by the different number of pixels) and the measurement have different lengths (thus producing different lenghts of data arrays), the only way to store them in IDL structures is by using pointers to the data variables. Every time data is read in or produced by reducing data from a lower level, the corresponding measurements are piled up to the existing ones on the corresponding level.
This has the advantage that correlations, comparisons, operations between different data sets are easily accessible and several capabilities are given to make use of this. Furthermore, after a data set has been read by PIA there is no further disk i/o and consequently all processing is very fast. It has, however, the disadvantage that PIA is "memory hungry", which might cause problems on computers with a small amount of RAM. To avoid such problems, it is possible to delete the contents of the PIA buffers at any time in order to release the memory again, after a certain processing step has been finished (cf. "Delete Buffers"). All buffers are automatically deleted, when the IDL session is closed and the blocked memory is released. The buffers are not deleted, however, if only PIA is quit. The buffers can then be made available under the standard IDL environment for manual analysis, which can be very useful for experienced users. In addition, because the information is not lost while IDL remains active, it is possible to re-enter PIA and continue the work. Re-entering PIA after the buffers have been modified using the IDL command interface is only recommended to real PIA and IDL experts as the results can be unpredictable, if all side effects are not properly taken into account.
In order to analyse and display the contents of the various PIA buffers a dedicated tool was written, which is called context sensitive from within PIA. For details on how to use this tool please refer to "The Show Structures - Plot Curves Menu". The details of the data structure definitions used within PHT Interactive Analysis:
Data Products, i.e. the data products available in the ISO archive. These products are generated by the pipeline software and contain all information from the spacecraft and the instrument in its various stages of processing. For the user the most important filetypes are ERD, SPD and AAP, comprising the main instrument data files. All these stages of processing can be read by PIA for further processing and their generation by the pipeline s/w can be emulated using PIA. There are various auxiliary files containing e.g. the s/c pointing information AOCS data (IIPH & IRPH), data from the ISOPHOT housekeeping data channel (GEHK) or instrument status data (CSH). These files are automatically read by PIA, when the related ERD, SPD or AAP product is addressed. In this respectr the user does not have to bother with their filename conventions.The detailed description of all product files is given in the IDPD [2]. For a good understanding of the PHT Interactive Analysis software the user is highly recommended to read this document to get familiar with the main product types and their contents. For a list of the available products and filenames, please refer to "Appendix E: Naming Conventions".Calibration Files, there are four types of calibration files for PHT: FCS calibration files (CAL-A), Uplink Tables (CAL-U), RTA/QLA calibration files (CAL-Q) and General (pipeline) calibration files (CAL-G). CAL-A and CAL-G files are products and consequently follow the FITS standard. All calibration files can be generated or updated using PIA. The data analysis PIA is performing is based on the pipeline CAL-G files or on results obtained with PIA itself. The FCS calibration files (CAL-A) are generated by the pipeline software and can be used by PIA to perform responsivity calibrations. The Uplink and RTA/QLA files (CAL-U and CAL-Q) are not in FITS format, but in a dedicated format used by the related s/w systems. These files are not used by PIA for calibrations purposes, but are read for checking purposes and also derived by PIA for inclusion into the SOC software.
Main files of this type are the Signal Reduced Data (SRD) and the Signal per Chopper Plateau Data (SCP). Both data types represent intermediate steps of processing between ERD and SPD pipeline products. It was found important to be able to keep these intermediate results, because the reduction step from ERD to SRD is by far the most CPU intensive one and considerably reduces the amount of data to be handled. In addition, the SRD and SCP levels of data processing allow the user to make a better judgement about the quality of the data than ERD and SPD level, for example w.r.t. drift behaviour, influence of glitches on signals, etc.
PIA SPD products can also be stored into FITS files. Although they are on the same level as SPD pipeline products, there are fundamental differences between these products. PIA products are measurement-based, e.g. one measurement on one file, while the SPG PHT-SPD products are per observation and detector, generally containing several measurements. The reduction parameters and corrections used for the derivation of the Pipeline products have to be extracted from the documentation of the used version, while the derivation of PIA products are documented on the file header itself by keywords, then they can be freely chosen by the user at any time. For an explanation of all the possible keywords please refer to "Appendix B: Internal Buffers Structures Definitions"
Also of fundamental importance, but for different reasons, is the possibility of storing highly reduced data in FITS format, on the 'Astrophysical Applications' level. This allows exporting data from PIA into other s/w packages for doing special processing, not contained in PIA. Up to now, only the possibility of producing mapping FITS files is given, but the future upgrades concerning polarimetry and (spectro-)photometry will be accompanied by an upgrade on this sector.
DDmmTTTTTTss.LLL is the general representation, where "DD" stands for detector ('P1', 'P2', 'P3', 'C1', 'C2', 'SS' and 'SL') followed by "mm" as the position of the measurement in the observation sequence for one detector (counting from 00 upwards). "TTTTTT" is the Target Dedicated Name (TDT) composed of 3 first digits for the orbit number and 3 last digits for the position within the orbit. "ss" is the sequence number as specified by the observer by observation planning. LLL finally gives the reduction level ('SRD', 'SCP', 'SPD', 'AAP', 'MAP'). Thus, a filename 'P20103700122.SRD' would represent the second (!) measurement with P2 on the first target during revolution 37. The number 22 could indicate a position within the proposal and the processing level is SRD, signal per ramp. Please note that a measurement with PHT-S is always an SS/SL measurement and up to the SPD level the information from all SS and SL detector pixels is kept together. In this case there are only 'SL...' files on the SRD and SCP level. On the SPD level they are separated (to correspond fully to the SPG definitions), thus 'SS....SPD' and 'SL....SPD' files exist.
Nomenclature: for every External PIA Data file the corresponding internal file exists. The proposed nomenclature to the user (since he/she also has the freedom here to change the name) has got the same prefix (as described above for External PIA Files), but the suffix contains the letter "I" to indicate the internal format, e.g. *.ISCP would be an internal Signal per Chopper Plateau file.
For calibration purposes the PHT Interactive Analysis keeps the main results of several or all in-orbit calibration measurements in little "data bases" for every type of calibration observation. The files in which these data are stored are called "Calibration Tables". As these files are written and maintained only by PIA, they are kept in an internal format which is not compatible with other systems. As these database files are usually common to all users of a PIA installation on a specific site, strict configuration control procedures must be observed when updating these files via CFDPs.
| Date | Author | Description |
|---|---|---|
| 15/06/1996 | Carlos GABRIEL (ESA/VILSPA-SAI) | First Version |
| 20/07/1996 | Carlos GABRIEL | Update |
| 10/06/1997 | Carlos GABRIEL | Update (V6.3) |