Section 1. Introduction

1.1 Purpose and Scope

This interface control document (ICD) specifies the structure and contents of the reference data described by the Calibration Data Base System (CDBS) and used by the OSS and PODPS Unified System (OPUS) and the Space Telescope Science Data Analysis System (STSDAS) to calibrate science data. (OSS is the Observation Support System, and PODPS is the Post Observation Data Processing System.) More formally, this ICD defines the relationship between two requirements documents: SOGS Design Manual, Requirements Section, DRD-SOGS-SE- 06-1, and STScI Calibration Data Base Software Requirements, SO-11.). The reference data described in CDBS are generated from calibrations of the telescope’s science instruments. Their characteristics are stored in an on-line calibration data base for use in both OPUS and STSDAS, while the reference data files are stored in various on-line caches, and in the HST archive holdings of the Mikulski Archive for Space Telescopes (MAST).

1.2 System Description

1.2.1 OPUS

OPUS is responsible for generating science data products for the Hubble Space Telescope (HST). STSDAS software is run within OPUS to help accomplish this task. After astronomical observations are made with HST, OPUS receives HST science telemetry files (PODs) from the Goddard Space Flight Center’s PACOR facility, partitions the PODs into instrument-specific exposure data sets, determines the appropriate reference data for each exposure via queries of the on-line calibration data base, and invokes the appropriate STSDAS calibration software to generate calibrated science data products. These products, along with the initial science telemetry files, are permanently saved in the HST archive, and an archive catalog describing the characteristics of these products is made available to archive researchers to assist in identifying HST data for download to accomplish scientific investigations.

This initial processing and archiving of HST data is later supplemented by two other processing modes that allow improvements to the science data products: on-the-fly reprocessing (OTFR) and reprocessing with static archiving. OTFR is implemented for HST science instruments that are still taking active observations, and whose calibration software and reference data have not been finalized. When an HST archive user makes a request for science data products for an OTFR-supported instrument, the science data products are regenerated “on-the-fly” during the archive retrieval, using the initial HST science telemetry files, the latest OPUS and STSDAS software, and the best-possible reference data for the exposures as described in CDBS. These updated science data products are distributed directly to the archive user, giving them the benefits of all updates made to the processing software and reference data since the data was originally archived. Once an HST science instrument is no longer taking active observations (due to decommissioning or a hardware failure) and final calibration software and reference data are produced, a final reprocessing run is made for all of the archived data for that instrument. The science data products generated during this final reprocessing run are saved in the HST archive and from then on, are distributed directly to archive users when requested. OTFR is disabled for these “legacy” instruments, since no further improvement can be obtained in the science data products through OTFR if no further updates to processing software or reference data are being made.

1.2.2 STSDAS

STSDAS is a suite of applications software, utility packages, and support software used to calibrate and analyze data from HST. STSDAS is layered on the Image Reduction and Analysis Facility (IRAF) software from the National Optical Astronomy Observatories (NOAO). The system runs on any computing platform for which an IRAF port is available, including most Unix systems. STSDAS can also be used with PyRAF.

The STSDAS calibration software is used without change in the OPUS calibration pipeline for all HST data. Thus, HST observers can recalibrate their data, examining intermediate calibration steps as necessary, or modifying the reference data, using exactly the same code used in the standard calibration procedures.

1.2.3 CDBS Tools

CDBS uses as set of tasks or scripts for preparing reference files for submission to the Calibration Data Base and viewing the contents of this data base. On-site users can view the data base through the URL http://arch.stsci.edu/schema_explorer/

The CDBS package provides a set of tasks to prepare reference files for installation in the Calibration Data Base System (CDBS). These task can be used via command line commands run from the science1.stsci.edu and science3.stsci.edu servers in the Linux environment. Users need to be logged into these machines with the correct set up for these to work.

There are several steps in submitting a reference file for installation in CDBS. First, the header keywords and column names (if the reference file is a table) are checked for correctness by running certify. Next, the load file is produced for the reference file by running mkload. Mkload leaves some of the fields in the load file blank. These need to be filled in with a text editor. If the original reference file contains a wild card (usually -1 or ANY) in some of the observation mode fields, these will also be present in the load file. In order to expand the values to the full range of observation mode values, expload is run. After completing changes to the load file, it too must be certified by running certify. A new name for the reference file and load file is then generated by running uniqname. If one or more pysynphot graph table is modified, mkcomptab is run to generate a new component lookup table for installation.

In order to set up your account in these machines to run certify and fitsverify add the following to your .setenv file:

setenv CDBS_ROOT /grp/hst/cdbs/tools/
setenv CDBS_DATA ${CDBS_ROOT}/data/
setenv PATH ${PATH}
setenv PATH ${CDBS_ROOT}/bin/linux:/grp/software/Linux/RH4.5/i386_64/GCC4.2.4/bin
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH
${CDBS_ROOT}/lib/linux:/grp/software/Linux/RH4.5/i386_64/GCC4.2.4/lib/gcc/x86_64-redhat-linux/4.2.4.

For this to take effect you need to logout and then login again.

1.3 OPUS/CDBS Interface

The interface between the OPUS and CDBS software comprises shared data and protocols that enable data exchange.

The CDBS provides OPUS with data in the form of both reference files and reference relations (i.e., tables) used to reduce science data from the Wide Field/Planetary Camera (WF/PC), Wide Field/Planetary Camera 2 (WFPC 2), Faint Object Camera (FOC), Faint Object Spectrograph (FOS), High Resolution Spectrograph (HRS), High Speed Photometer (HSP), Near Infrared Camera (NICMOS), Space Telescope Imaging Spectrometer (STIS), Advanced Camera for Surveys (ACS), Cosmic Origins Spectrograph (COS), and Wide Field Camera 3 (WFC3). In turn, OPUS provides CDBS with science data that are used to update these reference files and relations. A reference file is either a GEIS format image (also known as STSDAS image format) or a FITS-format image. All GEIS images consist of two files, an ASCII header and a binary data file. A FITS image is a single file consisting of an ASCII header combined with binary image data. A reference table is either an STSDAS format table or a FITS table. Tables consist of a single file that, unlike an image, may contain data of several types.

CDBS does not retain the reference files or tables themselves, only their names and applicable instrument configurations. The file names corresponding to an instrument configuration or those related to a particular observation may be obtained through Starview web (http://starview.stsci.edu/web/#). The actual files may then be requested from the Data Archiving and Distribution System (DADS) or in many cases found in on-line archives on /grp/hst/cdbs/ in subdirectories with names like xref, xtab, yref and ytab. The first letter is the distinguishing instrument letter, and the rest of the name indicates reference files or tables.

As observational data pass through the RSDP pipeline, the names of all reference files and tables used in the calibration process are written to the header files. All such tables and files are archived by CDBS to DADS and supplied by CDBS to OPUS for use in calibrating the pipeline observations.

The CDBS tool**uniqname** is used to name all of the calibration tables and reference files. Table 1-1 describes the criteria used by this task to assign a file name. Each root name will have nine characters, numbered left to right; these characters can be interpreted by replacing them with the values shown in the table.

Table 1-1. Criteria for Assigning a Uniqname

Position Parameter Characters Used Comments
1 Year 1-9, A-Z 1981 = 1
2 Month January = 1
3 Day 1-9, A-V First day of the month = 1
4-5 Hour 00-23
6-7 Minute 00-59
8 Seconds/2 0-9, A-T
9 Instrument ID F, M-O, V-Z See Table 1-2

Table 1-2 shows the instrument IDs for the last letter of the unique name.

Table 1-2. Instrument Codes

Letter Instrument
F FGS
I WFC3
J ACS
L COS
N NICMOS
O STIS
V HSP
U WFPC 2
W WF/PC
X FOC
Y FOS
Z HRS
M Multiple instruments

1.3.1 Calibration Tables

Calibration tables are instrument specific. Without a full complement of calibration tables for a given instrument, the pipeline calibration will seldom work. These calibration tables are binary files produced, manipulated, and examined by various STSDAS utilities.

The type of calibration table can be determined from the table name extension, which for legacy instruments follows certain conventions. Table 1-3 describes the conventions for these reference tables.

Table 1-3. Conventions for Legacy Instruments Reference Table Names

Position Characters Used Comments
1 C, T C = catalog, T = synthetic photometry
2 F, I-J, L-O, V-Z Instrument ID (see Table 1-2)
3 0-9, A-Z [1]
[1]Tables are generally numbered sequentially for each instrument. Certain tables have a specific character for the third position in the extension carried over from an “old” relation; e.g., CMG is the extension for the HRS and FOS Paired Pulse Tables.

Table 1-4 gives a comprehensive list of reference tables. The root name of these reference tables is determined by the naming algorithm in the STSDAS task uniqname. For STIS and NICMOS both images and tables are in FITS format and have a file name extension of the form XXX.FITS, where XXX is indicative of the calibration function of the file.

Table 1-4. List of Reference Tables

Instrument
Extension | | Comments
FOC
CXU | | Flat fields
FOS
CY0
CY1
CY2
CY3
CY4
CY5
CY6
CY7
CY8
Aperture parameters
Aperture position parameters
Emission lines
Filter widths
Wollaston/wavelength parameters
Sky shift parameters
Wavelength coefficients
GIMP-correction coefficients
Predicted background count rates
HRS
CZ1
CZ2
CZ3
CZ4
CZ5
CZ6
CZ7
CZ8
CZ9
CZA
CZB
Line mapping parameters
Sample mapping parameters
Detector parameters
Wavelength ranges
Spectral order constants
Dispersion constants
Thermal constants
Incidence angle constants
Echelle interpolation constants
Echelle non-interpolation constants
Scattered light subtraction scale factors
HSP
CV0
CV1
CV2
CV4
CV5
CV6
CV7
CV8
CV9
Aperture areas
High voltage factor
Gain factor CV3 Pre-amp
Efficiency
Dark count
Time bias
CVC offset
Dead time
Dark aperture
WF/PC
CW0
Photometry group parameters
NICMOS
PHT
BKG
IDC
ZPR
NIC
Photometric calibration table
Predicted background table
Geometric Distortion Coefficients Table
Nonlincor Zeropoint Scaling Table
Nonlinearity Power Law Tables
STIS
CCD
A2D
CRR
BPX
LIN
PHT
APT
APD
LMP
IDC
SDC
IAC
DSP
1DT
1DX
MOC
CDS
ECH
EXS
HAL
SRW
TDC
TEL
CCD parameters table
Analog-to-digital correction table
Cosmic ray rejection parameters table
Bad pixel table
MAMA linearity table
Photometric conversion table
Aperture throughput table
Aperture description table
Template cal lamp spectra table
Image distortion correction table
2-D spectrum distortion correction table
Incidence angle correction table
Dispersion coefficients table
1-D spectrum trace table
1-D extraction parameter table
MAMA offset correction table
Cross-Disperser Scattering table
Echelle Scattering Table
Echelle Cross-Dispersion Scattering Table
Detector Halo Table
Scattering Reference Wavelengths Table
Dark Correction Table
Telescope Point Spread Function Table
WFPC 2  
(No reference tables)
ACS
A2D
BPX
CCD
OSC
CRR
LIN
PHT
IDC
SYN
TMC
TMG
Analog-to-Digital Table
Bad Pixel Table
CCD Characteristics Table
Overscan Region Table
Cosmic Ray Rejection Parameter Table
MAMA Linearity Table
Photometry and Throughput Table
Image Distortion Coefficients Table
Synphot Table
COMPTAB
GRAPHTAB
FOS, HRS
CMG
Paired pulse
WFC3
A2D
BPX
CCD
OSC
CRR
IDC
Analog-to-Digital Table
Bad Pixel Table
CCD Characteristics Table
Overscan Region Table
Cosmic-Ray Rejection Parameter Table
Image Distortion Coefficients Table
Multiple
TMG
TMC
TAB
Component list lookup [2]
Component graph [2]
Component throughput as function of wavelength [2]
[2](1, 2, 3) These tables are currently used by the FOC, HSP, and WFPC 2.

1.3.2 Throughput Tables

Throughput tables are a special type of calibration table that uses different naming conventions. These tables are used to compute the sensitivity of an instrument in a particular mode. They are binary files that are produced, manipulated, and examined by various STSDAS utility programs.

The throughput table has the following conventions for the names:

  1. The extension is always _syn.fits
  2. The root name always has an underscore and three numbers at the right side representing the version number of the table.
  3. The first part of the name (up to the “-” character preceding the version number) is the name of the component corresponding to the throughput table.

This name should be used in cataloging the throughput tables.

1.3.3 Reference Files

Reference files are observation specific. Generally, different observations will require different sets of reference files for their calibrations. Reference files are composed of a header file (in FITS format) and a binary data file. The extensions to reference files are determined according to the conventions described in Table 1-5.

Table 1-5. Reference File Extensions

Position Characters Used Comments
1 R, B R = reference file, B = associated data quality file
2 1-9, A-Z Reference files are numbered sequentially for each instrument
3 H, D H = header file, D = data file

The root name of the reference files is determined by the naming algorithm used by the uniqname task in STSDAS. Table 1-6 shows a complete list of these reference files by instrument.

Table 1-6. Reference File Names

Instrument Extension Type Comments
FOC
R0H, R0D
R1H, R1D
R2H, R2D
R3H, R3D
R5H, R5D
R7H, R7D
BAC
ITF
UNI
SDE
GEO
BLM
Background
Intensity transfer function
Uniform detector efficiency
Spectrographic detector efficiency
Geometric distortion
Data quality initialization
FOS
R0H, R0D
R1H, R1D
R2H, R2D
R3H, R3D
R4H, R4D
R5H, R5D
BAC
FLT
IVS
RET
DDT
QIN
Background
Flat field
Inverse sensitivity
Retardation
Disabled diode
Data quality initialization
HRS
R0H, R0D
R1H, R1D
R2H, R2D
R3H, R3D
R4H, R4D
R5H, R5D
DIO
PHC
VIG
ABS
NET
QIN
Diode response
Photocathode response
Vignetting
Absolute sensitivity
Wavelength net
Data quality initialization
HSP    
(No reference files)
NICMOS  
MSK
NOI
LIN
DRK
TDD
FLT
TDF
ILM
PMD
PMK
SDK
Static bad pixel mask
Noise file
Linearity correction file
Dark current file
Temperature Dependent Darks
Flat field file
Temperature Dependent Flat Fields
Background illumination pattern file
Persistence Model Files
Persistence Mask Files
Special Darks for SAACLEAN
STIS  
BIA
DRK
SSC
PFL
DFL
LFL
SSD
Bias image file
Dark image file
Shutter shading correction
Pixel-to-pixel flat
Delta flat
Low-order flat
Small scale distortion correction
ACS  
BIA
DRK
PFL
LFL
SHD
FLS
DXY
Bias Reference Image
Dark Image File
Pixel-to-pixel Flat
Large Scale Flat
Shutter-Shading Correction Image
Post Flash Image
Delta Distortion Image
WFC3  
BIA
DRK
PFL
DFL
LFL
SHD
FLS
LIN
Bias Image File
Dark Image File
Pixel-to-Pixel Image File
Delta Flat Image File
Low-Order Flat Image File
Shutter-Shading Image File
Post-Flash Image
Linearity Correction File
COS  
TBS
 
WF/PC
R0H, R0D
R1H, R1D
R2H, R2D
R3H, R3D
R4H, R4D
R5H, R5D
R6H, R6D
MSK
A2D
BAS
PRF
SPG
DRK
FLT
Static mask
A-to-D lookup tables
Bias
Preflash
Superpurge
Dark
Flat field
WF/PC
B2H, B2D
B3H, B3D
B4H, B4D
B5H, B5D
B6H, B6D
BAS
PRF
SPG
DRK
FLT
Bias data quality
Preflash data quality
Superpurge data quality
Dark data quality
Flat field data quality
WFPC 2
R0H, R0D
R1H, R1D
R2H, R2D
R3H, R3D
R4H, R4D
R5H, R5D
MSK
A2D
BAS
DRK
FLT
SHD
Static mask
A-to-D lookup tables
Bias
Dark
Flat field
Shutter shading correction
WFPC 2
B2H, B2D
B3H, B3D
B4H, B4D
BAS
DRK
FLT
Bias data quality
Dark data quality
Flat field data quality

1.4 Communication Between OPUS and CDBS

The CDBS underwent a complete redesign in 1996, and the new system was brought on-line in early 1997. This coincided with the introduction of STIS and NICMOS.

The method of communication between CDBS and OPUS changed a great deal and incorporated more automatic methods and quality checks. CDBS releases to OPUS the reference files and calibration tables that have been delivered along with a catalog describing their use and an SQL command sequence that is used to install directly to the OPUS data base. The catalog and SQL commands also indicate which of the previously installed files have now been superseded and are to be removed from the OPUS file collection. In this way, OPUS only keeps files that will be needed for upcoming observations.

In parallel, CDBS delivers all files to DADS for permanent storage. In the event that a problem occurs in providing files to OPUS, they can normally be retrieved from this source.