Section 1. Introduction

1.1 Purpose and Scope

This HST Reference Files Information Document specifies the structure and contents of the reference data described by the Calibration Refernce Data System (CRDS) that is used during HST Science Data Processing (HSDP) to calibrate science data. More formally, this document defines the format and usage of all the reference data described in CRDS. This data are generated from calibrations of the telescope’s science instruments. Their characteristics are stored in an on-line calibration data base for use in both, the HSDP and stenv, while the reference data files are stored in various on-line caches, and in the HST archive holdings of the Mikulski Archive for Space Telescopes (MAST).

1.2 System Description

1.2.1 HSDP

HSDP is responsible for generating science data products for the Hubble Space Telescope (HST). stenv software is run within HSDP to help accomplish this task. After astronomical observations are made with HST, receives HST science telemetry files (PODs) from the Goddard Space Flight Center’s PACOR facility, HSDP partitions the PODs into instrument-specific exposure data sets, determines the appropriate reference data for each exposure via queries of the on-line calibration data base, and invokes the appropriate stenv calibration software to generate calibrated science data products. These products, along with the initial science telemetry files, are permanently saved in the HST archive, and an archive catalog describing the characteristics of these products is made available to archive researchers to assist in identifying HST data for download to accomplish scientific investigations.

This initial processing and archiving of HST data is later supplemented by a reprocessing mode that allows improvements to the science data products. Data reprocessing is implemented for HST science instruments that are still taking active observations, and whose calibration software and reference data have not been finalized. When a new calibration reference file for HST is delivered or a change is made to the calibration pipeline software, the science data products are regenerated using the initial HST science telemetry files, the latest HSDP and stenv software, and the best-possible reference data for the exposures as described in CRDS. These updated science data products are distributed directly to the archive user, giving them the benefits of all updates made to the processing software and reference data since the data was originally archived. Once an HST science instrument is no longer taking active observations (due to decommissioning or a hardware failure) and final calibration software and reference data are produced, a final reprocessing run is made for all of the archived data for that instrument. The science data products generated during this final reprocessing run are saved in the HST archive and from then on, are distributed directly to archive users when requested.

1.2.2 stenv

stenv is a suite of Python utility packages, and support software used to calibrate and analyze data from HST. stenv is a set of installable Conda environments maintained by the Space Telescope Science Institute (STScI) in Baltimore, Maryland. The system runs on MacOS and Linux and the documentation can be found on-line (https://stenv.readthedocs.io/en/latest/getting_started.html). The stenv calibration software is used without change in the HSDP calibration pipeline for all HST data. Thus, HST observers can recalibrate their data, examining intermediate calibration steps as necessary, or modifying the reference data, using exactly the same code used in the standard calibration procedures.

1.2.3 CRDS Tools

CRDS uses as set of tasks or scripts for preparing reference files for submission to the Calibration Reference Data System (CRDS) and viewing the contents of this data base. On-site users can view the data base through the URL https://hst-crds.stsci.edu/

The CRDS package provides a set of tasks to prepare reference files for installation in the CRDS. These task can be used via command line commands as indicated in the CRDS User Guide (https://hst-crds.stsci.edu/static/users_guide/index.html) or if authorized by the management team, via the special services for deliverers in the https://hst-crds.stsci.edu/ pages.

There are several steps in submitting a reference file for installation in CRDS. First, the header keywords and column names (if the reference file is a table) are checked for correctness by running certify. Using the CRDS Routine Submission Services option Extended Barch Submitt References deliver your data. This process requires to fill a form with needed metadata to pupulate the CRDS database and to keep track of the delivery of these files.

In order to set up your account in these machines to run certify and fitsverify add the following to your .setenv file:

setenv CRDS_ROOT /grp/hst/cdbs/tools/
setenv CRDS_DATA ${CRDS_ROOT}/data/
setenv PATH ${PATH}
setenv PATH ${CRDS_ROOT}/bin/linux:/grp/software/Linux/RH4.5/i386_64/GCC4.2.4/bin
setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}
setenv LD_LIBRARY_PATH
${CRDS_ROOT}/lib/linux:/grp/software/Linux/RH4.5/i386_64/GCC4.2.4/lib/gcc/x86_64-redhat-linux/4.2.4.

For this to take effect you need to logout and then login again.

1.3 HSDP/CRDS Interface

The interface between the HSDP and CRDS software comprises shared data and protocols that enable data exchange.

The CRDS provides HSDP with data in the form of both reference files and reference relations (i.e., tables) used to reduce science data from the Wide Field/Planetary Camera (WF/PC), Wide Field/Planetary Camera 2 (WFPC 2), Faint Object Camera (FOC), Faint Object Spectrograph (FOS), High Resolution Spectrograph (HRS), High Speed Photometer (HSP), Near Infrared Camera (NICMOS), Space Telescope Imaging Spectrometer (STIS), Advanced Camera for Surveys (ACS), Cosmic Origins Spectrograph (COS), and Wide Field Camera 3 (WFC3). In turn, HSDP provides CRDS with science data that are used to update these reference files and relations. A reference file is either a GEIS format image (also known as stenv image format) or a FITS-format image. All GEIS images consist of two files, an ASCII header and a binary data file. A FITS image is a single file consisting of an ASCII header combined with binary image data. A reference table is either an stenv format table or a FITS table. Tables consist of a single file that, unlike an image, may contain data of several types.

CRDS does not retain the reference files or tables themselves, only their names and applicable instrument configurations. The file names corresponding to an instrument configuration or those related to a particular observation may be obtained through Starview web (http://starview.stsci.edu/web/#). The actual files may then be requested from the Data Archiving and Distribution System (DADS) or in many cases found in on-line archives on /grp/hst/cdbs/ in subdirectories with names like xref, xtab, yref and ytab. The first letter is the distinguishing instrument letter, and the rest of the name indicates reference files or tables.

As observational data pass through the RSDP pipeline, the names of all reference files and tables used in the calibration process are written to the header files. All such tables and files are archived by CRDS to DADS and supplied by CRDS to HSDP for use in calibrating the pipeline observations.

The CRDS tool**uniqname** is used to name all of the calibration tables and reference files. Table 1-1 describes the criteria used by this task to assign a file name. Each root name will have nine characters, numbered left to right; these characters can be interpreted by replacing them with the values shown in the table.

Table 1-1. Criteria for Assigning a Uniqname

Position	Parameter	Characters Used	Comments
1	Year	1-9, A-Z	1981 = 1
2	Month	–	January = 1
3	Day	1-9, A-V	First day of the month = 1
4-5	Hour	00-23	–
6-7	Minute	00-59	–
8	Seconds/2	0-9, A-T	–
9	Instrument ID	F, M-O, V-Z	See Table 1-2

Table 1-2 shows the instrument IDs for the last letter of the unique name.

Table 1-2. Instrument Codes

Letter	Instrument
F	FGS
I	WFC3
J	ACS
L	COS
N	NICMOS
O	STIS
V	HSP
U	WFPC 2
W	WF/PC
X	FOC
Y	FOS
Z	HRS
M	Multiple instruments

1.3.1 Calibration Tables

Calibration tables are instrument specific. Without a full complement of calibration tables for a given instrument, the pipeline calibration will seldom work. These calibration tables are binary files produced, manipulated, and examined by various stenv utilities.

The type of calibration table can be determined from the table name extension, which for legacy instruments follows certain conventions. Table 1-3 describes the conventions for these reference tables.

Table 1-3. Conventions for Legacy Instruments Reference Table Names

Position	Characters Used	Comments
1	C, T	C = catalog, T = synthetic photometry
2	F, I-J, L-O, V-Z	Instrument ID (see Table 1-2)
3	0-9, A-Z	1

1

Tables are generally numbered sequentially for each instrument. Certain tables have a specific character for the third position in the extension carried over from an “old” relation; e.g., CMG is the extension for the HRS and FOS Paired Pulse Tables.

Table 1-4 gives a comprehensive list of reference tables. The root name of these reference tables is determined by the naming algorithm in the stenv task uniqname. For STIS and NICMOS both images and tables are in FITS format and have a file name extension of the form XXX.FITS, where XXX is indicative of the calibration function of the file.

Table 1-4. List of Reference Tables

Instrument	Extension | | Comments
FOC	CXU | | Flat fields
FOS	CY0 CY1 CY2 CY3 CY4 CY5 CY6 CY7 CY8		Aperture parameters Aperture position parameters Emission lines Filter widths Wollaston/wavelength parameters Sky shift parameters Wavelength coefficients GIMP-correction coefficients Predicted background count rates
HRS	CZ1 CZ2 CZ3 CZ4 CZ5 CZ6 CZ7 CZ8 CZ9 CZA CZB		Line mapping parameters Sample mapping parameters Detector parameters Wavelength ranges Spectral order constants Dispersion constants Thermal constants Incidence angle constants Echelle interpolation constants Echelle non-interpolation constants Scattered light subtraction scale factors
HSP	CV0 CV1 CV2 CV4 CV5 CV6 CV7 CV8 CV9		Aperture areas High voltage factor Gain factor CV3 Pre-amp Efficiency Dark count Time bias CVC offset Dead time Dark aperture
WF/PC	CW0		Photometry group parameters
NICMOS	PHT BKG IDC ZPR NIC		Photometric calibration table Predicted background table Geometric Distortion Coefficients Table Nonlincor Zeropoint Scaling Table Nonlinearity Power Law Tables
STIS	CCD A2D CRR BPX LIN PHT APT APD LMP IDC SDC IAC DSP 1DT 1DX MOC CDS ECH EXS HAL SRW TDC TEL		CCD parameters table Analog-to-digital correction table Cosmic ray rejection parameters table Bad pixel table MAMA linearity table Photometric conversion table Aperture throughput table Aperture description table Template cal lamp spectra table Image distortion correction table 2-D spectrum distortion correction table Incidence angle correction table Dispersion coefficients table 1-D spectrum trace table 1-D extraction parameter table MAMA offset correction table Cross-Disperser Scattering table Echelle Scattering Table Echelle Cross-Dispersion Scattering Table Detector Halo Table Scattering Reference Wavelengths Table Dark Correction Table Telescope Point Spread Function Table
WFPC 2			(No reference tables)
ACS	A2D BPX CCD OSC CRR LIN PHT IDC SYN TMC TMG		Analog-to-Digital Table Bad Pixel Table CCD Characteristics Table Overscan Region Table Cosmic Ray Rejection Parameter Table MAMA Linearity Table Photometry and Throughput Table Image Distortion Coefficients Table Synphot Table COMPTAB GRAPHTAB
FOS, HRS	CMG		Paired pulse
WFC3	A2D BPX CCD OSC CRR IDC		Analog-to-Digital Table Bad Pixel Table CCD Characteristics Table Overscan Region Table Cosmic-Ray Rejection Parameter Table Image Distortion Coefficients Table
Multiple	TMG TMC TAB		Component list lookup 2 Component graph 2 Component throughput as function of wavelength 2

2(1,2,3)

These tables are currently used by the FOC, HSP, and WFPC 2.

1.3.2 Throughput Tables

Throughput tables are a special type of calibration table that uses different naming conventions. These tables are used to compute the sensitivity of an instrument in a particular mode. They are binary files that are produced, manipulated, and examined by various stenv utility programs.

The throughput table has the following conventions for the names:

1. The extension is always _syn.fits

2. The root name always has an underscore and three numbers at the right side representing the version number of the table.

3. The first part of the name (up to the “-” character preceding the version number) is the name of the component corresponding to the throughput table.

This name should be used in cataloging the throughput tables.

1.3.3 Reference Files

Reference files are observation specific. Generally, different observations will require different sets of reference files for their calibrations. Reference files are composed of a header file (in FITS format) and a binary data file. The extensions to reference files are determined according to the conventions described in Table 1-5.

Table 1-5. Reference File Extensions

Position	Characters Used	Comments
1	R, B	R = reference file, B = associated data quality file
2	1-9, A-Z	Reference files are numbered sequentially for each instrument
3	H, D	H = header file, D = data file

The root name of the reference files is determined by the naming algorithm used by the uniqname task in stenv. Table 1-6 shows a complete list of these reference files by instrument.

Table 1-6. Reference File Names

Instrument	Extension	Type	Comments
FOC	R0H, R0D R1H, R1D R2H, R2D R3H, R3D R5H, R5D R7H, R7D	BAC ITF UNI SDE GEO BLM	Background Intensity transfer function Uniform detector efficiency Spectrographic detector efficiency Geometric distortion Data quality initialization
FOS	R0H, R0D R1H, R1D R2H, R2D R3H, R3D R4H, R4D R5H, R5D	BAC FLT IVS RET DDT QIN	Background Flat field Inverse sensitivity Retardation Disabled diode Data quality initialization
HRS	R0H, R0D R1H, R1D R2H, R2D R3H, R3D R4H, R4D R5H, R5D	DIO PHC VIG ABS NET QIN	Diode response Photocathode response Vignetting Absolute sensitivity Wavelength net Data quality initialization
HSP			(No reference files)
NICMOS		MSK NOI LIN DRK TDD FLT TDF ILM PMD PMK SDK	Static bad pixel mask Noise file Linearity correction file Dark current file Temperature Dependent Darks Flat field file Temperature Dependent Flat Fields Background illumination pattern file Persistence Model Files Persistence Mask Files Special Darks for SAACLEAN
STIS		BIA DRK SSC PFL DFL LFL SSD	Bias image file Dark image file Shutter shading correction Pixel-to-pixel flat Delta flat Low-order flat Small scale distortion correction
ACS		BIA DRK PFL LFL SHD FLS DXY	Bias Reference Image Dark Image File Pixel-to-pixel Flat Large Scale Flat Shutter-Shading Correction Image Post Flash Image Delta Distortion Image
WFC3		BIA DRK PFL DFL LFL SHD FLS LIN	Bias Image File Dark Image File Pixel-to-Pixel Image File Delta Flat Image File Low-Order Flat Image File Shutter-Shading Image File Post-Flash Image Linearity Correction File
COS		TBS
WF/PC	R0H, R0D R1H, R1D R2H, R2D R3H, R3D R4H, R4D R5H, R5D R6H, R6D	MSK A2D BAS PRF SPG DRK FLT	Static mask A-to-D lookup tables Bias Preflash Superpurge Dark Flat field
WF/PC	B2H, B2D B3H, B3D B4H, B4D B5H, B5D B6H, B6D	BAS PRF SPG DRK FLT	Bias data quality Preflash data quality Superpurge data quality Dark data quality Flat field data quality
WFPC 2	R0H, R0D R1H, R1D R2H, R2D R3H, R3D R4H, R4D R5H, R5D	MSK A2D BAS DRK FLT SHD	Static mask A-to-D lookup tables Bias Dark Flat field Shutter shading correction
WFPC 2	B2H, B2D B3H, B3D B4H, B4D	BAS DRK FLT	Bias data quality Dark data quality Flat field data quality

1.4 Communication Between HSDP and CRDS

The CRDS underwent a complete redesign in 1996, and the new system was brought on-line in early 1997. This coincided with the introduction of STIS and NICMOS.

The method of communication between CRDS and HSDP changed a great deal and incorporated more automatic methods and quality checks. CRDS releases to HSDP the reference files and calibration tables that have been delivered along with a catalog describing their use and an SQL command sequence that is used to install directly to the HSDP data base. The catalog and SQL commands also indicate which of the previously installed files have now been superseded and are to be removed from the HSDP file collection. In this way, HSDP only keeps files that will be needed for upcoming observations.

In parallel, CRDS delivers all files to DADS for permanent storage. In the event that a problem occurs in providing files to HSDP, they can normally be retrieved from this source.