Section 1. Introduction
------------------------

1.1 Purpose and Scope
~~~~~~~~~~~~~~~~~~~~~

This HST Reference Files Information Document specifies the structure and contents of the reference data described by the Calibration Refernce Data System (CRDS) that is used during HST Science Data Processing (HSDP) to calibrate science data. More formally, this document defines the format and usage of all the reference data described in CRDS. This data are generated from calibrations of the telescope's science instruments. Their characteristics are stored in an on-line calibration data base for use in both, the HSDP and *stenv*, while the reference data files are stored in various on-line caches, and in the HST archive holdings of the Mikulski Archive for Space Telescopes (MAST).

1.2 System Description
~~~~~~~~~~~~~~~~~~~~~~

1.2.1 HSDP 
^^^^^^^^^^

HSDP is responsible for generating science data products for the Hubble Space Telescope (HST). *stenv* software is run within HSDP to help accomplish this task.   After astronomical observations are made with HST, receives HST science telemetry files (PODs) from the Goddard Space Flight Center's PACOR facility, HSDP partitions the PODs into instrument-specific exposure data sets, determines the appropriate reference data for each exposure via queries of the on-line calibration data base, and invokes the appropriate *stenv* calibration software to generate calibrated science data products.   These products, along with the initial science telemetry files, are permanently saved in the HST archive, and an archive catalog describing the characteristics of these products is made available to archive researchers to assist in identifying HST data for download to accomplish scientific investigations.  

This initial processing and archiving of HST data is later supplemented by a reprocessing mode that allows improvements to the science data products. Data reprocessing is implemented for HST science instruments that are still taking active observations, and whose calibration software and reference data have not been finalized.  When a new calibration reference file for HST is delivered or a change is made to the calibration pipeline software, the science data products are regenerated using the initial HST science telemetry files, the latest HSDP and *stenv* software, and the best-possible reference data for the exposures as described in CRDS.   These updated science data products are distributed directly to the archive user, giving them the benefits of all updates made to the processing software and reference data since the data was originally archived.    Once an HST science instrument is no longer taking active observations (due to decommissioning or a hardware failure) and final calibration software and reference data are produced, a final reprocessing run is made for all of the archived data for that instrument.  The science data products generated during this final reprocessing run are saved in the HST archive and from then on, are distributed directly to archive users when requested.  


1.2.2 *stenv*
^^^^^^^^^^^^

*stenv* is a suite of Python  utility packages, and support software used to calibrate and analyze data from HST.
*stenv* is a set of installable Conda environments maintained by the Space Telescope Science Institute (STScI) in Baltimore, Maryland. 
The system runs on MacOS and Linux and the documentation can be found on-line (https://stenv.readthedocs.io/en/latest/getting_started.html).  The *stenv* calibration software is used without change in the HSDP
calibration pipeline for all HST data. Thus, HST observers can
recalibrate their data, examining intermediate calibration steps as
necessary, or modifying the reference data, using exactly the same code
used in the standard calibration procedures. 

1.2.3 CRDS Tools
^^^^^^^^^^^^^^^^

**CRDS** uses as set of tasks or scripts for preparing reference files for
submission to the Calibration Reference Data System (CRDS) and viewing the contents of this
data base. On-site users can view the data base
through the URL https://hst-crds.stsci.edu/

The **CRDS** package provides a set of tasks to prepare reference
files for installation in the CRDS. These task can be used via command
line commands as indicated in the CRDS User Guide (https://hst-crds.stsci.edu/static/users_guide/index.html) 
or if authorized by the management team, via the special services for deliverers in the https://hst-crds.stsci.edu/ 
pages.

There are several steps in submitting a reference file for installation
in CRDS. First, the header keywords and column names (if the reference
file is a table) are checked for correctness by running **certify**.
Using the CRDS Routine Submission Services option Extended Barch Submitt References deliver your data.
This process requires to fill a form with needed metadata to pupulate the CRDS database
and to keep track of the delivery of these files. 

In order to set up your account in these machines to run **certify** and **fitsverify**
add the following to your .setenv file: 
     
::

   setenv CRDS_ROOT /grp/hst/cdbs/tools/
   setenv CRDS_DATA ${CRDS_ROOT}/data/
   setenv PATH ${PATH}
   setenv PATH ${CRDS_ROOT}/bin/linux:/grp/software/Linux/RH4.5/i386_64/GCC4.2.4/bin
   setenv LD_LIBRARY_PATH ${LD_LIBRARY_PATH}
   setenv LD_LIBRARY_PATH
   ${CRDS_ROOT}/lib/linux:/grp/software/Linux/RH4.5/i386_64/GCC4.2.4/lib/gcc/x86_64-redhat-linux/4.2.4.
   
For this to take effect you need to logout and then login again.

1.3 HSDP/CRDS Interface
~~~~~~~~~~~~~~~~~~~~~~~

The interface between the HSDP and CRDS software comprises shared data
and protocols that enable data exchange.

The CRDS provides HSDP with data in the form of both reference files and
reference relations (i.e., tables) used to reduce science data from the
Wide Field/Planetary Camera (WF/PC), Wide Field/Planetary Camera 2 (WFPC
2), Faint Object Camera (FOC), Faint Object Spectrograph (FOS), High
Resolution Spectrograph (HRS), High Speed Photometer (HSP), Near
Infrared Camera (NICMOS), Space Telescope Imaging Spectrometer (STIS),
Advanced Camera for Surveys (ACS), Cosmic Origins Spectrograph (COS),
and Wide Field Camera 3 (WFC3). In turn, HSDP provides CRDS with science
data that are used to update these reference files and relations.
A reference file is either a GEIS format image (also known as *stenv*
image format) or a FITS-format image. All GEIS images consist of two
files, an ASCII header and a binary data file. A FITS image is a single
file consisting of an ASCII header combined with binary image data. A
reference table is either an *stenv* format table or a FITS table. Tables
consist of a single file that, unlike an image, may contain data of
several types.

CRDS does not retain the reference files or tables themselves, only
their names and applicable instrument configurations. The file names
corresponding to an instrument configuration or those related to a
particular observation may be obtained through Starview web (http://starview.stsci.edu/web/``#``).  The actual
files may then be requested from the Data Archiving and Distribution
System (DADS) or in many cases found in on-line archives on /grp/hst/cdbs/
in subdirectories with names like xref, xtab, yref and ytab. The first
letter is the distinguishing instrument letter, and the rest of the name
indicates reference files or tables. 

As observational data pass through the RSDP pipeline, the names of all
reference files and tables used in the calibration process are written
to the header files. All such tables and files are archived by CRDS to
DADS and supplied by CRDS to HSDP for use in calibrating the pipeline
observations.

The CRDS tool**uniqname** is used to name all of the calibration tables
and reference
files. Table 1-1 describes the criteria used by this task to assign a
file name. Each root name will have nine characters, numbered left to
right; these characters can be interpreted by replacing them with the
values shown in the table.

Table 1-1. Criteria for Assigning a Uniqname

+----------+--------------+----------------+---------------------------+
| Position |Parameter     |Characters Used |Comments                   |
+==========+==============+================+===========================+
| 1        |Year          |1-9, A-Z        |1981 = 1                   |
+----------+--------------+----------------+---------------------------+
| 2        |Month         | --             |January = 1                |
+----------+--------------+----------------+---------------------------+
| 3        |Day           |1-9, A-V        |First day of the month = 1 |
+----------+--------------+----------------+---------------------------+
| 4-5      |Hour          |00-23           | --                        |
+----------+--------------+----------------+---------------------------+
| 6-7      |Minute        |00-59           | --                        |
+----------+--------------+----------------+---------------------------+
| 8        |Seconds/2     |0-9, A-T        | --                        |
+----------+--------------+----------------+---------------------------+
| 9        |Instrument ID |F, M-O, V-Z     |See Table 1-2              |
+----------+--------------+----------------+---------------------------+

Table 1-2 shows the instrument IDs for the last letter of the *unique
name*.

Table 1-2. Instrument Codes

+--------+---------------------+
| Letter |Instrument           |
+========+=====================+
| F      |FGS                  |
+--------+---------------------+
| I      |WFC3                 |
+--------+---------------------+
| J      |ACS                  |
+--------+---------------------+
| L      |COS                  |
+--------+---------------------+
| N      |NICMOS               |
+--------+---------------------+
| O      |STIS                 |
+--------+---------------------+
| V      |HSP                  |
+--------+---------------------+
| U      |WFPC 2               |
+--------+---------------------+
| W      |WF/PC                |
+--------+---------------------+
| X      |FOC                  |
+--------+---------------------+
| Y      |FOS                  |
+--------+---------------------+
| Z      |HRS                  |
+--------+---------------------+
| M      |Multiple instruments |
+--------+---------------------+

1.3.1 Calibration Tables
^^^^^^^^^^^^^^^^^^^^^^^^

Calibration tables are instrument specific. Without a full complement of
calibration tables for a given instrument, the pipeline calibration will
seldom work. These calibration tables are binary files produced,
manipulated, and examined by various *stenv* utilities.

The type of calibration table can be determined from the table name
extension, which for legacy instruments
follows certain conventions.
Table
1-3 describes the conventions for these reference tables.

Table 1-3. Conventions for Legacy Instruments  Reference Table Names

+----------+-----------------+--------------------------------------+
| Position |Characters Used  |Comments                              |
+==========+=================+======================================+
| 1        |C, T             |C = catalog, T = synthetic photometry |
+----------+-----------------+--------------------------------------+
| 2        |F, I-J, L-O, V-Z |Instrument ID (see Table 1-2)         |
+----------+-----------------+--------------------------------------+
| 3        |0-9, A-Z         |[#f1]_                                |
+----------+-----------------+--------------------------------------+

.. [#f1] Tables are generally numbered sequentially for each instrument. Certain
    tables have a specific character for the third position in the extension
    carried over from an "old" relation; e.g., CMG is the extension for the
    HRS and FOS Paired Pulse Tables.

Table 1-4 gives a comprehensive list of reference tables. The root name
of these reference tables is determined by the naming algorithm in the
*stenv* task **uniqname**. For STIS and NICMOS both images and tables are
in FITS format and have a file name extension of the form XXX.FITS,
where XXX is indicative of the calibration function of the file.

Table 1-4. List of Reference Tables 
   
+------------+-------------+---------------------------------------------------------+
| Instrument | | Extension | | Comments                                              |
+============+==========+============================================================+
| FOC        | | CXU       | | Flat fields                                           |
+------------+-------------+---------------------------------------------------------+
| FOS        | | CY0       | | Aperture parameters                                   |
|            | | CY1       | | Aperture position parameters                          |
|            | | CY2       | | Emission lines                                        |
|            | | CY3       | | Filter widths                                         |
|            | | CY4       | | Wollaston/wavelength parameters                       |
|            | | CY5       | | Sky shift parameters                                  |
|            | | CY6       | | Wavelength coefficients                               |
|            | | CY7       | | GIMP-correction coefficients                          |
|            | | CY8       | | Predicted background count rates                      |
+------------+-------------+---------------------------------------------------------+
| HRS        | | CZ1       | | Line mapping parameters                               |
|            | | CZ2       | | Sample mapping parameters                             |
|            | | CZ3       | | Detector parameters                                   |
|            | | CZ4       | | Wavelength ranges                                     |
|            | | CZ5       | | Spectral order constants                              |
|            | | CZ6       | | Dispersion constants                                  |
|            | | CZ7       | | Thermal constants                                     |
|            | | CZ8       | | Incidence angle constants                             |
|            | | CZ9       | | Echelle interpolation constants                       |
|            | | CZA       | | Echelle non-interpolation constants                   |
|            | | CZB       | | Scattered light subtraction scale factors             |
+------------+-------------+---------------------------------------------------------+
| HSP        | | CV0       | | Aperture areas                                        |
|            | | CV1       | | High voltage factor                                   |
|            | | CV2       | | Gain factor CV3 Pre-amp                               |
|            | | CV4       | | Efficiency                                            |
|            | | CV5       | | Dark count                                            |
|            | | CV6       | | Time bias                                             |
|            | | CV7       | | CVC offset                                            |
|            | | CV8       | | Dead time                                             |
|            | | CV9       | | Dark aperture                                         |
+------------+-------------+---------------------------------------------------------+
| WF/PC      | | CW0       | | Photometry group parameters                           |
+------------+-------------+---------------------------------------------------------+
| NICMOS     | | PHT       | | Photometric calibration table                         |
|            | | BKG       | | Predicted background table                            |
|            | | IDC       | | Geometric Distortion Coefficients Table               |
|            | | ZPR       | | Nonlincor Zeropoint Scaling Table                     |
|            | | NIC       | | Nonlinearity Power Law Tables                         |
+------------+-------------+---------------------------------------------------------+
| STIS       | | CCD       | | CCD parameters table                                  |
|            | | A2D       | | Analog-to-digital correction table                    |
|            | | CRR       | | Cosmic ray rejection parameters table                 |
|            | | BPX       | | Bad pixel table                                       |
|            | | LIN       | | MAMA linearity table                                  |
|            | | PHT       | | Photometric conversion table                          |
|            | | APT       | | Aperture throughput table                             |
|            | | APD       | | Aperture description table                            |
|            | | LMP       | | Template cal lamp spectra table                       |
|            | | IDC       | | Image distortion correction table                     |
|            | | SDC       | | 2-D spectrum distortion correction table              |
|            | | IAC       | | Incidence angle correction table                      |
|            | | DSP       | | Dispersion coefficients table                         |
|            | | 1DT       | | 1-D spectrum trace table                              |
|            | | 1DX       | | 1-D extraction parameter table                        |
|            | | MOC       | | MAMA offset correction table                          |
|            | | CDS       | | Cross-Disperser Scattering table                      |
|            | | ECH       | | Echelle Scattering Table                              |
|            | | EXS       | | Echelle Cross-Dispersion Scattering Table             |
|            | | HAL       | | Detector Halo Table                                   |
|            | | SRW       | | Scattering Reference Wavelengths Table                |
|            | | TDC       | | Dark Correction Table                                 |
|            | | TEL       | | Telescope Point Spread Function Table                 |
+------------+-------------+---------------------------------------------------------+
| WFPC 2     |             | | (No reference tables)                                 |
+------------+-------------+---------------------------------------------------------+
| ACS        | | A2D       | | Analog-to-Digital Table                               |
|            | | BPX       | | Bad Pixel Table                                       |
|            | | CCD       | | CCD Characteristics Table                             |
|            | | OSC       | | Overscan Region Table                                 |
|            | | CRR       | | Cosmic Ray Rejection Parameter Table                  |
|            | | LIN       | | MAMA Linearity Table                                  |
|            | | PHT       | | Photometry and Throughput Table                       |
|            | | IDC       | | Image Distortion Coefficients Table                   |
|            | | SYN       | | Synphot Table                                         |
|            | | TMC       | | COMPTAB                                               |
|            | | TMG       | | GRAPHTAB                                              |
+------------+-------------+---------------------------------------------------------+
| FOS, HRS   | | CMG       | | Paired pulse                                          |
+------------+-------------+---------------------------------------------------------+
| WFC3       | | A2D       | | Analog-to-Digital Table                               |
|            | | BPX       | | Bad Pixel Table                                       |
|            | | CCD       | | CCD Characteristics Table                             |
|            | | OSC       | | Overscan Region Table                                 |
|            | | CRR       | | Cosmic-Ray Rejection Parameter Table                  |
|            | | IDC       | | Image Distortion Coefficients Table                   |
+------------+-------------+---------------------------------------------------------+
| Multiple   | | TMG       | | Component list lookup [#f2]_                          | 
|            | | TMC       | | Component graph [#f2]_                                |
|            | | TAB       | | Component throughput as function of wavelength [#f2]_ |
+------------+-------------+---------------------------------------------------------+

.. [#f2] These tables are currently used by the FOC, HSP, and WFPC 2.

1.3.2 Throughput Tables
^^^^^^^^^^^^^^^^^^^^^^^

Throughput tables are a special type of calibration table that uses
different naming conventions. These tables are used to compute the
sensitivity of an instrument in a particular mode. They are binary files
that are produced, manipulated, and examined by various *stenv* utility
programs.

The throughput table has the following conventions for the names:

1. The extension is always _syn.fits
2. The root name always has an underscore and three numbers at the right
   side representing the version number of the table.

3. The first part of the name (up to the "-" character preceding the
   version number) is the name of the component corresponding to the
   throughput table.

This name should be used in cataloging the throughput tables.

1.3.3 Reference Files
^^^^^^^^^^^^^^^^^^^^^

Reference files are observation specific. Generally, different
observations will require different sets of reference files for their
calibrations. Reference files are composed of a header file (in FITS
format) and a binary data file.
The extensions to reference files are determined according to the
conventions described in Table
1-5.

Table 1-5. Reference File Extensions

+----------+----------------+--------------------------------------------------------------+
| Position |Characters Used |Comments                                                      |
+==========+================+==============================================================+
| 1        |R, B            |R = reference file, B = associated data quality file          |
+----------+----------------+--------------------------------------------------------------+
| 2        |1-9, A-Z        |Reference files are numbered sequentially for each instrument |
+----------+----------------+--------------------------------------------------------------+
| 3        |H, D            |H = header file, D = data file                                |
+----------+----------------+--------------------------------------------------------------+

The root name of the reference files is determined by the naming
algorithm used by the **uniqname** task in *stenv*. Table 1-6 shows a
complete list of these reference files by instrument.

Table 1-6. Reference File Names 

+------------+-------------+--------+----------------------------------------+
| Instrument |Extension    |Type    |Comments                                |
+============+=============+========+========================================+
| FOC        | | R0H, R0D  | | BAC  | | Background                           |
|            | | R1H, R1D  | | ITF  | | Intensity transfer function          |
|            | | R2H, R2D  | | UNI  | | Uniform detector efficiency          |
|            | | R3H, R3D  | | SDE  | | Spectrographic detector efficiency   |
|            | | R5H, R5D  | | GEO  | | Geometric distortion                 |
|            | | R7H, R7D  | | BLM  | | Data quality initialization          |
+------------+-------------+--------+----------------------------------------+
| FOS        | | R0H, R0D  | | BAC  | | Background                           |   
|            | | R1H, R1D  | | FLT  | | Flat field                           |
|            | | R2H, R2D  | | IVS  | | Inverse sensitivity                  |
|            | | R3H, R3D  | | RET  | | Retardation                          |
|            | | R4H, R4D  | | DDT  | | Disabled diode                       |
|            | | R5H, R5D  | | QIN  | | Data quality initialization          |
+------------+-------------+--------+----------------------------------------+
| HRS        | | R0H, R0D  | | DIO  | | Diode response                       |
|            | | R1H, R1D  | | PHC  | | Photocathode response                |
|            | | R2H, R2D  | | VIG  | | Vignetting                           |
|            | | R3H, R3D  | | ABS  | | Absolute sensitivity                 |
|            | | R4H, R4D  | | NET  | | Wavelength net                       |
|            | | R5H, R5D  | | QIN  | | Data quality initialization          |
+------------+-------------+--------+----------------------------------------+
| HSP        |             |        | | (No reference files)                 |
+------------+-------------+--------+----------------------------------------+
| NICMOS     |             | | MSK  | | Static bad pixel mask                |
|            |             | | NOI  | | Noise file                           |
|            |             | | LIN  | | Linearity correction file            |
|            |             | | DRK  | | Dark current file                    |
|            |             | | TDD  | | Temperature Dependent Darks          |
|            |             | | FLT  | | Flat field file                      |
|            |             | | TDF  | | Temperature Dependent Flat Fields    |
|            |             | | ILM  | | Background illumination pattern file |
|            |             | | PMD  | | Persistence Model Files              |
|            |             | | PMK  | | Persistence Mask Files               |
|            |             | | SDK  | | Special Darks for SAACLEAN           |
+------------+-------------+--------+----------------------------------------+
| STIS       |             | | BIA  | | Bias image file                      |
|            |             | | DRK  | | Dark image file                      |
|            |             | | SSC  | | Shutter shading correction           |
|            |             | | PFL  | | Pixel-to-pixel flat                  |
|            |             | | DFL  | | Delta flat                           |
|            |             | | LFL  | | Low-order flat                       |
|            |             | | SSD  | | Small scale distortion correction    |
+------------+-------------+--------+----------------------------------------+
| ACS        |             | | BIA  | | Bias Reference Image                 |
|            |             | | DRK  | | Dark Image File                      |
|            |             | | PFL  | | Pixel-to-pixel Flat                  |
|            |             | | LFL  | | Large Scale Flat                     |
|            |             | | SHD  | | Shutter-Shading Correction Image     |
|            |             | | FLS  | | Post Flash Image                     |
|            |             | | DXY  | | Delta Distortion Image               |
+------------+-------------+--------+----------------------------------------+
| WFC3       |             | | BIA  | | Bias Image File                      |
|            |             | | DRK  | | Dark Image File                      |
|            |             | | PFL  | | Pixel-to-Pixel Image File            |
|            |             | | DFL  | | Delta Flat Image File                |
|            |             | | LFL  | | Low-Order Flat Image File            |
|            |             | | SHD  | | Shutter-Shading Image File           |
|            |             | | FLS  | | Post-Flash Image                     |
|            |             | | LIN  | | Linearity Correction File            |
+------------+-------------+--------+----------------------------------------+
| COS        |             | | TBS  |                                        |
+------------+-------------+--------+----------------------------------------+
| WF/PC      | | R0H, R0D  | | MSK  | | Static mask                          |
|            | | R1H, R1D  | | A2D  | | A-to-D lookup tables                 |
|            | | R2H, R2D  | | BAS  | | Bias                                 |
|            | | R3H, R3D  | | PRF  | | Preflash                             |
|            | | R4H, R4D  | | SPG  | | Superpurge                           |
|            | | R5H, R5D  | | DRK  | | Dark                                 |
|            | | R6H, R6D  | | FLT  | | Flat field                           |
+------------+-------------+--------+----------------------------------------+
| WF/PC      | | B2H, B2D  | | BAS  | | Bias data quality                    |
|            | | B3H, B3D  | | PRF  | | Preflash data quality                |
|            | | B4H, B4D  | | SPG  | | Superpurge data quality              |
|            | | B5H, B5D  | | DRK  | | Dark data quality                    |
|            | | B6H, B6D  | | FLT  | | Flat field data quality              |
+------------+-------------+--------+----------------------------------------+
| WFPC 2     | | R0H, R0D  | | MSK  | | Static mask                          |
|            | | R1H, R1D  | | A2D  | | A-to-D lookup tables                 |
|            | | R2H, R2D  | | BAS  | | Bias                                 |
|            | | R3H, R3D  | | DRK  | | Dark                                 |
|            | | R4H, R4D  | | FLT  | | Flat field                           |
|            | | R5H, R5D  | | SHD  | | Shutter shading correction           |
+------------+-------------+--------+----------------------------------------+
| WFPC 2     | | B2H, B2D  | | BAS  | | Bias data quality                    |
|            | | B3H, B3D  | | DRK  | | Dark data quality                    |
|            | | B4H, B4D  | | FLT  | | Flat field data quality              |
+------------+-------------+--------+----------------------------------------+

1.4 Communication Between HSDP and CRDS
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The CRDS underwent a complete redesign in 1996, and the new system was
brought on-line in early 1997. This coincided with the introduction of
STIS and NICMOS.

The method of communication between CRDS and HSDP changed a great deal
and incorporated more automatic methods and quality checks. CRDS
releases to HSDP the reference files and calibration tables that have
been delivered along with a catalog describing their use and an SQL
command sequence that is used to install directly to the HSDP data base.
The catalog and SQL commands also indicate which of the previously
installed files have now been superseded and are to be removed from the
HSDP file collection. In this way, HSDP only keeps files that will be
needed for upcoming observations.

In parallel, CRDS delivers all files to DADS for permanent storage. In
the event that a problem occurs in providing files to HSDP, they can
normally be retrieved from this source.