chandra_repro

Synopsis

Reprocess a Chandra dataset

chandra_repro  indir outdir [root] [badpixel] [process_events]
[destreak] [set_ardlib] [check_vf_pha] [pix_adj] [tg_zo_position]
[asol_update] [pi_filter] [patch_hrc_ssc] [cleanup] [clobber] [verbose]

Description

The chandra_repro reprocessing script automates the recommended data processing steps presented in the CIAO analysis threads. The script reads data from the standard data distribution (e.g. primary and secondary directories) and creates a new bad pixel file, a new level=2 event file, and a new level=2 Type II PHA file with the appropriate response files (grating data only).

chandra_repro may be used to reprocess any ACIS and HRC imaging or grating data.

There are parameters to control certain reprocessing steps, such as creating the new bad pixel file or destreaking the data. Users who wish to have finer-grained control over the reprocessing parameters should instead follow the step-by-step instructions in the CIAO Data Preparation Threads.

Where are the processed files?

The chandra_repro script stores the processed files in the location given by the outdir parameter. This can be relative to the input directory - e.g. a repro/ directory within each obsid - or everything can be placed into a single directory (when processing multiple directories). The grating products - PHA2, grating ARFs, and RMFs - are placed in the tg/ subdirectory of the outroot directory.

CALDB Version

The chandra_repro script does not overwrite the CALDBVER keyword in the event header since not all calibrations may have been applied. However, it does record the version of the CALDB in comments that users can check to verify that they applied the intended calibration updates.

unix% dmlist acis_repro_evt2.fits header | grep 'chandra_repro was run' 
 --  COMMENT               chandra_repro was run with CALDB 4.5.1
 --  HISTORY               PARM  :value=chandra_repro was run with CALDB 4.5.1            ASC00639

Examples

Example 1

unix% chandra_repro

Reprocess the dataset in the current working directory with the default script settings. The default verbosity of 1 means that status messages will be printed to the screen.

The script should be run from directory which contains the contains the primary/ and secondary/ data directories from a standard Chandra Data Archive download tarfile:

unix% pwd
/data/1838/

unix% ls
primary/  secondary/

Example 2

unix% chandra_repro verbose=0

Reprocess the dataset in the current working directory with the default settings. Nothing will be printed to the screen, unless the script detects an error.

Example 3

unix% chandra_repro indir=/data/1838 outdir=/data/1838/new cleanup=no

Reprocess the dataset in directory /data/1838, writing the new files to /data/1838/new . Save all of the intermediate files (cleanup=no).

Example 4

unix% chandra_repro set_ardlib=no

The data will be reprocessed, but the bad pixel filename will not be set in ardlib.par.

Example 5

unix% chandra_repro verbose=1 mode=h
Resetting afterglow status bits in evt1.fits file...

Running acis_build_badpix and acis_find_afterglow to create a new bad pixel file...

Running acis_process_events to reprocess the evt1.fits file...
Filtering the evt1.fits file by grade and status...
Applying the good time intervals from the flt1.fits file...
The new evt2.fits file is: /data/1838/repro/acisf01838_repro_evt2.fits

Updating the event file header with chandra_repro HISTORY record
Setting observation-specific bad pixel file in local ardlib.par.

Cleaning up intermediate files

WARNING: Observation-specific bad pixel file set for session use:
         /data/1838/repro/acisf01838_repro_bpix1.fits
         Run 'punlearn ardlib' when analysis of this dataset completed.

The data have been reprocessed.
Start your analysis with the new products in
/data/1838/repro

At the verbose=0 setting, only critical warnings and errors are printed to the terminal. At verbose=1 (and above) additional information about each data set is printed.

Example 6

unix% chandra_repro indir=1838,1839 outdir=""

Will reprocess the data in directories 1838/ and 1839/. The outputs will be placed in 1838/repro and 1839/repro. A warning will be issued since multiple indir are used and the default set_ardlib=yes are incompatible (see details below). The data are not reprojected to the tangent plane, nor are they corrected for any small astrometric shifts.

Example 7

unix% chandra_repro "*" outdir=""

chandra_repro will try to process all the obsids in the current directory. The "*" must be in quotes (single or double) or must be escaped, \*, from the shell. All other stack expansion rules also apply (see ahelp stack).

The default outroot parameter is set such that a "repro" directory will be created in each obsid's subdirectory.

unix% download_chandra_obsid 1838 
...
unix% /bin/ls 
1838
unix% chandra_repro "*" mode=h verb=1

Processing input directory '/data/1838'

Resetting afterglow status bits in evt1.fits file...

Running acis_build_badpix and acis_find_afterglow to create a new bad pixel file...

Running acis_process_events to reprocess the evt1.fits file...
...
unix% /bin/ls 1838/*
1838/axaff01838N002_VV001_vv2.pdf  1838/oif.fits

1838/primary:
acisf01838_000N003_bpix1.fits.gz  acisf01838N003_full_img2.fits.gz
acisf01838_000N003_fov1.fits.gz   acisf01838N003_full_img2.jpg
acisf01838N003_cntr_img2.fits.gz  orbitf084197100N001_eph1.fits.gz
acisf01838N003_cntr_img2.jpg      pcadf084244404N003_asol1.fits.gz
acisf01838N003_evt2.fits.gz

1838/repro:
acisf01838_000N003_bpix1.fits  acisf01838_asol1.lis
acisf01838_000N003_fov1.fits   acisf01838_repro_bpix1.fits
acisf01838_000N003_msk1.fits   acisf01838_repro_evt2.fits
acisf01838_000N003_mtl1.fits   acisf084245776N003_pbk0.fits
acisf01838_000N003_stat1.fits  pcadf084244404N003_asol1.fits

1838/secondary:
acisf01838_000N003_evt1.fits.gz     acisf084244478N003_3_bias0.fits.gz
acisf01838_000N003_flt1.fits.gz     acisf084244478N003_4_bias0.fits.gz
acisf01838_000N003_msk1.fits.gz     acisf084244478N003_5_bias0.fits.gz
acisf01838_000N003_mtl1.fits.gz     acisf084245776N003_pbk0.fits.gz
acisf01838_000N003_stat1.fits.gz    aspect
acisf084244478N003_0_bias0.fits.gz  axaff01838N002_VV001_vvref2.pdf.gz
acisf084244478N003_1_bias0.fits.gz  ephem
acisf084244478N003_2_bias0.fits.gz

Example 8

unix% chandra_repro 23206 outdir=""
tg_zo_position="83.8156671,-5.3861617"

By default chandra_repro will use the 0th order location taken from the standard Level2 event file, primary/*evt2.fits file. This will be for the brightest source. However, users can optionally supply the celestial coordinates for the source to use for the 0th order location. In this example we run chandra_repro with the RA and Dec coordinates for the source specified in decimal degrees. Users can also supply the coordinates in sexagesimal notation, e.g.

tg_zo_position="5:35:15.7594,-5:23:10.191"

Parameters

Detailed Parameter Descriptions

Parameter=indir (file required default=./ stacks=yes)

Input directory

The directory which contains the primary/ and secondary/ data directories from a standard Chandra Data Archive download tarfile. It may also be a directory which contains all of the downloaded data files (i.e. not organized into primary/ and secondary/ subdirectories).

Special Input Case: multi-OBI data

In order to run chandra_repro on observations that have multiple observation intervals (OBIs), the input data has to be separated into different input directories. Users can refer to the Multi-OBI Observations why topic for information on what these datasets are, and the splitobs script to create separate directories for the different OBIs.

Interleaved-mode datasets

The chandra_repro script can be run on interleaved mode datasets. In this case, the output files will use the labels "e1" and "e2" to separate the two sets of outputs. Alternately, the splitobs script can be used to create separate directories for the primary (e1) and secondary (e2) data periods, after which the data can be processed as separate datasets.

Multiple Datasets

indir can be a stack of input directories. Each subdirectory will be processed one at a time; each must have the primary and secondary directories. See the outdir parameter for details on where reprocessed products will be created. Note: since there is only one ardlib.par, the set_ardlib parameter is forced to be "no" since it cannot be set correctly for all directories. Multiple inputs are NOT reprojected to the same tangent plane nor are they corrected for astrometric shifts.

Parameter=outdir (file required default="")

Output directory

The output directory for the reprocessed data files. This directory will also contain symbolic links to the files which are required for data analysis, e.g. the aspect solution and the mask file. If the file system does not support symbolic links then copies are made.

The default, outdir="", will create a "repro" subdirectory underneath the indir directory.

Multiple Datasets

When the indir parameter is a stack, the outdir must be specified in one of the following ways. If outdir="", then a "repro" subdirectory will be created in each indir in the stack. If outdir="./repro", or any other single directory name, then all the outputs from all the indirs will be put into the same directory. Finally, users can specify a stack of outdirs that match one-to-one with the indirs for the most control over the location of where each indir goes.

Parameter=root (string not required)

Root for output filenames

If not specified, the script uses the first section of the level=1 event filename (up to the first underscore) to define the root, e.g. "acisf01838" is the root of acisf01838_000N002_evt1.fits.

New files that are created by the script have the filename format <root>_repro_<type>.fits, e.g. acisf01838_repro_evt2.fits or acisf01838_000N003_bpix1.fits.

Parameter=badpixel (boolean not required default=yes)

Create a new bad pixel file?

If "badpixel=yes" (the default), a new bad pixel file is created for the dataset.

ACIS Data:

• status bits 1-5, 14-20 and 23 in the input event file are reset by the acis_clear_status_bits script to remove any previous bad pixel identification and destreaking.
• destreak is run to identify streak events, if the observation used ACIS-S4 (ccd_id=8) and destreak=yes. Flagging the streaks first prevents the misidentification of pixels with a lot of streak events as being hot pixels.
• acis_build_badpix and acis_find_afterglow are run to create a new bad pixel file which flags hot pixel and afterglow events in the event file.

HRC Data:

• hrc_run_hotpix is run to define the valid coordinate regions in the detectors and to identify hot pixels in the event file.

The new bad pixel that is created is used in reprocessing the data, assuming "process_events=yes".

ACIS continuous-clocking mode

This parameter setting does not apply when the input data were taken in ACIS continuous-clocking mode, as the ACIS afterglow and hot pixel tool cannot be used with this data mode.

Related webpages:

o) Customizing an ACIS Bad Pixel File (experts only)

o) New Observation-Specific HRC Bad Pixel File thread

Parameter=process_events (boolean not required default=yes)

Create a new level=2 event file?

If "process_events=yes" (the default), then the acis_process_events or hrc_process_events tool is run to create a new level=1 event file with the latest calibration applied.

The standard grade, status, and good time filters are applied by the tool dmcopy to create a new level=2 event file. This includes the pulse-height filter to reduce background in LETG/HRC-S observations.

For grating data, a new level=2 Type II PHA file is extracted along with the ARF and RMF files which are stored in the tg/ subdirectory.

ACIS Reprocessing Steps

• the latest charge transfer inefficiency (CTI) correction
• the latest time-dependent gain adjustment
• the latest gain map
• a new bad pixel file
• PHA randomization
• a sub-pixel adjustment (if specified, see "pix_adj" parameter)
• clean the VFAINT background (see "check_vf_pha" parameter)
• continuous clocking mode times of arrival and pulse heights

HRC Reprocessing Steps

• the latest time-dependent gain adjustment
• the AMP_SF correction
• the degap correction
• recomputing average HRC dead time corrections
• pixel randomization (if specified, see "pix_adj" parameter)

Related webpages:

o) Reprocessing Data to Create a New Level=2 Event File thread

o) acis_process_events help file

o) hrc_process_events help file

Parameter=destreak (boolean not required default=yes)

Destreak the ACIS-8 chip?

There is a flaw in the serial readout of the ACIS chips, causing a significant amount of charge to be randomly deposited along pixel rows as they are read out. If "destreak=yes" (the default), the destreak tool is run on the ACIS-S4 chip to detect probable streak events. The flagged events are then filtered out of the new level=2 event file.

Related webpages:

o) Destreaking ACIS Data why topic

o) destreak help file

Parameter=set_ardlib (boolean not required default=yes)

Set ardlib.par with the bad pixel file?

If "set_ardlib=yes" (the default), the observation-specific bad pixel file is set in the ardlib.par file so that it is available to the CIAO tools during data analysis.

Setting "set_ardlib=no" may be useful if you are analyzing another dataset while chandra_repro is running. In this case, you would not want ardlib.par to be modified until the other analysis tasks are finished.

Remember to "punlearn" your ardlib.par file after completing analysis of this dataset to ensure that the proper bad pixel maps are used the next time that ardlib.par is referenced by a tool.

Related webpages:

o) Setting the Observation-specific Bad Pixel Files

Multiple datasets

If there are multiple input datasets, then the single ardlib.par cannot be setup correctly for all of them. Therefore chandra_repro script will omit this step if multiple input directories are used.

Parameter=check_vf_pha (boolean not required default=no)

Clean ACIS background in VFAINT data?

In ACIS very faint mode, acis_process_events can use the pulse heights in the outer 16 pixels of the 5x5 event island to help distinguish between good X-ray events and bad events that are most likely associated with cosmic rays.

If "check_vf_pha=yes" (not the default), then the ACIS particle background for very faint mode observations is cleaned. The potential background event - events where one or more of those outer pixels is greater than the default split threshold - is flagged and filtered out of the event file.

Prior to the June 2012 release, the default for check_vf_pha was set to "yes". It has been determined that this could remove good events in observations with modestly bright point sources. Therefore, the default is now set to "no".

The value of this parameter is passed to the "check_vf_pha" parameter of acis_process_events when "process_events=yes".

Related webpages:

o) ACIS VFAINT Background Cleaning why topic

Parameter=pix_adj (string not required default=default)

Pixel randomization: default|edser|none|randomize

If "pix_adj=default" (the default), then the default value of the tool's pixel randomization parameter is used. For ACIS timed exposure mode data, a subpixel algorithm is applied (acis_process_events pix_adj=EDSER). For ACIS continuous-clocking mode data, no subpixel adjustment is applied (acis_process_events pix_adj=NONE). For HRC data, randomization is not applied (hrc_process_events rand_pix_size=0.0).

The following table summarizes the chandra_repro pix_adj options and how each is passed to acis_process_events and hrc_process events (when "process_events=yes").

Related webpages:

o) ACIS Pixel Randomization why topic

Parameter=tg_zo_position (string not required default=evt2)

How to determine the gratings 0th order location.

The tg_zo_position parameter allows the user to choose how the gratings 0th order location is selected. The options are:

• "evt2" : If tg_zo_positoin="evt2", the default, then chandra_repo will use the 0th order location from the REGION block in the archived Level2 event file (located in the primary/ directory). This may be the best choice especially if the 0th order location was manually adjusted by the Verification and Validation (V&V) scientist. The region is defined in physical coordinates. If data has been reprojected or if a fine astrometric shift has been added to the aspect solution, this may invalidate the region coordinates.
• "detect" : If tg_zo_position="detect", then chandra_repro will run the tgdetect2 tool to detect the 0th order location. This will use either tgdetect or tg_findzo to determine the source location.
• RA and Dec. coordinates : Lastly, users can supply their own coordinates for the 0th order location. This may be needed if the source of interest is not the brightest source in the field. The source position can specified in either decimal degrees or in sexagesimal notation.

See ahelp coords_format for example of the supported syntax.

The 0th order location is used to create the gratings mask to determine which events are assigned gratings coordinates, and for the case of HETG, to assign events to the HEG or MEG arm.

Related webpages:

Source Position for Grating Data with a Piled or Blocked Zero Order

Correcting a Misplaced Zero-order Source Position.

Multiple datasets

If the user runs chandra_repro with multiple datasets, then the same position parameter option, including coordinates, are used for each gratings dataset.

Parameter=asol_update (boolean not required default=yes)

Apply boresight correction to aspect solution file(s), if necessary.

If this parameter is set to "yes" and the input aspect solution file has not already had the boresight correction applied, then chandra_repro will run the asp_offaxis_corr tool script to create an updated aspect solution file. If the aspect solution file has already been corrected then this step is skipped.

The effect of the boresight correction is to update the location of the aimpoint while preserving the celestial coordinates (RA and Dec) of the events. The sky coordinates (X,Y) and detector coordinates (DETX, DETY) may change by as much as 30 arcsec based (typical changes are much smaller).

Parameter=pi_filter (boolean not required default=yes)

Should the additional PI filter be applied to HRC-S+LETG data?

This parameter controls whether the optional HRC-S+LETG PI filter should be used. The PI filter is designed to suppress the background with only minor effect on the source spectrum.

Parameter=patch_hrc_ssc (boolean not required default=no)

Look for evidence of HRC Secondary Science Corruption (SSC) and patch dead-time factors if detected.

SSC results from a byte-shift anomaly which occasionally causes a portion of the housekeeping data to be corrupted. The symptom is dropouts in the dead-time-factor (dtf) which can be seen in a plot of the dtf1 file's dtf valuse vs time, where the dtf is significantly (>10%) below the median value (~1.0).

Event data are good during the SSC times. Standard data processing, however, will create multiple GTI intervals around the low dtf times, rejecting some events, and lowering the dead-time-correction factor (DTDOR).

Note: telemetry saturation from a bright source can also lower the dtf; however, this tends to be a more continuous and sustained lowering of the value.

Parameter=cleanup (boolean not required default=yes)

Cleanup intermediate files on exit

If "cleanup=yes" (the default), intermediate data files are deleted when the script ends. When the parameter is set to "no", these files remain in the "outdir" directory.

Parameter=clobber (boolean not required default=yes)

Clobber existing files?

Overwrite existing files with the same filename?

Parameter=verbose (integer not required default=1 min=0 max=5)

The amount of information printed to the screen

The default verbosity setting (1) prints status messages as the script runs. Higher verbosity settings print the commands that are being run. Setting verbose=0 turns off all screen output except errors.

Changes in the scripts 4.17.0 (December 2024) release

New HRC Science Science Corruption tool

A new parameter has been added: patch_hrc_ssc, which when set to "yes" will run the patch_hrc_ssc tool. This tool looks for corruption in the HRC secondary science data and will replace (patch) the corrupted values in the dead time factors file. This typically results in a very slightly higher average DTCOR value and elimination of some very short, erroneously tagged bad time intervals.

Changes in the scripts 4.16.2 (August 2024) release

On Board Computer (OBC) mode

Some observations, mostly of either the Earth or the Moon, were done without using the aspect camera to determine attitude and pointing. These observations rely on the On Board Computer (OBC) aspect solution, osol1.fits. This update allows chandra_repro to use the OBC solution files for these observations.

Solar System Objects

This fixes a problem with some solar system objects, including the OBC mode objects, that have multiple aspect solution files; they must be merged into a single file before running sso_freeze. This change also makes a failure in sso_freeze non-fatal. The rest of the event processing will proceed if sso_freeze is unable to compute object-centered coordinates. This happens for most of the EARTH observations.

Fix for older datasets

Internal change to how the r4_header_update script is run to deal with observations that are missing the "*FILE" keywords needed to automatically locate the parameter block file and aspect solution file(s).

Changes in the scripts 4.15.0 (December 2022) release

A problem when multiple SSO eph1 files are found has been fixed.

A bug in dmcopy prevented tgdetect2 from selecting the tg_findzo method. A work-around has been implemented to allow it to do so.

Changes in the scripts 4.14.2 (April 2022) release

Please read the V&V report

The script now notes the location of the Verification and Validation (V&V) report for each observation. Please read this, even if you decide not to run chandra_repro, as it is where possible issues in the observation are reported.

Specifying the zeroth-order location for grating observations

The parameter "recreate_tg_mask" has been replaced with the "tg_zo_position" parameter. For gratings datasets, users can now choose whether to use the 0th order location from the existing Level2 events file from the archive (tg_zo_position="evt2"), detect the zeroth order postion using tgdetect2 (tg_zo_position="detect"), or to supply the RA and Dec coordinates of the source location in either decimal degrees or sexagesimal notation.

Solar System Object Observations

chandra_repro will now run sso_freeze on the event file and aspect solution file if it finds a solar system ephemeris file in the primary/ directory along with the definitive ephemeris (orbit*_eph1.fits). The solar system ephemeris file names end with _eph1.fits and are prefixed with the object name (for example commetXYZ or jupiter). This will add additional object-centered coordinate columns to the event file and aspect solution files.

Changes in the scripts 4.14.0 (December 2021) release

Updated the Level 2 event file keywords : CONTENT="EVT2" and HDUCLAS2="ACCEPTED".

Changes in the script 4.13.3 (September 2021) release

The HRC-S/LETG bow-tie filter, designed to reduce background, is not appropriate for data acquired after the HRC instrument gain was changed. The bow-tie filter is marked invalid in CALDB 4.9.6 for these data. chandra_repro has been modified to correctly recognize that the file is not appropriate and continue processing without it.

Changes in the scripts 4.13.1 (March 2021) release

Now updates the reprocessed aspect solution file, pcadf*_repro_asol1.fits, to include the RA_NOM, DEC_NOM, and ROLL_NOM. They keywords are removed the by asp_offaxis_corr tool but are required by MARX.

The new parameter, pi_filter, controls whether the HRC-S+LETG background PI filter is applied or not, see the HRC Why topic on gain for details. The default is 'yes' to apply the filter. Users should only set this parameter to 'no' if they have been instructed to do.

Changes in the scripts 4.13.0 (December 2020) release

Update boresight correction in aspect solution files

If the script can only find original-format aspect solution files (those with CONTENT=ASPSOL) and with the new parameter "asol_update=yes", then chandra_repro will run the new asp_offaxis_corr tool to apply the DY,DZ, and DTHETA boresight corrections directly to the RA, Dec, ROLL, and quaternion values. This new aspect solution file will have CONTENT=ASPSOLOBI and should be used for all data analysis.

Other

Updates for tool changes in CIAO 4.13. This includes changes in the hrc_process_events and tg_resolve_events parameter files (several defunct parameters have been removed). HRC gratings datasets will also have additional columns including: AU|AV 1-3 and SUMAMPS.

Update for HRS-S+LETG to use automatic CALDB lookup to locate latest PI background filter file.

Update the recreate_tg_mask=yes option to use a clean event file (GTI, status, and grade filtered) with tgdetect2. The earlier behavior of using the Level 1 event file could lead to a poor zeroth order location.

Changes in the scripts 4.12.2 (April 2020) release

Synchronization with DS 10.8.3 updates

In the spring of 2020, a new aspect solution file is being created. This file has the DY, DZ, and DTHETA offsets folded back into the aspect solution itself (RA, Dec, Roll) and provides a better estimate of the off-axis angles. It can be identified by the file name (a single file with OBS_ID and the OBI_NUM) and by the CONTENT keyword set to "ASPSOLOBI". The event processing tools like acis_process_events and hrc_process_events handle these new aspect solution files transparently. Users will notice changes to their physical sky coordinates (x,y); however, the final celestial coordinates (RA,Dec) are the same to within a fraction of pixel.

While users should only have either the new CONTENT="ASPSOLOBI" or the CONTENT="ASPSOL" aspect solution files in their primary directory; it is possible that they will have both. chandra_repro now checks for this and will only use the CONTENT="ASPSOLOBI" file.

As a result of this change, the new skyfov method=convexhull algorithm can now be used. It provides a tighter bounding polygon around each chip which will provide a better estimate of the geometric area.

Changes in scripts 4.12.1 (December 2019) release

Updated to support new badpixels associated with bottom rows of each CCD due to 'shadow' cast by frame-store cover.

Change in scripts 4.11.5 (October 2019) release

Internal only changes to correct verbose handling in newer versions of Python.

Changes in scripts 4.11.4 (August 2019) release

Updated to support HRC observations with no Level 2 good time, ONTIME=0. Users get a recalibrated Level 1 event file when they set cleanup=no.

Changes in scripts 4.11.3 (May 2019) release

chandra_repro now creates individual TYPE:I source and background PHA files with the appropriate ANCRFILE, RESPFILE, BACKFILE, and BACKSCAL keywords set.

For HRC-S+LETG datasets, the script now creates the grating responses for orders -8, -7, -6, -5, -4, -3, -2, -1, +1, +2, +3, +4, +5, +6, +7, and +8. Previously only the -1 and +1 orders were calculated.

Changes in scripts 4.11.2 (April 2019) release

Changes to allow observations where the Level 2 ONTIME is 0 to be fully processed. Previously the script would error out when running the skyfov tool to create the FOV file. It will now create the file but without using a time filter.

Changes in scripts 4.11.1 (December 2018) release

For HRC, the script will now retain all the columns defined in the hrc_process_events stdlev1 event definition parameter. This includes the coarse coordinates in addition to the new SAMP column.

Change in scripts 4.9.4 (July 2017) release

Internal fix for uninitialized variable ("newasol") when processing an observation that does not have any aspect solution files. This is limited to observations of the Moon and the Earth. These observations still cannot be processed at this time but the error message is more informative.

Changes in scripts 4.9.2 (April 2017) release

These are only internal changes to the script to tweak how HISTORY records are written and to correct some verbose output.

Changes in scripts 4.8.4 (September 2016) release

For ACIS CC mode data, pix_adj="default" now sets acis_process_events pix_adj=NONE whereas before it would be set to "EDSER".

Changes in scripts 4.7.2 (April 2015) release

The following changes are included

• For consistency with the TGCat catalog, only the -1 and +1 order ARF and RMF are now created. Users can run mktgresp to generate the response files for higher orders.
• Errors out if the absolute path name for the input or output directories contains spaces.
• Added support for FAINT_BIAS mode datasets (only a few very early in the mission)
• Compares the files in the primary/ and secondary/ directory to the files listed in the event file header and issues a warning if they are different.
• Added a warning about using splitobs script for observations with multiple level 1 event files.
• ACIS TIME mode observations now have their GTIs aligned on exposure frame time boundaries using the new gti_align script.
• An additional time filter has been added that goes from the first aspect record time to the last. This is to address a rare condition where several of the first or last events in the level 2 event file are inside the GTIs but occur outside the times covered by the aspect solution -- which results in invalid sky coordinates during those times.

Changes in scripts 4.7.1 (December 2014) release

The default verbose level is now set to 1. Also there are internal changes to remove acis_process_events' calc_cc_times parameter.

Changes in scripts 4.6.6 (September 2014) release

The following changes were included

• The script will now create a FOV file - which ends in _repro_fov1.fits - along with the other output products. This may be especially useful if the input data have been reprojected to a different tangent point or filtered exclude a different time range than that used in the archival processing.
• Added STATFILT keyword to event that gives the 32bit status that was used to create the level 2 event file.

Changes in the scripts 4.6.1 (December 2013) release

The following changes were included

• The script now run r4_header_update to add the Repro-4 header keywords if needed.
• The centroid algorithm has been added as a pix_adj option.
• The script now runs tgdetect2 for grating data when recreate_tg_mask=yes

Change in the scripts 4.5.4 (August 2013) release

The following changes were included

• The pi=0:300 filter for HRC-S with no grating has been removed. Users will not see any difference unless they have processed their data with CALDB 4.5.7. The PI filter may optionally be used to reduce the background noise for some observations.
• Bug fix when exactly 2 event files were present in the secondary directory, but it was not an interleaved observation.
• Delay creation of output directory until the inputs have been verified.

Changes in the scripts 4.5.2 (April 2013) release

New recreate_tg_mask parameter

The recreate_tg_mask parameter is has been added to direct chandra_repro to use the existing REGION block attached the original evt2 file rather than re-run tgdetect2 and tg_create_mask. This is especially useful when the 0th order location was manually adjusted in standard data processing.

Grating response (ARF and RMF) files

chandra_repro will now create the appropriate ARF and RMF files for each order and grating arm that is in the _repro_pha2.fits file. It uses the default energy and channel grids. These products are stored in a "tg" subdirectory.

Interleaved mode

chandra_repro will now work on ACIS interleaved mode datasets without the user needing to split them into separate directories. The output products will have "e1" or "e2" in the file names that make the original primary and secondary exposures.

Skip directories with missing level 1 event files

If chandra_repro encounters a directory where it cannot find the level1 data products it requires it will now skip that directory rather than exiting.

Changes in the October 2012 Release

Support for multiple observations

The indir parameter can now take a stack of directories and the default outdir setting has changed to "".

Copying, rather than linking, files

The archive files - such as the aspect solution and mask files - are now copied rather than linked. This reverses the change made in the April 2012 release.

Changes in the June 2012 Release

• The default for the check_vf_pha parameter has been changed from "yes" to "no". It has been determined that this algorithm can remove good events for modestly bright point sources.
• In the event that a file systems does not support symbolic links (some external hard drives and thumb/USB drives), copies of the auxiliary files will be made.

Changes in the April 2012 Release

Warning messages

Warnings produced by acis_process_events and hrc_process_events are now written out instead of being hidden (this happens even when verbose=0). Please see the Caveats section of the acis_process_events and hrc_process_events bugs pages for information on whether these warnings can be safely ignored.

Links are now used when possible

The Archive data files necessary for data analysis are linked from the output directory instead of being copied.

Changes in the February 2012 Release

• Improved error checking on the input directory.

Changes in the December 2011 Release

• The tools acis_find_afterglow (new in CIAO 4.4) and acis_build_badpix are used in place of acis_run_hotpix because acis_find_afterglow is more efficient at identifying afterglows that have a small number (i.e. 4-10) of events.
• The status bits that are set by destreak or acis_process_events are reset with the acis_clear_status_bits script before the data are reprocessed. Previously, only the bits set by acis_detect_afterglow were reset.
• The destreak tool is run before acis_find_afterglow to improve the streak detection efficiency and to prevent pixels with several streak events from being misidentified as hot pixels.
• The default filename root is chosen from the first segment of the level=1 event file, e.g. changed from "acisf01838_000N002" to "acisf01838".

About Contributed Software

This script is not an official part of the CIAO release but is made available as "contributed" software via the CIAO scripts page. Please see this page for installation instructions.

Caveats

• Large Datasets: chandra_repro uncompresses files in memory before using them. This may consume too much memory on some machines and cause chandra_repro to fail. Since chandra_repro can work with files that have already been gunziped, the work around is to gunzip all the files before running this tool.
• Multi-obi datasets: as noted above, multi-obi obsids need to be separated before running chandra_repro, which can be performed by the splitobs script.

Bugs

Caveats

Problem with large files  (18 Sep 2012)

chandra_repro uncompresses files into memory to be written back out to disk. With large files, this can exhaust system resources and cause the script to fail with a non-informative error message

Unable to read evt1 file
# chandra_repro (09 April 2012): ERROR read error

Workaround:

chandra_repro will use the uncompressed version of the file so the easiest work around it to gunzip the files before running chandra_repro

unix% gunzip primary/*gz secondary/*gz

dsAPEPULSEHEIGHTERR -- WARNING: pulse height is less than split threshold when performing serial CTI adjustment.

The following warning may be displayed when running chandra_repro on an ACIS observation

# acis_process_events (CIAO): The following error occurred 2 times:
    dsAPEPULSEHEIGHTERR -- WARNING: pulse height is less than split threshold when performing serial CTI adjustment.

As is discussed in the acis_process_events bug page, this warning can safely be ignored as long as the number of times printed is small compared to the total number of events.