Last modified: November 2023

URL: https://cxc.cfa.harvard.edu/ciao/ahelp/specextract.html
Jump to: Description · Examples · Parameters · Creating RMFs: mkrmf vs. mkacisrmf · Weighted vs. Unweighted Responses · Corrected vs. Uncorrected Unweighted ARFs · Combining Output Spectra and Responses · Output Files · Grouping Source and Background Spectra · Fitting the Spectra · HRC Spectra · What tools are used? · Changes in the December 2010 Release · Changes in the February 2011 Release · Changes in the April 2011 Release · Changes in the July 2011 Release · Changes in the September 2011 Release · Changes in the October 2011 Release · Changes in the December 2011 Release · Changes in the February 2012 Release · Changes in the December 2012 Release · Changes in the July 2013 Release · Changes in the December 2013 Release · Changes in the June 2014 Release · Changes in the September 2014 Release · Changes in the November 2014 Release · Changes in the April 2015 Release · Changes in the April 2016 Release · Changes in the October 2016 Release · Changes in the December 2016 Release · Changes in the April 2017 Release · Changes in the April & May 2018 Release · Changes in the April 2020 Release · Changes in the December 2020 Release · Changes in the March 2021 Release · Changes in the May 2021 Release · Changes in the scripts 4.14.0 (December 2021) release · Changes in the scripts 4.14.2 (April 2022) release · Changes in the scripts 4.15.0 (December 2022) release · Changes in the scripts 4.15.1 (January 2023) release · Changes in the scripts 4.15.2 release · Changes in the scripts 4.15.3 release · Changes in the scripts 4.16.0 release · Bugs · See Also


AHELP for CIAO 4.17

specextract

Context: Tools::Composite

Synopsis

Create source and optional background spectra for Chandra imaging-mode and zeroth-order grating data.

Syntax

specextract  infile outroot [bkgfile] [asp] [dtffile] [mskfile]
[rmffile] [badpixfile] [dafile] [bkgresp] [weight] [weight_rmf]
[resp_pos] [refcoord] [correctpsf] [combine] [readout_streakspec]
[grouptype] [binspec] [bkg_grouptype] [bkg_binspec] [energy] [channel]
[energy_wmap] [binarfcorr] [binwmap] [binarfwmap] [parallel] [nproc]
[tmpdir] [clobber] [verbose]

Description

The specextract script creates source, and optional background, spectra for Chandra imaging-mode. The zeroth-order spectra from gratings data can be also extracted if the "[tg_m=0,Null]" filter is included with the input events file. Please see the Grating Spectroscopy threads for information on how to create spectra and responses for grating spectra.

The main decisions you have to make when running this script are:

The examples below highlight the different modes, and further details are provided below, after the parameters.


Examples

Example 1

unix% punlearn specextract
unix% specextract "acis_evt2.fits[sky=region(3c273.reg)]" extract/spec \
bkgfile="acis_evt2.fits[sky=region(3c273_bg.reg)]" \
grouptype=NONE binspec=NONE

Extract source and background spectra from the same event file using the region files "3c273.reg" and "3c273_bg.reg", respectively. The output files all begin with "spec" and are in the newly created sub-directory, "extract", named from the outroot parameter. Neither of the spectra will be grouped (binned), since grouptype is set to "NONE". The default value for the 'weight' parameter ("yes") is used for creating the weighted ARF, but the RMF will be unweighted, since the 'weight_rmf' parameter defaults to ("no"). Since 'weight=yes', the 'correctpsf' parameter is ignored, and since only one file was input to the 'infile' and 'bkgfile' parameters, the 'combine' parameter is ignored. The events file is used to locate all the necessary ancillary files (asol, msk, and badpix).

Example 2

unix% punlearn specextract
unix% specextract "acis_evt2.fits[sky=region(3c273.reg)]" 3c273 \
bkgfile="acis_evt2.fits[sky=region(3c273_bg.reg)]" asp=@asol1.lis \
mskfile=msk1.fits badpixfile=bpix1.fits \
grouptype=NONE binspec=NONE

This is a repeat of the previous example, except that the ancillary files are explicitly specified.

Example 3

unix% punlearn specextract
unix% specextract "acis_evt2.fits[sky=region(3c273.reg)]" 3c273 \
bkgfile="acis_evt2.fits[sky=region(3c273_bg.reg)]" asp=@asol1.lis \
mskfile=msk1.fits badpixfile=bpix1.fits weight=no \
refcoord="12:29:06.70 +02:03:08.60" \
grouptype=NONE binspec=NONE

This is a repeat of the previous example, except that the unweighted responses are created assuming the source center is at RA=12:29:06.70 Dec=+02:03:08.60.

Example 4

specextract "acis_evt2.fits[(x,y)=region(1447_src.reg)]" dafile=CALDB
outroot=acis_1447 weight="yes" binarfwmap=4 asp=asphist.fits
mskfile=msk1.fits bkgfile="acis_evt2.fits[(x,y)=region(1447_bkg.reg)]"
grouptype=NUM_CTS binspec=10 bkg_grouptype=NUM_CTS bkg_binspec=10

Similar to the first example, except the source and background spectra are each grouped such that there are 10 counts in a bin, and the 'asp' parameter points to an aspect histogram file (output of asphist) rather than an aspect solution. The dafile parameter is set to apply the dead area correction, and no bad pixel file has been set. The binarfwmap parameter sets the binning of the WMAP weight map (output by sky2tdet) used to create the weighted ARF; 1 is the normal recommended setting, but in this case a coarser binning is required since the very large source region would otherwise cause mkwarf to hit a memory limit.

Example 5

specextract
"acis_5027_evt2.fits[sky=region(5027_srcA.reg)],acis_5027_evt2.fits[sky=
region(5027_srcB.reg)]" dafile=NONE outroot=5027 bkgfile="" ptype=PI
grouptype=NONE binspec=NONE weight="no" correctpsf="yes" combine="yes"
asp=asphist.fits mskfile=""

Two event files are given as a stack, so two sets of output files are created based on the outroot value "5027": 5027_src1 and 5027_src2. No grouping is applied to the source spectra, and no background spectra are created. The 'weight' parameter is set to 'no' for creating the unweighted ARF and RMF files appropriate for point-source analysis. The output stacks of source spectra and unweighted source response files are combined, producing the additional output files "5027_combined_src.pi", "5027_combined_src.arf", and "5027_combined_src.rmf".

Example 6

specextract @input.lis @output.lis

The input event files and output filenames are all defined as stacks:

unix% cat input.lis
acisf00459N004_evt2.fits[sky=region(3c273.reg)]
5027_repro_evt2.fits[sky=region(5027_src.reg)]

unix% cat output.lis
459_3c273
5027_sl4

Default values are used for all the other parameters, which will attempt to pick up the ancillary files based on the input event file's header keywords.

Example 7

specextract \
infile="hrcf11240N002_evt2.fits[sky=circle(16246.516,16417.228,13)]" \
outroot=11240 mskfile=hrcf11240_000N002_msk1.fits \
badpixfile=hrcf11240_000N002_bpix1.fits \
dtffile=hrcf11240_000N002_dtf1.fits bkgresp=no \
correctpsf=no

Extract the HRC spectrum of the central source in the supernova remnant, explicitly specifying the mask, bad pixel, and dead time factor files. Unweighted responses are created for HRC data; in this case, aperture correction is not applied to the unweighted ARF.

Example 8

unix% punlearn specextract
unix% specextract
"acis_evt2.fits[sky=region(fov.fits)][sky=region(src.reg)]"
extract/spec_off-chip \
bkgfile="acis_evt2.fits[sky=region(bg.reg)]" \
grouptype=NONE binspec=NONE

The extraction region is intersected with the observation's FITS field-of-view file. This is useful when the extraction region includes parts of the sky that fall off the chip so that the appropriate BACKSCAL value can be calculated, excluding the portion of the extraction region that is not exposed to the detector.


Parameters

name type ftype def reqd stacks
infile file input   yes yes
outroot file     yes yes
bkgfile file input     yes
asp file input     yes
dtffile file input     yes
mskfile file input     yes
rmffile file input CALDB   no
badpixfile file input     yes
dafile file input CALDB   yes
bkgresp boolean   yes    
weight boolean   yes    
weight_rmf boolean   no    
resp_pos string   REGION    
refcoord string        
correctpsf boolean   no    
combine boolean   no    
readout_streakspec boolean   no    
grouptype string   NUM_CTS    
binspec string   15    
bkg_grouptype string   NONE    
bkg_binspec string        
energy string   0.3:11.0:0.01    
channel string   1:1024:1    
energy_wmap string   300:2000    
binarfcorr string   1    
binwmap string   tdet=8    
binarfwmap string   1    
parallel boolean   yes    
nproc integer   INDEF    
tmpdir string   ${ASCDS_WORK_PATH}    
clobber boolean   no    
verbose integer   0    

Detailed Parameter Descriptions

Parameter=infile (file required filetype=input stacks=yes)

The source event file(s)

The primary input to this tool is an event file (or stack of event files) with a filter defining the extraction region for the spectrum. See "ahelp dmfiltering" for information on defining region filters, and "ahelp stack" for details on using stacks of files.

If a stack of event files is entered, and 'asp' contains a stack of aspect solution files, then the lists of aspect solution and event files must be matched; this requires that all event files in the 'infile' list have a valid OBS_ID header keyword value.

Parameter=outroot (file required stacks=yes)

Root for the output filenames or a stack of filenames

If a string is given, it is used as a prefix in all filenames created by specextract. In this case, the tool autmotically creates output files designated as "src1", "src2", etc. The background files are named "bkg1", "bkg2", etc., to correspond to the source numbering.

Alternatively, a stack of explicit output filenames can be specified, one for each input file. See "ahelp stack" for details on using stacks of files.

Parameter=bkgfile (file filetype=input stacks=yes)

The background event file(s)

The background event file (or stack of event files) with a filter defining the extraction region for the spectrum. See "ahelp dmfiltering" for information on defining region filters, and "ahelp stack" for details on using stacks of files.

If background spectra are desired, the source and background stacks must contain the same number of elements. If background ARF and RMF files are also desired, the 'bkgresp' parameter should be set to "yes". It is allowable to leave the 'bkgfile' parameter value blank, which results in no background spectra or responses being created.

Parameter=asp (file filetype=input stacks=yes)

The aspect solution or aspect histogram file(s) for the input source observation(s)

One or more aspect solution files, or one aspect histogram file created with the asphist tool, for each input observation. If aspect solution files are entered, the asphist tool is invoked to generate the required aspect histogram input to sky2tdet when 'weight=yes' and mkarf when 'weight=no'. These files are used to determine which parts of the detector lie within the celestial source region during the observation.

If a stack of aspect solution files is entered, and 'infile' contains a stack of event files, then the lists of aspect solution and event files must be matched; this requires that all aspect solution files in the 'asp' list have a valid OBS_ID header keyword value.

Parameter=dtffile (file filetype=input stacks=yes)

The dead time factor file(s)

This parameter is only used with HRC observations and is ignored for ACIS data.

If the parameter is empty and HRC data is being used, then the DTFFILE keyword in the HRC events file(s) will be used. A parameter value of "NONE" will mean that only a mean DTF value will be used instead of the time-resolved DTFFILE.

The DTF file has a name like hrcf*dtf1.fits and is included in the output directory of the chandra_repro script.

Use the stack syntax - see 'ahelp stack' - to specify multiple files either as a comma (or space) separated list of file names or read from a file using the '@' syntax.

Unlike most of the other ancillary files, the order of the DTF files must match that of the event files given in the infile parameter.

Parameter=mskfile (file filetype=input stacks=yes)

The detector mask file(s) for the input source observation(s)

The secondary data product file msk1.fits for each input observation, for input to mkwarf when 'weight=yes' and mkarf when 'weight=no'.

Parameter=rmffile (file filetype=input default=CALDB stacks=no)

The CALDB directive used to determine the RMF calibration and choose the RMF generation tool

CALDB directive (calquiz calfile parameter) used to search for the P2_RESP calibration file(s) corresponding to the input file, the presence of which is used to determine which RMF tool to use. If a P2_RESP file is returned by the CALDB search, mkacisrmf is used; otherwise, mkrmf is used.

rmffile defaults to "CALDB", which will cause infile header keywords to be used to select the P2_RESP calibration file, along with the ccd_id value of the input source region (an rmffile value of "CALDB" gets converted internally to "CALDB(ccd_id=#)"). If the user wishes to override any keyword, CALDB parameters can be specified using the form:

CALDB(PARAMETER=VALUE;PARAMETER=VALUE...)

In this case, users are urged to include a "CCD_ID=VALUE" filter in their custom CALDB directive, as specextract will not automatically add one in this case. This is to ensure that mkrmf will always be chosen over mkacisrmf for an observation taken at the -110 C focal plane temperature, where the source region lies on a front-illuminated chip.

It is expected that rmffile is left at CALDB for HRC data.

Parameter=badpixfile (file filetype=input stacks=yes)

The bad pixel file(s) for the input source observation(s)

The appropriate bad pixel file to use for an input source observation, e.g. the primary/secondary data product bpix1.fits file. When this parameter is set, the ardlib parameter file is updated as necessary.

Parameter=dafile (file filetype=input default=CALDB stacks=yes)

ACIS "dead area" coefficients file, which may have the values "NONE" (no dead area computation), CALDB (for automatic lookup), or an explicit file reference to an ACIS "dead area" coefficients FITS table.

The ACIS dead area refers to a slight decrease in detector efficiency due to the background cosmic ray flux which temporarily renders some pixels useless. The calibration product is a coefficient (per CCD) which gives the fractional area lost (or "deadened") per second. Since the area lost increases with time, the magnitude of the effect depends upon the ACIS clocking parameters (number of rows, window location, frame-time) which determine how long a pixel was exposed to the background cosmic ray flux during the primary exposure and during electronic readout from the frame-store area. For full-frame timed-exposure, the dead area is about 4% at maximum CHIPY and about 2% at the readout. It is smaller for subarrays.

The dafile parameter is ignored for HRC data.

Parameter=bkgresp (boolean default=yes)

Determines whether or not to write background responses.

A Boolean switch to specify if background ARF and RMF response files should be written for each background event file input to 'bkgfile'. If set to "no", only background spectra will be output. If no background event files are entered, 'bkgresp' is ignored. The background ARF will be weighted while the background RMF will inherit the weighting picked by "weight_rmf".

Note that if blank sky files are used in the 'bkgfile' parameter, background responses cannot be created. Set "bkgresp=no", otherwise specextract will exit with an error.

Parameter=weight (boolean default=yes)

Determines whether to generate weighted or unweighted ARFs.

A Boolean switch to determine if the source ARF should be spatially weighted or not, as appropriate for extended versus point source analysis. Weighted ARFs are generated with mkwarf - where the input WMAP in TDET coordinates is created with sky2tdet - and unweighted ARFs are created by mkarf assuming all the flux is from a single celestial position. HRC observations will default to using unweighted ARFs. Background ARFs will always be weighted.

Parameter=weight_rmf (boolean default=no)

Determines whether to generate weighted or unweighted RMF files.

A Boolean switch to determine if the ACIS RMF responses should be weighted or not, if weight=yes, otherwise this parameter is ignored. Weighted or unweighted RMFs are created using either mkacisrmf or mkrmf, depending upon the existence of the P2_RESP calibration file in the CALDB for the corresponding observation. If mkacisrmf is chosen, the input WMAP (in TDET coordinates by default) is that which is created by dmextract in the spectral extraction step of the script, with the binning and energy specified in the 'binwmap' and 'energy_wmap' parameters.

Parameter=resp_pos (string default=REGION)

Method to refine the unweighted response position and to determine the coordinates to query ColDen for the Galactic neutral hydrogen column density. This parameter is only used if the 'refcoord' parameter is not set.

This parameter can be used to refine the detector location of where the unweighted responses are generated. The refined positions are based on images using a sky binsize of 2 pixels is used if 'weight=no' and 'correctpsf=no' or when 'weight=yes'. A binsize equal to 'binarfcorr' is used if 'weight=no' and 'correctpsf=yes'.

Parameter=refcoord (string default=)

Reference coordinates of unweighted response files.

This parameter defines the location in RA and Dec for which an unweighted ARF and RMF are produced.

The default setting ("") means that the value is calculated from the defined input file extraction region, using the position of the peak counts value within the region. When set, while the extracted spectrum will include counts from the defined region, responses will be created appropriate for a source center at the defined refcoord location. This approach is more robust when few or no source counts are within the region.

The refcoord parameter is set as a space- or comma-separated value taken to be the RA and Declination to use (ICRS) in decimal degrees or colon-separated sexagesimal formats. Examples include:

Parameter=correctpsf (boolean default=no)

Determines whether or not to apply a point-source aperture correction to unweighted ARFs.

A Boolean switch to specify whether or not to apply the arfcorr point-source aperture correction to the unweighted ARF generated by mkarf when 'weight=no', as well as to add an additional PSF_FRAC column to this file (see the arfcorr ahelp file for details). Aperture correction is only applied to source ARFs.

Parameter=combine (boolean default=no)

Determines whether or not to combine output spectra and responses.

A Boolean switch to specify whether or not to combine the spectra and ARF and RMF files output by specextract, in the event that stacks of observations were input to the script, and spectra and responses were successfully created for these observations. This is done by invoking the combine_spectra script when 'combine=yes', which will sum source spectra, sum area- and exposure-weighted background spectra, compute an exposure-weighted ARF, and an ARF-weighted RMF; see the ahelp for details. The output combined spectra and response filenames will include the specextract 'outroot' string plus the string "combined", e.g. "5027_combines_src.pi", "5027_combines_bkg.rmf", etc.

Parameter=readout_streakspec (boolean default=no)

If the source extraction region is for a readout streak, determine effective exposure time of the streak spectrum.

If the source extraction region is for the ACIS readout streak, if 'readout_streakspec=yes', the number of ACIS rows encompassed by the region will be estimated to determine the effective exposure time of the streak spectrum. This parameter requires specifying the corresponding point source location with the 'refcoord' parameter and 'weight=no' in order to generate the appropriate unweighted responses.

The effective exposure time for the readout streak is considerably shorter than the exposure time for data obtained when the detectors are not being read out. Therefore, the exposure time needs to be corrected. If the extraction region for the readout streak includes N_rows rows, then the effective exposure time is given by: Streak_Exposure = N_frames * N_rows * xfer_time, where N_frames = EXPOSURE / EXPTIME of the observation and the xfer_time is the ACIS frame transfer time of 40 [usec/row/frame].

For users with complicated composite polygonal regions, it may be worth verifying the determined number of rows and effective exposure time are reasonable.

Parameter=grouptype (string default=NUM_CTS)

Source spectrum grouping type

The allowed values the same as those in dmgroup: NONE, BIN, SNR, NUM_BINS, NUM_CTS, ADAPTIVE, ADAPTIVE_SNR, BIN_WIDTH, MIN_SLOPE, MAX_SLOPE, and BIN_FILE.

More information on grouping is available from "ahelp dmgroup".

Parameter=binspec (string default=15)

Source spectrum grouping specification

A numerical value used for the grouping method. The format of the grouping specification depends on what is chosen for the "grouptype" parameter. Here are some example values for each option:

More information on grouping is available from "ahelp dmgroup".

Parameter=bkg_grouptype (string default=NONE)

Background spectrum grouping type

The allowed values are the same as those in dmgroup: NONE, BIN, SNR, NUM_BINS, NUM_CTS, ADAPTIVE, ADAPTIVE_SNR, BIN_WIDTH, MIN_SLOPE, MAX_SLOPE, and BIN_FILE; see the "grouptype" parameter description for details. More information on grouping is available from "ahelp dmgroup".

Parameter=bkg_binspec (string)

Background spectrum grouping specification

A numerical value used for the background grouping method. The format of the grouping specification depends on what is chosen for the "bkg_grouptype" parameter; see the "binspec" parameter description for details. More information on grouping is available from "ahelp dmgroup".

Parameter=energy (string default=0.3:11.0:0.01)

Energy grid in keV

The grid is specified by giving the lower bound, upper bound, and binning step, separated by the ":" character. For example, the default value "energy=0.3:11.0:0.01" bins from 0.3 to 11.0 keV in steps of 0.01 keV.

The ACIS detector is calibrated over the range 0.224004 - 12 keV; choosing values outside this range may result in errors from the tool. The default value of this parameter is suitable for most analyses.

Parameter=channel (string default=1:1024:1)

RMF binning specification in pixels

The grid is specified by giving the minimum, maximum channel, and binning step, separated by the ":" character. For example, "channel=1:1024:1" bins from channel 1 to 1024 in steps of 1. The default value is suitable for most analyses of PI spectra. If a value other than the default is specified, the channel filter will also be applied to the PI spectrum.

Parameter=energy_wmap (string default=300:2000)

Energy range for WMAPs

The WMAP (weight map) file is an image of the extraction region. Specifically, the PHA FITS file has in its primary header an image containing a low resolution map of the source in detector coordinates. This allows downstream software to determine the appropriate weighting for calibrations which depend on detector position (for instance, effective areas may depend on the off-axis angle).

An energy filter may be included when creating a WMAP to better represent the event distribution in a particular energy range, e.g. the default value of 300:2000 (the units are eV).

The energy_wmap parameter is used in conjunction with the "binwmap" value for creating the low-resolution WMAP in DET coordinates with dmextract (for input to mkacisrmf), as well as the high-resolution WMAP in TDET coordinates with sky2tdet (for input to mkwarf for the 'weight=yes' setting).

Parameter=binarfcorr (string default=1)

Detector pixel bin size used for arfcorr PSF correction.

Detector pixel binnning factor to determine the size and scale of PSF to derive aperture corrections by arfcorr. The binsize should only be reduced when working with very small source regions (sub-pixel analysis) and must have a value greater than zero.

Parameter=binwmap (string default=tdet=8)

Binning factor for RMF WMAPs.

The binwmap parameter allows the user to define the binning of the WMAP generated by dmextract in the spectral extraction step of the script, for input to mkacisrmf if this tool is chosen over mkrmf for creating the RMF; see the "energy_wmap" parameter for a description of what a WMAP is. We recommend 'binwmap="tdet=8"' (the default) to create a map in TDETX,TDETY coordinates binned by a factor 8. (Note that this parameter does not influence the binning of the WMAP in TDETX,TDETY coordinates generated by sky2tdet for input to mkwarf, used for the 'weight=yes' setting.) If a TDET column is not found in an input source or background event file, 'binwmap="det=8"' is used.

This parameter is used in conjunction with the "energy_wmap" value.

Parameter=binarfwmap (string default=1)

Binning factor for ARF WMAPs.

The binarfwmap parameter allows the user to specify the binning of the WMAP generated by sky2tdet for input to mkwarf when weighted responses are created ('weight=yes'). Setting this parameter to a value greater than 1 can significantly reduce the time required to create the weighted ARF in the event that the source region is suitably large, however reducing the the WMAP resolution in this way may introduce aliasing effects in the CHIP->TDET coordinate transform performed by sky2tdet. It is for this reason that increasing the binning of the WMAP is recommended only for the cases where mkwarf cannot process the unbinned (bin=1) input file (mkwarf may hit a memory limit when trying to process a large file).

Parameter=parallel (boolean default=yes)

Run processes in parallel?

When run on a multi-processor system, many of the tasks can be run in parallel to reduce the execution time of the script. Since the tasks are likely to be memory or I/O-bound, the reduction in run time will be less than the number of cores on a machine. When parallel=yes, the default behaviour is to use all the CPU processors, but this can be changed with the nproc parameter.

This option can be ignored when run on a single-processor system.

Parameter=nproc (integer default=INDEF)

Number of processors to use

This parameter is only used when parallel=yes. It determines the number of processors to use. If maxproc is the actual number of processors on your machine, then a value of INDEF - the default value - means that all maxproc processors will be used. A positive value means to use that number of processors (any value larger than maxproc will be set to maxproc). A negative value is added to maxproc (and any value less than one is set to one).

Parameter=tmpdir (string default=${ASCDS_WORK_PATH})

Directory for temporary files.

Directory for storing temporary files that require further processing before becoming useful. If the directory does not exist then it will be created for use by the script, and then deleted.

Parameter=clobber (boolean default=no)

Specifies if an existing output file should be overwritten.

Parameter=verbose (integer default=0)

Specifies the level of verbosity (0-5) in displaying diagnostic messages.


Creating RMFs: mkrmf vs. mkacisrmf

The tool mkacisrmf represents a new method for creating ACIS response matrices; details on why mkacisrmf is more accurate than mkrmf are available in the the mkacisrmf help file. The Creating ACIS RMFs with mkacisrmf why topic contains details about the tool and its calibration.

specextract uses the input event file to query the CALDB for a P2_RESP file, the calibration used by mkacisrmf. If a calibration file exists, specextract runs mkacisrmf to create the RMF. This improvement means that specextract will choose mkacisrmf for all the valid observation cases. If a calibration file for the data is not available, specextract creates the RMF(s) with mkrmf.

Weighted vs. Unweighted Responses

The specextract 'weight' and 'weight_rmf' Boolean parameter allows the user to choose between generating weighted versus unweighted ARF and RMF files. If 'weight=yes' (default), specextract will create a weighted ARF by running mkwarf, where the input detector WMAP to mkwarf is created with the sky2tdet tool in TDET coordinates, with the user-specified binning ('binarfwmap' parameter). A weighted RMF is generated via mkrmf or mkacisrmf - only if 'weight=yes' and 'weight_rmf=yes' - depending on the existence of the P2_RESP calibration file for the observation in the CALDB (refer to the "Creating RMFs: mkrmf vs. mkacisrmf" section above for details). If mkacisrmf is run, the input WMAP is not the same WMAP in TDET coordinates input to mkwarf, but is that which is created by dmextract in the spectral extraction step of the script, with the binning and energy specified in the specextact 'binwmap' and 'energy_wmap' parameters. It is required that one detector mask file per input source observation be input to the specextract 'mskfile' parameter for both 'weight=no' and 'weight=yes' options.

If 'weight=no', mkarf is used to create an unweighted ARF (extension .arf), which is considered appropriate for point-source analysis. In this case, an unweighted RMF (extension .rmf) is generated via mkrmf or mkacisrmf , depending on the existence of the P2_RESP calibration file for the observation in the CALDB (refer to the "Creating RMFs: mkrmf vs. mkacisrmf" section above for details). HRC data will only use unweighted responses.

It is required that one or multiple aspect solution files, or one aspect histogram file (output of asphist) per input source observation, be input to the specextract 'asp' parameter for both the 'weight=yes' and 'weight=no' options. The is because an aspect histogram file is required input to the sky2tdet tool invoked for 'weight=yes', and for mkarf for the 'weight=no' option. The 'asp' parameter accepts either a list of aspect solution files (one or more per source observation), or a list of aspect histogram files (only one per source observation), not a mix of both types. If aspect solution files are entered, the script invokes the asphist tool for generating the required aspect histogram file input.

When both the 'infile' and 'asp' parameters contain a list of files - source observation files and aspect solution files, respectively - it is required that all files in each list have a non-NULL OBS_ID header keyword value to ensure fail-safe matching of aspect solution files to source files. To work around the OBS_ID header key requirement, aspect histogram files may be entered into 'asp' instead of aspect solution files.

Corrected vs. Uncorrected Unweighted ARFs

For the case of 'weight=no', the user has the option to apply a point-source aperture correction to the SPECRESP column contained in the ARF generated by mkarf, via the specextract 'correctpsf' Boolean parameter. If 'correctpsf=yes' (and 'weight=no'), specextract will invoke the arfcorr tool to apply this correction, as well as to add an additional PSF_FRAC column to the ARF output by mkarf; see the arfcorr ahelp file for details. The output corrected ARF will have extension '.corr.arf'. If 'correctpsf=no' (and 'weight=no'), the ARF output by mkarf will be unaltered, with extension '.arf'.

All source regions input to the specextract 'infile' parameter are converted to physical coordinates via dmmakereg for point-source analysis with the ARF correction ('weight=no' and 'correctpsf=yes'), as arfcorr requires input regions to be in physical coordinates. (Background ARFs do not get corrected since they are extended.)

Combining Output Spectra and Responses

The specextract 'combine' Boolean parameter gives the user the option to combine a stack of source and background spectra generated by specextract - in the event that stacks of soure/background observations were input to the tool and spectra were successfully created - as well as associated ARFs and RMFs output by specextract. This is done by invoking the combine_spectra script when 'combine=yes', which will sum source spectra, sum area- and exposure-weighted background spectra, compute an exposure-weighted ARF, and an ARF-weighted RMF; see the ahelp for details. The output combined spectra and response filenames will include the specextract 'outroot' string plus the string "combined", e.g. "5027_combined_src.pi", "5027_combined_bkg.rmf", etc.

Output Files

The number of files created depends on if a background event file was provided, if source and/or background grouping was specified, and the setting for the 'correctpsf' and 'combine' parameters. In the most general case (source and background both provided and grouping given for both, 'weight=yes', 'combine=no'), the output files will be:

Grouping Source and Background Spectra

The user can choose to group the output source spectrum; all the grouptype options available with dmgroup are allowed. However, the dmgroup parameters "xcolumn" and "ycolumn" are hard-coded to "channel" and "counts" respectively, as appropriate for standard PHA files. If grouping is chosen, both the grouped and ungrouped source and background spectra files are written out; the grouped are designated by the "_grp.pi" ending in the filename. The RESPFILE and ANCRFILE keywords in the header of the grouped source and background spectrum will be updated, as will the BACKFILE header keyword in the source spectra only. Note that output grouped spectra and responses will not be combined if the specextract 'combine' parameter is set to 'yes'; only ungrouped files may be combined via the combine_spectra script invoked by specextract in this case.

Fitting the Spectra

As mentioned, the BACKFILE, ANCRFILE, and RESPFILE header keywords in the source and background spectra are updated appropriately by specextract. This means that the spectra can then be read into Sherpa and all the supporting files will automatically be read as well; the background (if available) will be defined, as will the source and background response files. Please see the Sherpa threads for more information on using Sherpa to fit spectral data.

HRC Spectra

The High Resolution Camera (HRC) has extremely limited energy resolution, and only when coupled with a transmission grating is it possible to get meaningful energy and order-sorting information, that can disentangle the source from background events.

While the intrinsic energy resolution of the HRC micro-channel plates is poor, particularly when compared with the ACIS CCDs, there is some ability to distinguish hard and soft spectra. The HRC PI spectra is extracted from the observations using PHA values derived from the appropriate calibration gain maps, and are to be used in conjunction with the appropriate RMFs constructed for HRC-I and HRC-S, available in the CalDB.

With that said, the RMF should not be used in spectral fits. The response is not sufficiently constrained to provide good fits with reasonable errors since specific energy ranges cannot be assigned to PI ranges. The mapping between PI bins and energy values is poorly defined and non-monotonic; therefore, when using the HRC RMFs with the corresponding PI spectra, analysis should be performed in channel-space, and not energy-space.

However, if the spectral shape of the source is known, the ARF and RMF created by specextract may be used to convert the PI spectrum to a broad band flux.

What tools are used?

The script combines the dmextract, mkwarf OR mkarf, mkrmf OR mkacisrmf, dmgroup, dmhedit, dmmakereg, sky2tdet, and arfcorr tools, as well as the acis_set_ardlib script; see the individual help files for information on each of these, e.g. "ahelp dmextract".

Changes in the December 2010 Release

The specextract script in CIAO 4.3 is an enhanced version of the specextract available in CIAO 4.2. Enhancements include:

Additionally, weighted ARFs generated by mkwarf are more accurate as a result of running this tool in conjunction with the new sky2tdet tool; this required adding the new parameter 'mskfile' to specextract.

Finally, both the 'weight=yes' and 'weight=no' options require the user to supply an aspect histogram file, therefore the 'asp' parameter was also added to specextract.

Changes in the February 2011 Release

The 'mskfile' input is now applied to point-source ARFs (previously, the mkarf 'mskfile' value was hardcoded as "NONE" as in psextract).

A new 'badpixfile' parameter is available for setting a different bad pixel file in ARDLIB for each input source observation.

Both aspect solution files (pcad*asol1.fits) and aspect histogram files (output of asphist) are supported in the 'asp' parameter (one type or the other, not a mix).

Updated to work around a bug in the arfcorr tool (which is invoked when 'weight=no' and 'correct=yes' for point-source analysis with ARF correction): user-input source regions are converted to physical coordinates before being input to arfcorr.

If the ObsIDs of input background observations are mismatched to input source observations, background input will be ignored. Users should be sure to add the appropriate OBS_ID header key to any input observations which may be missing one, e.g., a blank-sky background file input to the 'bkgfile' parameter.

Entered source or background files missing the CTI_APP header keyword, required by many CIAO tools, will be updated according to the following rules, to avoid an acis_fef_lookup error:

The default "det=8" value for the 'binwmap' parameter has been changed to "tdet=8".

The script no longer generates temporary output files.

Changes in the April 2011 Release

There is a new parameter, 'bkgresp', to determine whether a background ARF and RMF should be created. Set to "no" to create just a background spectrum (e.g. if the background will be subtracted in analysis).

The user is now prompted for 'bkgfile' parameter input (was hidden previously).

Input source regions are converted to physical coordinates before input to arfcorr when "correct=yes" and "weight=no".

An ObsID header keyword is no longer required in background file input; this resolves problems when using the blank-sky background files, for instance. (An ObsID header keyword is required in the source and aspect solution file input only when stacks of each are entered and must be matched.)

If the input file does not have a TDET column, the ARF file is created with binwmap="det=8" instead of "tdet=8".

The script prints the version number when verbosity is greater than 0.

For 'weight=no' analysis with zeroth-order grating data as input, the mkarf 'grating' parameter is now set to the value of the GRATING header keyword in the input file ('NONE' if null or nonexistent).

It is no longer required that source regions be input in file format for the 'weight=no'/'correct=yes' branch of analysis.

Changes in the July 2011 Release

A "CCD_ID=VALUE" filter is now automatically added to the CALDB directive used to locate calibration files for determining which RMF tool to use, when the rmffile parameter is left at its default value "CALDB". (If the user has entered a custom directive of the form "CALDB(PARAMETER=VALUE;PARAMETER=VALUE...)", they are urged to include a "CCD_ID=VALUE" filter, as specextract will not automatically add one in this case. This is to ensure that mkrmf will always be chosen over mkacisrmf for an observation taken at the -110 C focal plane temperature, where the source region lies on a front-illuminated chip.)

The script no longer fails when using large stacks for both the bkgfile and asp parameters. This was due to a bug in an underlying module; a different routine is now used to avoid that bug.

When specextract is used to extract sources for tens of sources, the combining spectra step (combine=yes) no longer fails due to a string length problem. An update to an underlying module resolved the issue.

Several functions were enhanced for speed and other improvements when large files stacks are entered.

Changes in the September 2011 Release

When background spectra are extracted and combine=yes, the background spectra will be summed, even if bkgresp=no. The previous version of the script required background responses when combine=yes.

All input files are now checked for readability before an attempt is made to use them in analysis, to avoid long or confusing error messages which resulted in some cases.

Changes in the October 2011 Release

When a compressed (gzipped) source event file was input to the script without the '.gz' extension (something which most CIAO tools can handle), the arfcorr tool (invoked when weight=no/correct=yes) would fail on account of not being able to locate the file. The script now adds the extension to the input file at the arfcorr step, avoiding this error.

If a source region is entered in a format which is not supported by CIAO, the script will exit since the tools it invokes will determine that the region contains zero counts. In this case, the script now catches the error and exits nicely.

Changes in the December 2011 Release

There is a new 'binarfwmap' parameter tied to the new 'bin' parameter of the sky2tdet tool, to allow users to change the binning of the WMAP used to create weighted ARFs (in previous releases, the binning was hardcoded to a value of 1). This parameter is helpful to users who run up against mkwarf's memory limit when they input large source regions to specextract with 'weight=yes'; setting the ARF WMAP binning to a value greater than 1 works around this issue.

File input parameters were updated to support the following null values, in addition to "": "NONE", "none", "None".

Changes in the February 2012 Release

A bug was fixed which caused the script to exit in error when the 'outroot' parameter contained a stack, and 'weight=no' and 'correct=yes'.

Changes in the December 2012 Release

The script now fails when input data lacks the CTI_APP header keyword required by many CIAO tools (previously, it ran the now-deprecated check_ctiapp.sh script to add the missing keyword).

Updated to support stk module changes.

Changes in the July 2013 Release

The script will look for ancillary files if they are not explicitly given based on the header of the input events files. The "correct" parameter has been renamed "correctpsf".

Additionally, support for spectral extraction in PHA-space is dropped.

Changes in the December 2013 Release

The script introduces three new parameters: dtffile, weight_rmf, and refcoord.

HRC support added. A PI spectrum will be extracted and an unweighted ARF is created, along with the appropriately selected RMF from the CalDB. The dead time factor file, accounted for in the 'dtffile' parameter, is needed for each observation in order to produce appropriate aspect histograms used to create ARFs.

The 'weight' parameter now controls whether or not only ARFs are weighted. Only when "weight=yes" will the 'weight_rmf' parameter be used. By default, "weight_rmf=no" since the calculations needed to produce a weighted RMF is computationally expensive - especially for very large regions - generally for a few percent improvement over an unweighted RMF.

The 'refcoord' parameter is introduced to produce unweighted responses at a specific RA and Dec, instead of necessarily requiring the responses be produced coincident with the spectral extraction region, which is the default script behavior. The position should be provided as a space- or comma-separated string, "RA Dec", in decimal degrees or ICRS sexagesimal, colon-separated format.

We no longer distinguish between unweighted and weighted responses in the file extension (.arf vs .warf and .rmf vs .wrmf). Only the .arf and .rmf extensions are now used for response files.

If a blank sky map, either from the CalDB or M. Markevitch's site, is used as a background file, background responses cannot be produced for these files (since they are aggregates of many observations). Please set "bkgresp=no" in this case, otherwise specextract will exit with an error.

Changes in the June 2014 Release

Handling of observations with multiple aspect solution files has been improved and issues handling outroots with nested directory paths has been fixed.

Galactic neutral hydrogen column densities derived from the velocity-resolved Bell Labs survey (ApJ Suppl. 79, p77, 1992) and the Dickey and Lockman NRAO all-sky interpolation of the Bell Labs survey (ARA&A, 28, p.215) are added to the spectrum files with the header keywords "Bell_nH" and "NRAO_nH", respectively, for the source extraction position derived from COLDEN.

Changes in the September 2014 Release

A new "tmpdir" parameter has been added to allow users to choose specific directory to write temporary files and handling of bad pixel files has been re-worked to avoid corrupting the ardlib.

Changes in the November 2014 Release

A bug was fixed causing failures when a weighted ARF and an unweighted RMF are created for -110C ACIS data.

Changes in the April 2015 Release

Handling of non-bad pixel file ardlib parameters enhanced and COLDEN failures fixed. The script will error out with warning if the absolute input and output paths contain any whitespace, that can lead to OS-related failures.

Changes in the April 2016 Release

Check added to see if the input file is a blanksky background file and will error out due to the inability of creating response files for the extracted spectra.

A bug was fixed where the specextract history was being attempted to be written to the FEF in the CALDB when mkrmf is used.

Changes in the October 2016 Release

The "binarfcorr" parameter added to set the detector pixel size used for arfcorr PSF correction.

Additional checks and warning messages added for warm observations with focal plane temperature >-110C. This includes a change for creating unweighted RMFs to ensure that any observation that has a P2_resp calibration file available for the chip will use mkacisrmf (and run to completion) by changing mkacisrmf's "infile" from "CALDB" to "CALDB(CCD_ID=x)" and explicitly setting the detector the extraction region is on.

Changes in the December 2016 Release

Behavior changed when "bkgresp=yes" where the calculated background ARF will always be weighted. The background RMF creation is determined by the "unweight_rmf" parameter.

Changes in the April 2017 Release

A bug was fixed that caused the script to fail when trying to combine spectra when "bkgresp=no" and exactly *3* sets of spectra were being co-added. The data products created by specextract were otherwise correct. Focal plane temperature testing of HRC observations is now skipped; the script would error out since the FP_TEMP header keyword does not exist in HRC data.

Changes in the April & May 2018 Release

PI filtering of spectrum added if the 'channel' parameter is not the default instead of the filter being applied to just the RMF. The 'rmffile' parameter was enabled for use with HRC observations by the request of the HRC Calibration Group; typical users will use the default setting of 'rmffile=CALDB'.

Changes in the April 2020 Release

The script has been updated to query the CalDB for the appropriate HRC RMFs provided by the calibration team, instead of using hard coded paths.

Changes in the December 2020 Release

This update is internal clean up, re-working logic needed to eliminate many of the IDLisms in the script. No changes in file outputs, although some error message formatting has changed.

Changes in the March 2021 Release

The frame-store shadow is now included when calculating the ARF for ACIS observations. This means that a small number of rows of the CCDs are now excluded. Fixed bug handling blanksky background files. For background spectra extacted from blanksky files, the AREASCAL keyword is set to the BKGSCALn scaling factor calculated by the 'blanksky' script to appropriately scale the background during analysis; if the spectra is being used for background subtraction, set AREASCAL to 1.

Changes in the May 2021 Release

Quashed erroneous warning messages being generated when handling blanksky background files and fixed bug causing 'combine=yes' to exit and throw an error before 'combine_spectra' is run.

Changes in the scripts 4.14.0 (December 2021) release

specextract has been re-written, but usage and file outputs are unchanged. Parallelization has been applied where possible, introducing the "parallel" and "nproc" parameters controlling whether parallelization and the number or processers available for use, respectively; performance improvement is most noticeable when both source and background [weighted] responses are generated. Enhanced error checking and more informative error messages are provided.

Changes in the scripts 4.14.2 (April 2022) release

The "readout_streakspec" parameter has been added to support spectral extraction from the ACIS readout streak and determine the effective exposure time for the streak spectrum.

Changes in the scripts 4.15.0 (December 2022) release

New parameter

The "resp_pos" parameter has been added to provide the user several options on determining the detector position where unweighted responses are generated. Prior versions of 'specextract' would default to the maximum location encompassed by the extraction region on an image binned by a factor of 2 (to avoid duplicate pixel values). The new default is to use the center defined by the extraction region with 'resp_pos=REGION'.

Parallel processing

The way that the tools are run in parallel has been reworked to improve the behavior.

Screen output

The screen output - when used in an interactive terminal - has been updated to better handle the case where multiple files are being processed.

Name of the combined files

A fix for an odd bug with the name of the combined files not matching the outroot parameter has been fixed.

Changes in the scripts 4.15.1 (January 2023) release

When using a pixel mask to filter the event file - that is, include a filter of "sky=mask(...)" in the event file - then the resp_pos setting can only be set to CENTROID or MAX if the refcoord parameter is not set, otherwise the script will error out.

A number of bugs added in 4.15.0 have been fixed: the script history is now written out correctly when run interactively, the presence of the OBS_ID keyword in blanksky or stowed background files is no-longer a problem, and the script no-longer creates a HRC background response when 'bkgresp=no' is used.

Changes in the scripts 4.15.2 release

Improved UX behavior with cleaner erroring out when invalid parameters are used on the command-line and warning messages when multiple shapes defined in a background region file. Internal infrastructure changes making use of ciao_contrib.runtool.make_tool.

Changes in the scripts 4.15.3 release

Applicable to only background spectra extracted from event files generated by the 'blanksky' script, this update fixes an AREASCAL issue introduced in the scripts 4.13.1/March 2021 Release.

This changes the AREASCAL value to be 1/BKGSCALn rather than BKGSCALn since the background counts spectrum should be scaled as:

BKGSCALn * (src_exposure/bkg_exposure) * (src_backscal/bkg_backscal)

but spectral analysis software typically will scale the background as:

(src_exposure/bkg_exposure) * (src_backscal/bkg_backscal) * (src_areascal/bkg_areascal)

Chandra data nominally has src_areascal = 1.0 so that setting bkg_areascal = 1/BKGSCALn will provide the correct background scaling for model fitting and background subtraction. This problem can be easily overlooked during analysis since it can have a very small impact, but for BKGSCALn << 1, it can potentially over estimate the blanksky/stowed background level to be several times higher than the observed background level.

Changes in the scripts 4.16.0 release

Support for event files generated by MARX added and handling of ancillary file parameters set to 'none' has been improved, in particular when 'badpixfile=none' which could possibly revert to using the default CalDB bad pixel file.


Bugs

Output spectra and responses were not combined because spectra and/or responses were not created for every item in the input stack(s) of files.

The above warning message may occur even if the spectrum, RMF, and ARF were successfully created for each of the individual observations.

Workaround:

As a work around, users just need to run the combine_spectra script with the list of spectra, .pi files, as input

$ combine_spectra "spectract_output/*pi" outroot=combo 

Caveats

Using correctpsf=yes with small region may yield unreliable results.

The PSF aperture correction users get when using correctpsf=yes with weight=no may be unreliable when using very small regions, where small is relative to the size of the PSF. For example using a circle with a 0.5arcsec radius, ie 1 ACIS pixel, for an on-axis source is likely going to incorrectly compute the PSF fraction correction.

Workaround:

Users can work around this issue by running the arfcorr tool separately after running specextract. The key is to use an input image with a small bin size compared to the region. For example

% pset arfcorr infile="acis_evt2.fits[sky=region(small.reg)][bin sky=0.1]"
% pset arfcorr arf=small_reg.arf
% pset arfcorr outfile=small_reg_psfcorrected.arf
% pset arfcorr x=4171.265139 y=3985.193939
% pset arfcorr region="region(small.reg)"
% arfcorr mode=h
The same asp and mask files are used for source and background responses  (28 Dec 2010)

The script assumes that corresponding items in the source and background stack are defined from the same event file, and uses the specified asp and mskfile files for creating both the source and background responses.

For example, if the input command includes:

specextract infile="file1.fits[sky=region(i.reg)],file2.fits[sky=region(i.reg)]" \
bkgfile="file1.fits[sky=region(bg.reg)],file2.fits[sky=region(bg.reg)]" \
asp="asphist1.fits,asphist2.fits" outroot="src_i" 

The "asphist1.fits" file is used in creating both the source and background WMAP files:

sky2tdet
"file1.fits[sky=region(i.reg)][energy=300:2000][bin
sky=1]" asphist1.fits "src1_tdet.fits[wmap]" clobber=yes

sky2tdet
"file1.fits[sky=region(bg.reg)][energy=300:2000][bin
sky=1]" asphist1.fits "bkg1_tdet.fits[wmap]" clobber=yes

Users who wish to define the source and background from different event files will need to run the script twice to do so.

Remember that if you plan on subtracting the background, then you do not need to create a background RMF and ARF. You may simply use dmextract to create the spectrum. In this case, leave the bkgfile parameter blank so that specextract will only create the source spectrum and responses.

The mkwarf tool may run out of memory for large regions.  (15 Dec 2011)
# mkwarf (CIAO): dsALLOCERR -- ERROR: Could not allocate memory.

If a weighted ARF is being created for a large region, on order of a full chip (or larger), mkwarf may hit the intrinsic memory limit of a 32-bit application.

Workaround:

In CIAO 4.4, there is a new specextract parameter - binarfwmap - that specifies the binning of the sky2tdet WMAP, which is used as input to mkwarf.

If you encounter this error, increase the binarfwmap value to create a coarser WMAP.

ERROR: no non-null/0/nan pixels are in the input image  (02 Mar 2012)
# specextract (<date>): ERROR An error occurred while running 'sky2tdet':
# sky2tdet (CIAO): ERROR: no non-null/0/nan pixels are in the input image

The most likely cause of this error is that the region and energy filter set in the energy_wmap parameter do not select any events:

unix% dmlist "evt2.fits[sky=region(ciao.reg)][energy=300:2000]" counts
0 

Workaround:

Adjust the energy_wmap value to match that of your events. The range of event energies in the region can be found with dmstat:

unix% dmstat "evt2.fits[sky=region(ciao.reg)][cols energy]"
energy[eV]
min: 2231.7507324 @: 61
max: 7980.6259766 @: 35
mean: 4012.0070413
sigma: 1252.6216735
sum: 429284.75342
good: 107
null: 0

See Also

calibration
ardlib
psf
psf
tools::aspect
asphist, dither_region
tools::background
acis_bkgrnd_lookup, hrc_bkgrnd_lookup, readout_bkg
tools::composite
combine_grating_spectra, combine_spectra
tools::coordinates
sky2tdet
tools::core
dmextract
tools::response
acis_fef_lookup, acis_set_ardlib, addresp, dmarfadd, eff2evt, find_mono_energy, fullgarf, make_instmap_weights, mean_energy_map, mkacisrmf, mkarf, mkexpmap, mkgarf, mkgrmf, mkinstmap, mkosip, mkpsfmap, mkrmf, mkrprm, mkwarf, psf_project_ray, rmfimg
tools::statistics
aprates