CIAO 4.12.1 Release Notes

---

CIAO 4.12.1 is distributed for the following platforms:

• Linux 64 bit
• Apple macOS 10.12 (Sierra) through macOS 10.15 (Catalina)

CIAO is now available via the conda package manager: conda builds are available for Linux and Mac for Python 3.7, 3.6, and 3.5. More information can be found in the conda section of the Installation notes below.

There is no support for 32 bit operating systems, older Linux (CentOS 5 era) or older macOS platforms (OS-X El Capitan and earlier). More details can be found on the Platform Support page.

---

• Notable changes and improvements in CIAO 4.12.1
• Notable changes and improvements in CIAO 4.12
• How CALDB 4.9.3 Affects Your Analysis
• How CALDB 4.9.2.1 Affects Your Analysis
• How CALDB 4.9.1 Affects Your Analysis
• How CALDB 4.9.0 Affects Your Analysis
• Installation
• Tools
• Parameter Files
• ChIPS
• Sherpa
• Graphical User Interfaces
• Analysis Scripts
• Python Modules
• Libraries
• Environment
• Documentation

---

Notable changes and improvements in CIAO 4.12.1

---

Notable changes and improvements in CIAO 4.12

---

How CALDB 4.9.3 Affects Your Analysis

ACIS Imaging and Grating Data

• Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

  The new, combined time-dependent ACIS Gain (T_GAIN) file for May-July 2020 (Epoch 82) and February-April 2020 (Epoch 81) are introduced. The combined Epochs 79+80 file has also been updated since CALDB 4.9.2.1. Therefore, the new T_GAIN affects observations taken since 2019 May 02. Other observations are unaffected by these new calibration files.

  Users working with ACIS data taken since 2019 Mayy 02 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

  See the time-dependent ACIS T_GAIN why page for more information.

  The gain corrections are at the nominal and expected levels of less than 1% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

  Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

---

How CALDB 4.9.2.1 Affects Your Analysis

ACIS Imaging and Grating Data

• Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

  This update affects all ACIS observations made at the -120 C focal plane temperature, covering Epochs 1-80 (29 Jan 2000-31 Jan 2020). The chandra_repro script and reprocess the data thread to improve the T_GAIN calibration may be used to apply the new calibration results. The DATE-OBS header keyword records the observation start date.

  The time-dependent ACIS gain files applies a new algorithm to improve statistical averaging for data with decreasing counting statistics, first introduced in the ACIS T_GAIN Epoch 75-76 file (CalDB 4.8.3, May 2019), to all ACIS time-dependent gain files. The algorithm has resulted in considerable improvement in the resulting noise level in the corrected PHA/ENERGY data in the events lists.

  See the time-dependent ACIS T_GAIN why page for more information.

  While the gain corrections improves results for imaging spectroscopy, grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

• ACIS Blank-sky Background Files

  This update to the blanksky background files changes header keywords, expanding the focal plane temperature boundaries to let them be selected using acis_bkgrnd_lookup when working on ObsIDs with warmer ACIS focal plane settings.

HRC Data

• HRC-S Time-Dependent Gain Maps

  Rapid loss of HRC-S gain since the year 2015 has led to problems with filtering background events from HRC-S/LETG LETG/HRC-S observation spectra, requiring the development of a new time-dependent method to generate the gain maps. Making use of these new T_GMAPs require changes to hrc_process_events which were introduced in CIAO 4.12.

  This results in being able to filter on pulse height in LETG spectra as a function of dispersed wavelength in order to reduce the background by over 50% for wavelengths >20 Å, compared with standard Level 2 processing.

• HRC-S QE Files

  Regular calibration observations have shown that the HRC-S QE is declining at longer wavelengths (energies <200 eV, >60 Å), decaying linearly since the 2012-03-29 high-voltage adjustment onboard. This spectral variation in the QE loss is accounted for using a series of time-displaced QE files introduced in this CALDB.

  These CIAO tools and scripts automatically apply the HRC-S QE files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp

• HRC-I Time-Dependent QE Files

  The HRC-I quantum efficiency QE has been decaying for some time, particularly since the year 2011, when a 5% to 10% per year reduction had been measured. To address this issue, the HRC-I Calibration Team had introduced a time-dependence into the QE files in 2018 which are now extended in time to the latest epoch of observations. While the time-dependent modeling of the QE has improved, this update introduces a retroactive change of ~0.5%, even for the oldest epochs. The QE update affects the results from mkinstmap and fluximage and is applicable to all previous epochs of HRC-I observations.

---

How CALDB 4.9.1 Affects Your Analysis

ACIS Imaging and Grating Data

• Time-dependent ACIS Gain (T_GAIN) Files for -120 C Data

  The new, combined time-dependent ACIS Gain (T_GAIN) file for November 2019-January 2020 (Epoch 80) and August-October 2019 (Epoch 79) are introduced. The combined Epochs 77+78 file has also been updated since CALDB 4.8.4.1. Therefore, the new T_GAIN affects observations taken since 2019 February 01. Other observations are unaffected by these new calibration files.

  Users working with ACIS data taken since 2019 February 01 may wish to run chandra_repro and reprocess the data to improve the T_GAIN calibration. The DATE-OBS header keyword records the observation start date.

  See the time-dependent ACIS T_GAIN why page for more information.

  The gain corrections are at the nominal and expected levels of less than 1% of the energy value and users interested in CCD spectroscopy may benefit in applying the new gain adjustment. Grating spectroscopy benefits as well, in the form of improved order sorting. It is unnecessary to apply this adjustment for only doing timing or imaging analysis, although doing so will not have a negative effect.

  Note that only spectra with several hundreds of counts and/or prominent features (in emission or absorption) will show changes from the T-gain refinement that exceed the uncertainties from the gain calibration.

• DET_GAIN and P2_RESP FI CCDs Upgrade

  For some time, the ACIS calibration team has been aware of a small gain drop for each front-illuminated CCD, persisting since the beginning of the mission. Currently, these "gain droops" have some energy dependence in their depth as well as their spatial extent, seen in the mid-chip region of the FI CCDs corresponding to: CHIPX=~480-540, and CHIPY=~100-900. The current ACIS DET_GAIN file version N0007 (along with earlier versions) does not correct for this effect. The new version N0008 DET_GAIN file accounts for and corrects for the gain droop on all eight of the FI chips. Along with a corresponding P2_RESP N0008 file (which is the input for CIAO 4.12 tool mkacisrmf, and must be matched to the DET_GAIN file), the new DET_GAIN file is applicable to non-GRADED DATAMODE cases, wherein the CTI_APP keyword is set to "PPPPPBPBPP", indicating the configuration where all ACIS chips are CTI-corrected. At present, there is no corrected DET_GAIN or P2_RESP applicable to GRADED mode data sets.

  Note that the BI chips ACIS-S1 and ACIS-S3 are not affected.

  These CIAO tools and scripts automatically apply the gain map files:

  • acis_process_events
  • chandra_repro

  and the P2_RESP files are used by:

  • specextract
  • mkacisrmf

HRC Data

• HRC-S Time-Dependent QEU Files

  The HRC-S time-varying quantum efficiency uniformity files have been replaced with a new set, calculated based on unfiltered calibration observations. This latest set (N0013) shows very minor change over the prior set of QEU files, but incorporates results from the latest HZ43 calibration observations; however the newest QEU for observations taken since 2019 August 22 includes the ~5% drop of QE over the 0.060-0.652 keV range, above that range is essentially unchanged. The set of QEU files are unevenly distributed in effective dates and affects every HRC-S observation since the start of the mission.

  These CIAO tools and scripts automatically apply the HRC-S QE and QEU files when creating response files:

  • mkgarf
  • fullgarf
  • mktgresp
  • mkinstmap
  • fluximage
  • merge_obs

---

How CALDB 4.9.0 Affects Your Analysis

ACIS Imaging and Grating Data

• ACIS Bad Pixel Library

  The latest CalDB ACIS bad pixel files have been updated to make use of status bit 17, so that the ACIS framestore shadow can be accounted for in TE mode observations. The ACIS framestore shadow is a rectangular region due to shadowing by the framestore cover, which differs from chip-to-chip, that includes the first several rows of each chip, and is henceforth designated as bad pixel regions.

  Events falling in these regions are being handled differently in the software, so they may be re-included manually, but by default are filtered out in level 2 event file.

  Users working with ACIS data may wish run chandra_repro to reprocess the data, since this change directly affects the acis_build_badpix tool. This change will not be incorporated in to the Standard Processing pipeline until early 2020. Futhermore, the framestore shadow is currently not accounted for in CC-model data sets.

---

Installation

Supported Platforms

• CIAO 4.12 is supported on:

  • Linux 64 bit
  • Apple macOS 10.12 (Sierra) through macOS 10.15 (Catalina)

Updates to the ciao-install Installation Script

• Various updates to support a single Linux and a single Mac download.

• Updated ciao-install to check for strings utility before trying to use it.

• The ciao-install script now retrieves the CIAO tar files from via using encrypted https://. The CALDB tar files continue to be served using ftp://.

• Improved support for internationalization during installation.

• Changed behavior of the --caldb option to correctly deal with linked CALDBs when applying updates.

• The smoke tests have been updated to avoid conflicts due to the different versions of IPython used by CIAO (e.g. CIAO 4.11 provided IPython 6 whereas CIAO 4.12 contains IPython 7).

• The smoke tests have also been updated for the removal of chips, and SKIP some tests if running in an environment with X11 ( eg headless or remotely logged into another system).

Installing with the conda package manager

• CIAO can now be installed using the conda package and environment management system.

  The conda release is available from the custom CXC channel, and can be installed with the following command if you have conda installed (see the conda installation instructions for more information, including known problems and incompatabilities):

  $ conda create -n ciao-4.12 -c https://cxc.cfa.harvard.edu/conda/ciao/ ciao sherpa ds9 caldb_main marx
  

  With the conda edition of users can choose to install CIAO with Python 3.7 (the default), 3.6, or 3.5. Other OTS such as numpy and matplotlib will also be be different based on the python version.

Source builds

• Improved consistency of build flags such as compiler optimization.

• Remove invalid C++ warning flags

IPython Settings

• Users will be prompted to update their IPython profiles the first time they run sherpa

Remove old parameter files

• With every new CIAO release, some parameter files are changed: new parameters may be added and occasionally old ones removed or renamed. Deleting or renaming the local parameter directory ensures that the correct parameter files will be accessed the first time a tool is run:

  unix% rm ~/cxcds_param4/*
  

---

Tools

axbary

• Updated to recognize MJD-OBS keyword and convert from the unsupported MJD_OBS keyword.

acis_build_badpix

• Added support for new badpixel status bit 17 : frame store shadow.

  There is a "shadow" cast on the bottom rows of each ACIS CCD by the frame-store covering. Since there are fewer X-ray events the calibration in these rows is more uncertain and are now marked as bad.

  The default bitflag parameter has been updated to include bit 17.

• Fixes possible memory corruption when setting bad status values.

acis_process_events

• Added support for new badpixel status bit 17 : frame store shadow.

  There is a "shadow" cast on the bottom rows of each ACIS CCD by the frame-store covering. Since there are fewer X-ray events the calibration in these rows is more uncertain and are now marked as bad.

• A number of obsolete parameter have been removed. This includes: alignmentfile, time_offset, tstart, tstop, and instrume.

• Implemented new polynomial coefficients for the temperature dependent CTI corrections.

  With CALDB 4.9.0, the higher order coefficients are all 0; therefore there is currently no change in the temperature dependent calibration. Future CALDB updates will include updated calibrations which are expected to affect data obtained at warm temperatures.

• Update rules for calculating and propagating PHA_RO values from input file based on calibration files and parameter settings.

• Update CTI_APP keyword based on input file, available calibrations, and parameter settings.

• Added support for upcoming changes to aspect solution files.

acis_streak_map

• Fix syntax error with higher verbose messages.

addresp

• Updated to handle XMM files where the first channel is 0 instead of 1.

celldetect

• Correct bug misidentifying double sources when using fixed cell sizes.

dither_region

• Remove constraint that stack of asol files share the same NOM values since those values now are obtained from the wcsfile.

dmcoords

• Fixes interactive bug where a string value was copied over itself potentially causing memory corruption.

dmcontour

• Fix bug closing polygons when contour hits the edge of the image.

dmfilth

• Now correctly propagates units from the input image to the output image.

dmimgadapt

• Now correctly propagates units from the input image to the output image.

dmimgcalc

• Now correctly propagates units from the input image to the output image.

dmimgdist

• Now correctly propagates units from the input image to the output image.

dmimgfilt

• Now correctly propagates units from the input image to the output image.

dmimgpm

• Now correctly propagates units from the input image to the output image.

dmimgreproject

• Now correctly propagates units from the input image to the output image.

dmkeypar

• Update handling for unsigned long keyword values.

dmmaskbin

• Now correctly propagates units from the input image to the output image.

• Corrects initialization error when pixel at 0,0 is NaN.

dmmaskfill

• Now correctly propagates units from the input image to the output image.

dmnautilus

• Now correctly propagates units from the input image to the output image.

dmregrid2

• Now correctly propagates units from the input image to the output image.

dmsort

• Now will correctly sort columns with IEEE special values: NaN < -Inf < Real Numbers < Inf.

hrc_build_badpix

• Update to new degap calibration file to use modified HRC_OFFX and HRC_OFFY keywords. Previous calibration files used non-standard 9-character keyword names.

hrc_process_events

• Updated to use MJD-OBS keyword.

• New PI gain calibration algorithm.

• Update to new degap calibration file to use modified HRC_OFFX and HRC_OFFY keywords. Previous calibration files used non-standard 9-character keyword names.

reproject_image

• Now correctly propagates units from the input image to the output image.

reproject_image_grid

• Now correctly propagates units from the input image to the output image.

skyfov

• A new algorithm to create a smaller bounding polygon has been implemented using the new method=convexhull parameter.

  However, for observations with large changes in the aspect solution DY,DZ columns, this new FOV boundary does clip the observation. A redefinition of the aspect solution files is being developed which eliminates this concern.

  The default method=minmax implements the original algorithm.

• The output FOV will now only contain polygons for the chips included in the input file (ie after any filters have been applied).

• Improved consistency checks between input file, aspect solution, and mask file.

• Improved error checking for cases with 0 rows in aspect solution (eg all bad time).

• Use _AVG keyword keywords from the input file is asol file is omitted.

tgdetect

• Restore the xoffset and yoffset parameters.

wavdetect

• Three new vector columns have been added to assist users when dealing with duplicate source detections:

  1. CORRELATION_MAX(X_CORRELATION_MAX,Y_CORRELATION_MAX) : The location of the correlation maximum with the wavelet. For well behaved sources, the correlation maximum will be close to the source centroid (X,Y), and will be close to the location of the flux maximum. Some duplicate sources will have large distances between the correlation_max and the flux_max.
  2. FLUX_MAX(X_FLUX_MAX,Y_FLUX_MAX) : The location of the local maximum closest to the correlation maximum. (See correlation_max discussion above.)
  3. SCALES(X_WAVELET_SCALE,Y_WAVELET_SCALE): the actual wavelet scale the source was reconstructed using. Users should find that the scales are different for duplicate sources.

---

Parameter Files

acis_build_badpix

• The default bitflag=00000000000000122221100020022222 now includes bit 17 for the frame store shadows.

acis_process_events

• A number of obsolete parameter have been removed. This includes: alignmentfile, time_offset, tstart, tstop, and instrume.

skyfov

• A new method parameter has been added. Users should leave the parameter at its default method=minmax to use the standard FOV algorithm. The new method=convexhull algorithm should not be used at this time.

tgdetect

• Reintroduced the xoffset and yoffset parameters.

---

ChIPS

---

Sherpa

Switching to Matplotlib

• The default behavior is for Sherpa to use Matplotlib for its plotting in CIAO 4.12.

  If you have used the Sherpa application with previous versions of CIAO then your sherpa configuration file is likely to still be set up to use ChIPS rather than Matplotlib. In this case you will see the following warning message when you start Sherpa:

  WARNING: chips is not supported in CIAO 4.12+, falling back to matplotlib.
  WARNING: Please consider updating your $HOME/.sherpa.rc file to suppress this warning.
  

  and Sherpa will switch itself to using Matplotlib. Please see the Sherpa FAQ entry on how to change the resource file to stop seeing this warning.

  

  Tip

  Unlike CIAO 4.11, users of the Sherpa application no-longer have to use either the %matplotlib IPython magic command, or say import matplotlib.pyplot as plt to have access to Matplotlib in interactive mode.

  Sherpa contains a number of functions that change the plot preferences (i.e. how new plots will be displayed), such as set_xlog. However, users may be used to using ChIPS commands - such as limits or log_scale - to change existing plots. Please see the ChIPS to Matplotlib conversion guide to find out the appropriate replacements (which in this example are plt.xlim, plt.ylim, plt.xscale, and plt.yscale).

Plotting improvements

• The plot functions - that is, all functions beginning plot_ (but not plot) can now accept keyword arguments to over-ride the plot preferences. This lets you ensure the plot has logarithmic axes, such as

  plot_source(xlog=True, ylog=True)

  or overplot fits while using a consistent color scheme for the data and model:

  plot_fit(1, color='blue')
  plot_fit(2, color='orange', overplot=True)
  

  The supported keywords are the same as the entries in the dictionaries returned by get_data_plot_prefs() and get_model_plot_prefs().

• Two new plotting routines have been added - plot_fit_ratio and plot_bkg_fit_ratio - which create a figure containing two components, where the top plot is the fit, and the bottom one is the ratio of the model to the data. These functions complement the existing plot_fit_resid and plot_fit_delchi routines.

  There are no ahelp pages for these new commands. Please use the native Python help system - i.e. at the sherpa prompt say:

  help(plot_fit_ratio)

• The plot_fit_resid, plot_fit_delchi, and (new) plot_fit_ratio functions now create figures where the top plot is larger than the bottom plot. In CIAO 4.11 this was not the case with Matplotlib (but did work correctly when using ChIPS).

  The order of the model and data in plot_fit has been adjusted with the Matplotlib backend (changes in Matplotlib 2.0 meant that plots created with CIAO 4.11 and Maplotlib had the model line overdrawn by the error bars).

• The plot_pvalue function has seen improvements when fitting PHA data.

XSpec

• The XSPEC models have been updated to version 12.10.1n.

  The following additive models have been added: xsagnsed, xscph, xskyrline, xsqsosed, xsvcph, xszbknpower, and xszlogpar.

  The update to version 12.10.1 of XSPEC has meant that the default cross-section table has changed: it now defaults to vern when it was bcmc in CIAO 4.11 and earlier. Note that this default setting is now read from your $HOME/.xspec/Xspec.init configuration file (if it exists), using the XSECT setting.

  The default abundance table will now be set from your $HOME/.xspec/Xspec.init file, if the ABUND line is set.

Notable changes

• The calc_ftest function has been updated to better match the XSPEC version and the documentation.

Bug fixes

• The full list of changes made in this version of Sherpa are:

  Remove ChiPS dependency from Sherpa startup script

  Sherpa's IPython profile startup script does not assume ChiPS is installed and it skips the version check. The code that imports the ChiPS commands into the root namespace has been left for compatibility. Users upgrading to CIAO 4.12 and then going back to CIAO 4.11 will be asked if they want to "update" their profile scripts. If they don't, and they have ChiPS as their preferred backend, ChiPS will be imported in the shell. Additionally, Sherpa now warns the user if ChiPS is selected as a backend but it is not available in the installation. In this case Sherpa will also try and fall back to the pylab backend.

  Enable Matplotlib in sherpa shell

  Since ChiPS has been removed from CIAO, the shell wrapper will now run %matplotlib when using the pylab backend so Matplotlib plots will appear on the screen when plot commands are issued.

  Update ipython_config.py to work with ipython 7.2

  Modified the ipython_config script used for the 'sherpa' command shell to work with ipython 6 or 7. The prompt api changed between the two ipython versions and broke compatibility.

  Remove ChiPS reference in Sherpa wrapper and update the rc file check in ciao_app_setup

  The references to ChiPS have been removed from the Sherpa wrapper and ciao_app_setup.sh.

  Update white spacing in ipython profile compatibility warnings

  Updated white spacing in ciao_app_setup.sh so that the messages issued by Sherpa contain less spaces.

  User Model: unable to change parameter values (Fix #609)

  A regression introduced in Sherpa 4.10.0 prevented users from changing user-model parameter values through direct access. This issue has been fixed. Several tests were added.

  update ciao default plotter to Matplotlib

  The default Sherpa plotting package has been changed to Matplotlib for CIAO (it had been the default for standalone Sherpa for several years). The update will be applied when the `sherpa` command is run. Users of the previous versions would need to edit/regenerate the sherpa.rc file.

  stop using numpy.typeNA which is now deprecated

  A numpy deprecation warning was fixed by removing the usage of typeNA, which was not documented and will be removed in a future release of numpy. The print_fields function has been changed to include a default mapping using the current typeNA implementation.

  update pager code (fix #445 #561)

  Replace the use of an external pager (such as more or less) with a Python one. This means that Sherpa no-longer uses the PAGER environment variable, and that the screen output from the show_*series of commands should now appear in the correct location when using a Jupyter notebook or the spyder application.

  Fix regression with responses `get_x` method (Fix #572)

  Version 4.10.1 introduced a regression which made the get_x() method fail when called on RMF and ARF objects. This has been resolved.

  fix LevMar when dof < 0 and Fix LevMar when dof < 0 and harmonise the handling of dof is zero or negative in fit and calc_stat_info (fix #565)

  The fit and calc_stat_info methods now use the same code to calculate the reduced statistic (rstat) and "quality of fit" (qval) values. This avoids a TypeError when the number of degrees of freedom is negative in calc_stat_info and ensures that NaN is returned for both values when the goodness-of-fit calculation has failed (e.g. the reduced statistic is zero or less or the statistic is negative), whereas in previous versions the qval value could be None or 1 depending on the code path.

  fix gammapy/gammapy#1905, catch errors with psf pixel size

  Sherpa did not properly catch errors when checking the PSF pixel size matches the image pixel size. This has now been fixed.

  Fix XSPEC models: mtable table models and ismabs (fix #535 #538 #540)

  This PR fixes:

  • #535 XSPEC multiplicative table models (mtable models loaded with load_xstable_model)
  • #538 incorrect evaluation of the ismabs model
  • #540 the kerrd model evaluated to 0 for XSPEC 12.10.0 and later

  For the first two, the models were being multiplied by the value of the last parameter of the model. At best this would lead to fits where the overall model amplitude was incorrect but could be corrected. However, it is likely that such fits were not robust (due to the accidental correlation between the overall amplitude and the last parameter of the model), and so it is not guaranteed that the best-fit would be found. The kerrd model was due to a change in the naming of the XSPEC functions which had not been reflected in the XSPEC model.dat file in XSPEC 12.10.0 or 12.10.1b, which meant that the kerrd model in XSPEC 12.10.0 or later was wrong in Sherpa.

  XSPEC 12.10.1 support and mtable fix

  Sherpa now supports XSPEC 12.10.1. Several issues were fixed to ensure compatibility with this release. Note that there are known issues with version 12.10.1 (no patches), so at least v12.10.1a must be used.

  ensure XSPEC 12.10.0 uses ATOMDB 3.0.9 by default

  Ensure that the default AtomDB version is 3.0.9 when using XSPEC 12.10.0. This is to fix an issue with the models-only XSPEC installation in XSPEC 12.10.0 which uses a default version of 3.0.7 but only provides data files for 3.0.9. Without this change models such as XSapec would return all 0's (unless the AtomDB version was set manually by the user).

  Be explicit about "invalid escape characters" in strings

  Convert strings that contain "invalid" escape characters, which newer versions of Python complain about, to raw strings by preceding them with a 'r' character. This avoids the test suite from reporting DeprecationWarnings.

  Ensure the ui modules have a unique list of exported symbols (fix #502)

  Remove the sherpa.utils.erf symbol from the two ui modules as it is "over-written" by the ModelWrapper-created symbol when applied to sherpa.models.basic.Erf. Tests have been added to point out if this ever happens again: for example, if any new models are added to Sherpa, they could conflict with an existing symbol, and these tests will pick that up.

  Fix "get_int_proj does not work when recalc=True" (#543)

  get_int_proj did not work when recalc=True on Python 3. This has now been fixed.

  Asymmetric Errors

  Sherpa now supports asymmetric error bars. Errors can be read through a new `load_ascii_with_errors` high level function, or through the new `Data1DAsymmetricErrs` class. Sherpa uses bootstrap for estimating the uncertainties.

  plot_pvalue

  Updates to utilize the appropriate response files (ARF and RMF) for X-ray spectra and changes to the p_value output to 1/(number of simulations) when p_value is 0 and the number of simulations in not large enough.

  Run optimization algorithms over multiple cores

  This PR enables the user to run the optimization algorithms (DifEvo, LevMar, and NelderMead) on multi-cores.

  PSF rebinning (fix #43)

  Sherpa now supports using a PSF with a finer resolution than 2D data images. If Sherpa detects that the PSF has a smaller pixel size than the data, it will evaluate the model on a "PSF scale" that has the same resolution (pixel size) as the PSF and the same footprint as the data image, then rebin the evaluated model back to the data space for calculating the statistic.

  Refactor data classes (fix #563, #627, #628)

  Sherpa's basic data classes have been refactored and cleaned up to help facilitating fixing bugs and implementing new features. New tests were added to reduce the chances of introducing regressions in the future.

  Reduce integration tolerance (tol)

  Changed integration routine tolerance from double epsilon to float epsilon since optimization routines have a tolerance of ~1.0e-7.

  enable run-time-option to test cache (default=True)

  Capability to allow the user to turn off caching at runtime for testing purpose.

  Do not create warnings about error bars in plots if no error bars are to be shown (fix #621)

  Remove the "The displayed errorbars have been supplied with the data or calculated using chi2xspecvar..." warnings that appear for data, residual/ratio, and fit plots when the user has explicitly turned off the display of errorbars (by setting the yerrorbars plot-preference setting to False).

  add plot_fit_ratio and plot_bkg_fit_ratio functions

  Add the plot_fit_ratio function to the sherpa.ui layer (to match plot_fit_resid and plot_fit_delchi) and plot_bkg_fit_ratio to the sherpa.astro.ui layer. Added tests for a number of plot types.

  fix 'tuple index out of range' error in the fit.est_errors method

  Sherpa's Confidence class is confused if a parameter is frozen then thawed after a fit resulting with an exception of the form: IndexError: tuple index out of range. The error does not show up if conf() is called from the ui level.

  fix numpy hist warnings

  When plotting a probability density function (PDF) Sherpa now calls numpy.histogram with density=True|False rather than normed=True|False. There is no change visible to the user other than a warning that will not be issued anymore. normed=True was broken for nonuniform bins, but our code always produces uniform bins, so we should never hit the problematic case.

  Handle files with MJD_OBS or MJD-OBS keywords (datastack module)

  The sherpa.astro.datastack module has been updated to deal with files that use the MJD-OBS keyword instead of MJD_OBS. This is only used by the show_stack routine, which prints out several keywords from each file in the stack.

  Remove python 2.7 support (fix #498)

  Remove support for Python 2.7 from Sherpa. The metadata used by pip now requires python 3.5 or higher (but not version 4),

  XSPEC: build against 12.10.1m by removing support for models

  Remove support for XSrelline, XSrelline_lp, and XSrelline_lp_ext models. These were added in 12.10.1 but removed in 12.10.1m.

  Remove the unused parameter in sherpa.astro.optical.LogEmission model (fix #219)

  Remove the unused parameter in the sherpa.astro.optical.LogEmission model. The LogEmission model only has four parameters (fwhm, pos, flux and skew) consistent within the calc method. The hidden parameter limit and it docs were removed.

  Remove ChiPS from the documentation

  Remove mention of ChiPS from the documentation (both on read the docs and in the sherpa.astro.ui and sherpa.ui modules), replacing with Matplotlib where appropriate.

  Improve documentation and testing for get_source_plot when sent lo/hi arguments

  Improve the documentation for the sherpa.astro.ui.get_source_plot routine, describing how touse the result when the lo or hi arguments are sent.

  Add default config options

  Whenever an option is read from the configuration file (e..g. ~/.sherpa.rc) provide a default option in case the configuration file is missing (or is missing this option).

  jointplot now respects the ratio argument (top plot is larger) with Matplotlib backend

  The matplotlib backend now makes the main (top) plot taller than the secondary (bottom) plot when using the sherpa.ui.plot_fit_resid and sherpa.ui.plot_fit_delchi routines (this also holds for the sherpa.astro.ui variants).

  Add link/unlink parameter tests

  Added three new tests for sherpa's un/link commands. The tests make sure that all the members of the Parameter class are correct for the low and ui levels of sherpa's link function.

  Address indexing DeprecationWarning/IndexError from NumPy for PHA data with filter + ignore_bad (fix #361)

  Add warning about masked array to sherpa.astro.ui.load_arrays

  Updated docstring to contain warning consistent with PR#698.

  modelCacher1d needs to have a couple of deep instead of shallow copy()

  Update to make deep copy of cached function values (fix to IS#673).

  Improve Matplotlib plot_fit plots (order of data and model)

  Changes to 'zorder' of plot objects drawn by Matplotlib to make plots more readable.

  Allow users to override plot preferences when creating a plot

  Change to allow kwargs to be specified to change plot preferences at creation time.

  Add warnings in documentation about masked arrays

  Updated docstring to indicate that Sherpa doesn't support numpy masked arrays.

  change calc_ftest from delta dof to dof (similar to XSPEC) - fix for IS#641

  Updated input to calc_ftest() to reflect setting specified in the documentation and also aligned with the XSPEC interface.

  a fix for internal regression test issues

  add warning when ChiPS is selected but cannot be imported (rebase of #648)

  Sherpa now warns the user if ChiPS is selected as a backend but it is not available in the installation.

  raise ModelErr when model using Data1DInt have overlapping bins

  Addition of a check to 'regrid' function to make sure that the integrated grid has non-overlapping bins. The PR is a fix for issue #569.

---

Graphical User Interfaces

prism

• prism now use Matplotlib for plotting.
• prism now displays an error message when trying to modify a compressed file.

peg

• Updates to support GTK 3.

taskmonitor

• Updates to support GTK 3.

obsvis

• IPAC-2mass survey has been added

  Internal work to use ds9 command line options

• Added NED@CDS, SIMBAD@CDS, VIZIER@SAO and VIZIER@CDS to the obsvis list of name servers.

• Add an option to specify edge and center blanking for HRC-I and HRC-S through obsvis.

---

Analysis Scripts

chandra_repro

• Updated to include new badpixel bit, 17, when recomputing badpixels.

check_ciao_version

• Updated to work with CXC secure web server

• Updated to work with conda edition of CIAO.

check_ciao_caldb

• Updated to work with CXC secure web server

combine_grating_spectra

• Remove the TG_M and SPEC_NUM keywords from the output file. Previously the values were taken from 1st file in the merge.

combine_spectra

• Remove any TG_M filter from the output file subspace. This cleans up the final output file to remove extraneous subspace components.

dax

• Updated to create residual plots for spectral fits and radial fits.

deflare

• Updates to support matplotlib 3.1

make_instmap_weights

• The default setting for the xsect parameter has changed from bcmc to vern to match the behavior of the XSPEC 12.10.1 model library.

tgsplit

• Fixes a bug when trying to work with TYPE:II pha files created by the tgextract2 tool.

ciao_contrib.runtool

• Updated for parameter changes in this release.

lightcurves

• The module has been updated to work with matplotlib 3.1.

ciao_contrib.all

• This module no-longer imports the chips_contrib module.

sherpa_contrib.utils

• The plot_instmap_weights routine has been updated to use matplotlib preference settings (color) rather than ChIPS ones (linecolor and linethickness) from get_data_plot_prefs. The preferences can also be over-ridden when the plot is created.

sherpa_contrib.xspec.xsconvolve

• Added the load_xskyconv routine to support the kyconv convolution model. Please note that support for XSPEC convolution models is considered experimental and there are known problems, so use carefully.

chips_contrib

• The chip_contrib.decorators, chips_contrib.helix, chips_contrib.images, chips_contrib.regions, chips_contrib.scatter, and chips_contrib.utils modules have been removed, as ChIPS is no longer distributed as part of CIAO. The Matplotlib plotting package should be used instead.

  

  ChIPS to Matplotlib conversion guide

  Please see the ChIPS to Matplotlib conversion guide if you need help. The CXC Helpdesk can be used if you still have problems.

---

Python Modules

caldb

• Remove hard coded build dependencies.

crates

• CrateData objects can now take unit=None and desc=None. These are converted to empty strings.

  keywords and columns can now be added at a specific index (order) using add_key() and add_column().

• Update to correctly copy image units

• Fixed bug where a 1x1 image was returned as a 1D array.

• Speed improvements

group

• Improved configuration to support various python versions.

history

• Extended interface to allow for creating a HistoryRecord object without any parameters.

paramio

• Increased maximum string length to 2048 characters to match C library.

  Improved error checking when string length max is exceeded.

• Improved configuration to support consistent optimization levels.

pixlib

• Allow greater flexibility when setting parameters (duck typing)

  Improved error handling: throwing exceptions rather than setting hidden error status codes.

• Improved pkgconfig dependencies for easier builds

psf

• Improved configuration to support consistent optimization levels.

region

• Improved configuration to support consistent optimization levels.

• Region shapes can now be sliced to obtain a subset of the region

  >>> from region import *
  >>> srcs = circle(10,10,10)+circle(20,20,20)
  >>> excld = field()-srcs
  >>> slce = excld[1:]
  

  The shapes that are returned are always included and are logically OR'ed together.

stk

• Improved configuration to support consistent optimization levels.

transform

• Improved configuration to support consistent optimization levels.

---

Libraries

ardlib

• CIAO now provides a pkgconfig file to make building tools with this library easier.

cxcparam

• Updates to allow for mode=H. This mode forces exit when illegal parameters are encountered rather than prompting the user for a legal value.

  Improved error checking for invalid mode values.

  Additional internal code cleanup.

  These updates were provided by Diab Jerius.

• Updated float to string conversion to omit ends (end string) from C++ classs.

dm

• CIAO now provides a pkgconfig file to make building tools with this library easier.

• Improved composite filtering using NULL values. Filters such as [(tg_m!=0)&&(tg_part!=99)] could allow the tg_m NULL values through the filter. These are now properly omitted.

• Miscellaneous internal code clean up.

• ASCII TSV files with duplicate columns names are now displayed correctly.

• ASCII files with long column or keyword descriptions can now be processed.

• Improved parsing of strings with leading numeric values: for example when column contain dates of the form: 2019-12-15.

dslib and hdrlib

• Chandra has been improperly using the keyword MJD_OBS in file headers. FITS WCS paper II specifies that the correct keyword name is MJD-OBS (a dash instead of an underscore). This has lead to some minor incompatibilities interpreting the World Coordinate Systems when using Chandra data products with 3rd party applications. This change ensures that the proper MJD-OBS keyword is created and will be corrected when propagating header information.

  This change affects the output from most CIAO tools.

pixlib

• Internal code cleanup.

region

• CIAO now provides a pkgconfig file to make building tools with this library easier.

---

Environment

Environment variables

• The ASCDS_WORK_PATH environment variable is used to specify the temporary directory used by several CIAO tools. The ciao setup scripts will now use the same values as the TMPDIR environment variable if it is set. Otherwise, it will default to the /tmp.

• The ASCDS_OTS environment variable is now set for both bash and tcsh shells.

Off-the-Shelf (OTS) Package Versions

• The following OTS packages are included with CIAO 4.12 installed with ciao-install. For more information on how the OTS packages are built for use with CIAO, refer to the INSTALL_SOURCE file distributed with the software.

  

  These do not hold for the conda-installed version of CIAO

  The conda-installed version of CIAO 4.12 has different OTS versions, and it also depends on the Python version you have chosen to use.

  Executables

  • ds9 8.1

  Python

  • Python 3.5.4
    Frameworks on OSX have been disabled to avoid a problem with users picking up the wrong version of NumPy.
  • NumPy 1.15.4
  • ipython 7.2.0
  • six 1.12.0
  • pip 19.1.1
  • setuptools 28.8.0
  • cython 0.285
  • Matplotlib 3.0.2
  • jupyter notebook 6.0.0
  • pytest 4.0.2

  Libraries

  • cfitsio 3.470
  • CCfits 2003.12.08
  • wcssubs 5.19
  • readline 6.3
  • ncurses 5.7
  • XSPEC v12.10.1n (models only)
  • fftw 3.3.8
  • gsl 2.5
  • xerces 3.2.2
  • GTK 3.22.30
  • adwaita-icon-theme 3.20
  • glib 3.56.2
  • hicolor theme 0.17
  • fontconfig 2.13.1
  • freetype 6.17.0
  • jpeg 9.3.0
  • tcl 8.6.9
  • tk 8.6.9
  • xpa 2.1.18
  • openSSL 1.0.1e

Compiler versions

• The following versions of the gcc compiler were used to compile the binary CIAO releases installed with ciao-install. The conda-installed compiler versions are different.

  System	Build System	Version
  Linux	CentOS 6	gcc 4.8.5
  macos	Sierra	llvm 8.0.0

---

Documentation

---