CIAO 4.2 Release Notes

Sherpa v2 Release

The Sherpa v2 patch to CIAO 4.2 was released on 19 July 2010.

This patch contains changes to Sherpa, CIAO's modeling and fitting package. Sherpa v2 has a number of enhancements, such as two new iterative fitting methods, refinements to parallelization, and many improvements to instrument and source models.

Support for the S-Lang scripting language has been removed from Sherpa in this release. S-Lang is still included for the CIAO tools and modules and for the ChIPS plotting application. The CXC is committed to helping existing S-Lang users transition to Python; contact Helpdesk if you need assistance.

CIAO 4.2 & CALDB Release Notes

• Platform Support
• Sherpa
• Analysis Scripts
• Documentation

Platform Support

Mac OS X 10.4 PowerPC

• The Sherpa v2 patch is not available for the Mac OS X PPC 10.4 platform. PPC users who install CIAO 4.2 will get Sherpa v1.

Sherpa

Removal of S-Lang Support

• Sherpa v2 no longer supports S-Lang. The S-Lang interface has been removed and there is no method for importing Sherpa as a S-Lang module, e.g. 'require("sherpa")', into slsh or any other S-Lang application.

  As a consequence, the option to use slsh as the interpreter in the Sherpa application (the "-l" switch) has been removed.

Starting Sherpa

• Sherpa uses the $HOME/.ipython-ciao/ directory to store configuration information. When Sherpa is started, you may see a message about updating this file:

   ATTENTION: Out of date IPython profile for Sherpa found in:
             /home/username/.ipython-ciao
  
             Local version (40201) vs latest (40202).
             Update to latest? [Y/N] :
  

  Unless you have changed your $HOME/.ipython-ciao/ipythonrc-sherpa file, answer "Y". The outdated file is renamed with a timestamp to preserve it.

  The new profile is installed read-only. If the user wants to modify how Sherpa uses IPython functions, or call "execfile" to load Python scripts when Sherpa starts, the customization file $HOME/.ipython-ciao/ipythonrc-sherpa-user should be used.

Iterative Fitting

• Two iterative fitting methods have been added to Sherpa: Primini's methods and sigma-rejection. Both were fitting methods in Sherpa 3.4, and their ports to Sherpa 4 have been completed.

  The essence of an iterative fitting method is that the fit method can be called several times, until some criterion is met.

• Primini's method is to re-calculate statistical errors, using the best-fit model parameters from the previous fit, until the fit can no longer be improved.

  Sigma-rejection is based on the IRAF SFIT function. In successive fits, data points for which ((data - model) / error) exceeds some threshold are added to the filter, and automatically excluded from the next fit.

  Primini's method and sigma-rejection can only be called when the statistic is a chi-squared statistic. They cannot be used with least-squares, Cash or C-statistic.

• Several new UI functions have been added, to allow users to set the iterative fitting method, to find out what the current iterative fitting method is, and to get and set options for this method. These functions are:

  • set_iter_method(<string>)
  • get_iter_method_name()
  • list_iter_methods()
  • get_iter_method_opt(<string>)
  • set_iter_method_opt(<string>, value)

  If the iterative fitting method is "none" (the default value), then no iterative fitting is done - when "fit()" is called, the optimization method is called once, and Sherpa otherwise operates as expected.

  The statistic and optimization methods are selected independently of the iterative fitting method - thus:

  sherpa>  set_stat("chi2datavar")
  sherpa>  set_method("neldermead")
  sherpa>  set_iter_method("primini")
  sherpa>  fit()                      # Primini's method is called
  sherpa>  set_iter_method("none")
  sherpa>  fit()                      # Nelder-Mead is called once, as expected
  

Filtering and Showing Data

• Updates to load_filter() include the ability to read FITS images that hold filter information. There is a new keyword argument 'ignore' that indicates whether the filter information should be used to notice or ignore data points. 'ignore' is False by default.

  load_filter(id=1, filename, bkg_id=None, ignore=False, ncols=2)
  set_filter(id=1, val, bkg_id=None, ignore=False)
  

• The file header information is no longer included in the output of show commands.

• show_all() displays the correct response information for PHA data sets. Information on background datasets has also been restored in show_all().

• show_bkg_model() now displays the background scale factor.

Instrument Responses

• New function get_response() returns the associated PHA instrument (RMF + ARF) or any combination or iteration. This response object is callable for use in a model expression. Backgrounds are supported using the bkg_id argument. This is especially useful when dealing with multiple responses.

  rsp = get_response()
  
  set_full_model(rsp(xsphabs.abs1*powlaw1d.p1))
  

• High level functions get_arf() and get_rmf() return 'instrument' versions of the ARF or RMF dataset that include a callable functionality. This allows the user to define a response in a model expression using arf and rmf instances. The multiplication with the exposure time is implicit.

  arf = get_arf()
  rmf = get_rmf()
  
  set_full_model(rmf(arf(xsphabs.abs1*powlaw1d.p1)))
  

Source and Background Models

• Updates to plot_source() now support the 'factor' setting of set_analysis(). Calling plot_source() with a setting of factor=1 corresponds to the XSPEC plot eufspec, a setting of factor=2 represents eeufspec.

  eufspec: E f(E)
  
  set_analysis("energy", factor=1)
  plot_source()
  
  eeufspec: E^2 f(E)
  
  set_analysis("energy", factor=2)
  plot_source()
  
  eufspec: \lambda f(\lambda)
  
  set_analysis("wave", factor=1)
  plot_source()
  
  eeufspec: \lambda^2 f(\lambda)
  
  set_analysis("wave", factor=2)
  plot_source()
  

• Sherpa now allows the user to define model expressions that apply response matrices, or PSFs, to some models, while not applying the response or the PSF to the rest of the model expression. An example of this kind of model is an expression where a spectral model is defined in energy space, and folded through a response matrix; then, a background model defined in counts, which is not folded through the response, is added to the model expression.

  The new functions, set_full_model() and set_bkg_full_model(), allow users to explicitly define instruments and convolutions that are applied to specified model components.

  Legacy functionality is still supported with set_source() and set_model(); CIAO 4.2 scripts using these functions will continue to work in the current Sherpa.

  Automatic            Manual Definition
  =============        =================
  set_source()         set_full_model()
  set_model()
  
  set_bkg_source()     set_bkg_full_model()
  set_bkg_model()
  

• A new high level UI function add_model() that assigns a user-defined model class as a Sherpa model type. User-defined model classes that inherit from the Sherpa Arithmetic model class or other Sherpa models are accepted.

  from sherpa.models import PowLaw1D
  class MyPowLaw(PowLaw1D): pass
  add_model(MyPowLaw)
  set_model(mypowlaw.p1)
  

• New Sherpa models called scale1d and logparabola. The scale1d model is simply const1d with the integrate flag turned off by default. If a user sets scale1d as a model with a integrated data set, it will behave as a simple constant by default. A 2D version, scale2d, will come next week.

  The logparabola model has the following form

                 /  x \ - p[1] - p[2] * log10(x/p[0])
     f(x) = p[3] |----|
                  \p[0]/
  

• Bug fix: powlaw1d results corrected for gamma very close to 1.

• The atten model ignores the integrate setting.

• Bug fix: the refflag parameter in the XSpec model xsgrad should always be frozen.

• Bug fix: show_bkg_source() was ignoring the bkg_id value.

• Improved error messages during model evaluation when arrays are different sizes.

• XSPEC model parameter bug fixes - flag parameters are now always frozen.

• The Sherpa Model base class now includes startup() and teardown() methods for one-time initialization and cleanup. This is primary used for noticing instrument responses before and after model evaluation, but extends to all derived Model classes including UserModel and user extensions.

  These methods are called once during fitting, confidence, confidence plotting. The methods are called every time with calc_stat().

PSF and Table Models

• The Sherpa table model has been updated to support interpolation of data points on the data set grid from the grid supplied from file. The grids need not be of constant or comparable bin size. If the table model grid is not sorted, Sherpa will sort it in ascending order.

• XSPEC-style table models are now supported using load_table_model(). Additive and multiplicative table models are supported (Atable and Mtable).

  load_table_model("xstbl", "mymod.mod")
  set_model(xstbl)
  

• show_psf() now hides the header information if the PSF is from file.

• PSF and Table model instances now have consistent signatures to their fold() methods. fold() takes a single dataset instance.

• Bug fix to correctly handle rectangular PSFs from file. Rectangular PSF images are also correctly displayed from file in DS9.

• Bug fix: allow simultaneous fitting when a different PSF model is assigned to each data set in a fit.

Parallelization

• Parallel evaluation of proj() and conf() is now automatically turned off on single core machines.

• When Ctrl-C is issued while proj() or conf() are running, Sherpa will now correctly kill off slave processes from any parallel evaluation.

• A new utility function, parallel_map(), is available in sherpa.utils. All usage of the function divide_run_parallel() have been replaced with parallel_map() (except within OptMethods). divide_run_parallel() is to be deprecated.

  This function is a parallelized version of the native Python map which evaluates a function over a list of arguments.

  from sherpa.utils import parallel_map
  parallel_map(lambda x: x*x, [1,2,3]) ->  [1,4,9]
  

Additional Enhancements & Bug Fixes

• There is a new vectorized function 'interpolate' found in the module sherpa.utils.

  from sherpa.utils import interpolate
  yout = interpolate(xout, xin, yin, method='linear'|'nearest')
  

• The energy and photon flux is calculated more efficiently for PHA data sets with multiple responses.

• Error messages for get_arf(), get_rmf(), and get_bkg() have been improved.

• Bug fix: display plot errorbars correctly when statistical error is calculated by the statistic and systematic error is set by the user.

• sample_energy_flux() and the associated functions use covar() instead of proj().

• set_analysis("wave") sets the background_up and background_down units to wavelength for grating PHA data.

• Sherpa now uses a private DS9 window for communication, with id 'sherpa' for session imaging.

• User creation of WCS objects does not require keywords to be of type numpy ndarray.

Analysis Scripts

fit_primini

• The Primini iterative fitting method is included in the CIAO 4.2 Sherpa v2 release. It is no longer necessary to load the sherpa_contrib.primini module.

Documentation

Removal of S-Lang Support in Sherpa

• The Sherpa documentation - including ahelp files and analysis threads - only contain Python-language syntax for the functions.

  The CIAO 4.2 Sherpa v1 documentation is still available online for users who have not yet updated to Sherpa v2.