Synopsis
Introduction to Sherpa, the CIAO modeling and fitting package
Description
Welcome to Sherpa, the CIAO modeling and fitting package. Sherpa enables the user to construct complex models from simple definitions and fit those models to data, using a variety of statistics and optimization methods.
This document provides an introduction to Sherpa; more information is available on the Sherpa website.
Sherpa is designed for use in a variety of modes: as a user-interactive application and in batch mode. The Sherpa command launches an interactive Python session (using the IPython interpreter), in which Sherpa commands (which are Python functions) can be used, as well as Matplotlib commands (to create and modify plots); Crates commands for reading and writing files; as well as other Python functions. It is possible to write Python scripts that import the relevant Sherpa modules, and so run non-interactively.
The Sherpa session can be saved into a platform-independent state file. The session can then be restored at any time.
Starting Sherpa
From the CIAO command line, type:
unix% sherpa [-x|-n|-b|-rcfile|-norcfile] <file>
Any or all of the following options which may be supplied when Sherpa is started:
- -x : launch Sherpa shell in separate display terminal.
- -n : prevents the Sherpa banner from being displayed on startup.
- -b : runs Sherpa in batch mode
- -rcfile : specify a specific .sherpa.rc file to use; refer to "ahelp sherparc" for details.
- -norcfile : do not load any .sherpa.rc file; overrides "-rcfile" if both are set.
- <file>: Python command file to execute
The startup script loads the Sherpa module and the CRATES module, which handles file input and output ("ahelp crates").
The "Starting Sherpa" thread has more details.
The Sherpa Resource File: .sherpa.rc
When Sherpa is started, it processes the $HOME/.sherpa.rc resource file. The resource file defines default behavior for the Sherpa commands and can be customized to the user's preferences; refer to "ahelp sherparc" for details.
Loading Modules
To import the Sherpa, NumPy, and Matplotlib modules in Python without using the "sherpa" startup script:
from sherpa.astro.ui import * from matplotlib import pyplot as plt import numpy as np
It is likely that you will want to include the numpy and matplotlib modules, so they are included in the example above too!
Getting Help
There are several ways to access the Sherpa help files.
Within Sherpa
Within the Sherpa application, the native Python help system can be used:
sherpa> help(<command>)
It is also possible to do a wildcard search for commands, by taking advantage of the native IPython support:
sherpa> plot*? sherpa> *psf?
Alternatively, the ahelp system can be used either directly:
sherpa> ahelp("<command>")
sherpa> ,ahelp <command>or by running the ahelp command-line program:
sherpa> !ahelp <command>
From the CIAO command line
Syntax, description and examples for a specific command:
unix% ahelp <command>
Differences between help and ahelp
For most terms, the ahelp pages are generated from the Python docstrings, and so contain the same information as the help version, but organised slightly differently.
Contributing to Sherpa
Sherpa is developed on GitHub and can be found at https://github.com/sherpa/sherpa. Please consider contributing to Sherpa development - whether it is reporting bugs, providing documentation updates, fixing bugs, or adding new functionality. Using this repository, Sherpa can be installed outside of CIAO, and so used with Python packages that can not be installed into the CIAO environment.
Citing Sherpa
The Astrophysical Journal Supplement Series paper Sherpa: An Open-source Python Fitting Package by Siemiginowska et al. (2024) is a general-purpose paper on Sherpa. The sherpa module contains a citation function which will display the Zenodo record for Sherpa if reference to a particular version of Sherpa is needed. For instance:
sherpa> import sherpa
sherpa> sherpa.citation('4.18.0')Changes in CIAO 4.18
XSPEC
CIAO 4.18 is released with XSPEC 12.14.0k, which is the same as used in CIAO 4.17. Several paramaters have had their defaults changed to match changes made to XSPEC. The get_xsabundances command can now be sent a "table" argument, which will then select that table rather than the default abundance table. The get_xsxset command can now be called with no argumemnt, to return all the values that the user has set, and using an unknown keyword now raises a KeyError rather than returning the empty string.
Optional optimizers
Additional optimizers to "levmar", "neldermead" or "simplex", "moncar", and "gridsearch" are available if either the SciPY or optimagic Python packages are installed into the CIAO environment.
The list_methods call can be used to check what is available: for instance, with both SciPy and optimagic installed (with pip) the response will be:
sherpa> list_methods() ['gridsearch', 'levmar', 'moncar', 'neldermead', 'optimagic', 'scipy_basinhopping', 'scipy_differentialevolution', 'scipy_direct', 'scipy_dualannealing', 'scipy_minimize', 'scipy_shgo', 'simplex']
Penalized CSTAT
The CStat statistic in Sherpa does not deal well with models that can create negative values for some combination of parameter choices and grid the model is evaluated on. This can result in fits getting stuck and not finding a minimum location. CIAO 4.18 adds the "cstatnegativepenalty" statistic that adds a penalty term for any model bin that is negative, scaled by the distance (so the more-negative the bin is the larget the penalty).
Where has the optimizer expored?
The new "record_steps" argument to fit makes it easy to find out the points in the search space that were explored in the fit. The FitResults structure returned by get_fit_results will have a "record_steps" field containing the pre-iteration data from the optimizer, including the statistic and the values of the thawed parameters:
sherpa> fit(record_steps=True) sherpa> res = get_fit_results() sherpa> store = res.record_steps sherpa> plot_trace(store["statval"])
This is similar to the existing "outfile" argument, but without the need to read the data back from a file.
save_all output
The Python file created by the save_all command has seen a number of internal changes - in that, files that are automatically loaded due to header keywords are no-longer explicitly loaded by default. The new "auto_load" parameter can be changed if the old output is preferred. The output should now be usable if one or more PHA2 file were loaded.
PHA files starting at channel 0
The dataspace1d command will now respect the start value when dstype=DataPHA, rather than always starting the channel at 1. The fake_pha command will now work when used with PHA data that starts at channel 0.
Sampling parameter ranges
The covariance matrix is now automatically calculated by the get_draws and eqwidth routines if needed. This means that covar is no-longer needed to be called before calling either of these. This change may lead to some differences for fits with multiple datasets when only a subset of the data is being used, since it was not clear what data was being used for the covariance matrix (this is only an issue if the "covar_matrix" argument was not set).
The normal_sample, t_sample, and uniform_sample routines now use the random state set by set_rng, have gained a "clip" parameter, and the return value has gained a column (indicating whether the parameters were clipped or not), The "sigma" parameter of the normal_sample routine has been renamed to "scale". Note that previously changing "sigma" would not change the output of normal_sample.
Plotting improvements
Most plot commands that take an identifier argument, such as plot_data and plot_fit, can now be sent a list of identifiers. The result will be the different datasets will be displayed in the same plot. To go with this, plot attributes can be set to a sequence of values, matching the per-identifier values. For example
sherpa> plot_data([1, 2], color=["orange", "brown"], alpha=0.5)
will display both datasets 1 and 2 in the same plot, using orange for the first dataset and brown the second. Both will be displayed partially opaque (alpha=0.5 is applied to both).
The plot command mirrors this, and it also accepts an "ids" argument which specifies the plot identifiers to be used for each plot type. Examples include:
sherpa> plot("data", [1, 2], "resid", [1, 2])
sherpa> plot("data", "resid", "ratio", ids=[1, 2], cols=1)Bugs
See the bugs pages on the Sherpa website for an up-to-date listing of known bugs.