Using Python with CIAO

CIAO provides a number of Python modules that support data analysis using CIAO in Python, with the main ones being:

crates: The Python interface to the CIAO Data Model. This module supports reading and writing FITS and ASCII files. The transform module provides support for coordinate conversion (linear and WCS) for tables and images.
Sherpa: The CIAO modeling and fitting package.

More information on Python support in CIAO can be found in "ahelp python", and there is limited customization support as described in users may opt to run their own installation instead; see "ahelp ciaorc".

Note

We strongly suggest that users install Python packages into their CIAO environment, rather than trying to install CIAO into their Python environment.

What Python packages are provided with CIAO?

The Python packages installed with CIAO depends on how CIAO was installed—whether ciao-install or the conda package manager was used—as well as what packages were installed and, for the conda version, the version of Python.

conda

When using conda, the ciaover command lists the "base" CIAO packages that have been installed (in the example below the CIAO CALDB has not been installed):

unix% ciaover -v
The current environment is configured for:
  CIAO        : CIAO 4.17.0 Friday, December 06, 2024
  Contrib     : Package release 0  Tuesday, December 10, 2024
  bindir      : /soft/ciao-4.17/bin
  Python path : /soft/ciao-4.17/bin
  CALDB       : Not installed

System information:
Linux is.awesome.edu 4.18.0-553.44.1.el8_10.x86_64 #1 SMP Thu May 1 10:27:05 EDT 2025 x86_64 x86_64 x86_64 GNU/Linux

A full list of packages can be shown with conda list and pip3 list. The output depends on the OS in use (Linux or macOS), what version of Python was selected and what other packages are installed.

ciao-install

The pip3 tool can be used to report on the Python packages installed in CIAO. For the CIAO 4.17 Linux build this reports:

unix% pip3 list --format=columns | grep -E 'matplotlib|numpy|pip|python'
ipython                   8.30.0
matplotlib                3.9.1
matplotlib-inline         0.1.7
numpy                     1.26.4
pip                       24.3.1
python-dateutil           2.9.0.post0
python-json-logger        2.0.7
types-python-dateutil     2.9.0.20241003

It is likely that you will see a note about pip being out of date, as it changes rapidly!

How do I install a Python package into CIAO?

The method for installing Python packages depends on how CIAO was installed. The recommended approach is to use a conda installation of CIAO.

Updating versions

CIAO depends on a number of libraries and modules (e.g. NumPy), and it has only been tested against the versions available at the time of release. The CXC does not guarantee that CIAO will work correctly if these packages are updated.

Please take care when installing software alongside CIAO as it can result in updating these dependencies.

conda

As you are within a standard conda environment, packages can be installed with conda or pip3.

ciao-install

For users with CIAO installed via ciao-install, packages should be installed with the pip3 tool, and it requires that you have write access to the CIAO installation directory. If this fails you can try installing with the --user flag.

How to Avoid Updating NumPy

Installing Python packages into the CIAO installation area runs the risk of corrupting the CIAO system. The main concern is accidentally upgrading (or downgrading) a component that CIAO relies on (in particular NumPy. It is strongly suggested that you ensure you have the files that ciao-install downloaded when installing CIAO, so that CIAO can be re-installed if there is a problem.

The version of NumPy provided with CIAO 4.17 is version 1.26.4. Installing recent versions of Python packages—such as AstroPy or PyMC3—may try to upgrade the installed NumPy. The main issue is that packages, such as Sherpa, that were compiled against NumPy 1.i.j will be incompatible with NumPy 2.m.n and vice versa. To avoid this scenario, a constraints file may be used with pip3 to ensure that NumPy remains at version 1.26.4.

The constraints file can be generated with the freeze option:

unix% pip list --format=freeze > $ASCDS_INSTALL/constraints.txt
unix% grep numpy $ASCDS_INSTALL/constraints.txt
numpy==1.26.4

or you can just create a file with this—e.g.

unix% echo "numpy==1.26.4" > $ASCDS_INSTALL/constraints.txt

The location of the file is not particularly important, but placing it in $ASCDS_INSTALL means that you have it stored in a memorable location!

This file is then used in a pip3 install line with the -c flag, as will be shown below. Note that not all packages require this set of constraints, but it is hard to know before hand, so it is simplest to always use it.

Example: installing corner

As an example, the following set of routines will install the corner package, which is used to visualize multidimensional samples (in particular the results of Markov Chain Monte Carlo simulations). This package was picked as it is small, Python only, and does not require installing any extra code. Note that the screen output the exact output depends on the version available at the time of installation, and details of the CIAO installation, so may differ slightly from that shown here):

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt corner
Collecting corner
  Downloading corner-2.2.3-py3-none-any.whl.metadata (2.2 kB)
Requirement already satisfied: matplotlib>=2.1 in /soft/ciao-4.17/lib/python3.11/site-packages (from corner) (3.9.1)
Requirement already satisfied: contourpy>=1.0.1 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (1.3.1)
Requirement already satisfied: cycler>=0.10 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (0.12.1)
Requirement already satisfied: fonttools>=4.22.0 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (4.55.2)
Requirement already satisfied: kiwisolver>=1.3.1 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (1.4.7)
Requirement already satisfied: numpy>=1.23 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (1.26.4)
Requirement already satisfied: packaging>=20.0 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (24.2)
Requirement already satisfied: pillow>=8 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (9.4.0)
Requirement already satisfied: pyparsing>=2.3.1 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (3.2.0)
Requirement already satisfied: python-dateutil>=2.7 in /soft/ciao-4.17/lib/python3.11/site-packages (from matplotlib>=2.1->corner) (2.9.0.post0)
Requirement already satisfied: six>=1.5 in /soft/ciao-4.17/lib/python3.11/site-packages (from python-dateutil>=2.7->matplotlib>=2.1->corner) (1.17.0)
Downloading corner-2.2.3-py3-none-any.whl (15 kB)
Installing collected packages: corner
Successfully installed corner-2.2.3

As a final check we see whether we can import the module (the name used in the import statement depends on how the package is designed and need not match the installation name):

unix% python -c 'import corner'

Example: installing AstroPy and SciPy

Following the previous example, we can install AstroPy and SciPy by saying:

unix% pip3 install -c $ASCDS_INSTALL/constraints.txt astropy scipy
Collecting astropy
  Downloading astropy-7.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (10 kB)
Collecting scipy
  Using cached scipy-1.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (61 kB)
Requirement already satisfied: numpy>=1.23.2 in /scratch2/ciao_download_tests/ciao-install-nextgen/ciao-4.17/lib/python3.11/site-packages (from astropy) (1.26.4)
Requirement already satisfied: pyerfa>=2.0.1.1 in ./.local/lib/python3.11/site-packages (from astropy) (2.0.1.5)
Requirement already satisfied: astropy-iers-data>=0.2025.1.31.12.41.4 in ./.local/lib/python3.11/site-packages (from astropy) (0.2025.3.10.0.29.26)
Requirement already satisfied: PyYAML>=6.0.0 in /scratch2/ciao_download_tests/ciao-install-nextgen/ciao-4.17/lib/python3.11/site-packages (from astropy) (6.0.2)
Requirement already satisfied: packaging>=22.0.0 in /scratch2/ciao_download_tests/ciao-install-nextgen/ciao-4.17/lib/python3.11/site-packages (from astropy) (24.2)
Downloading astropy-7.0.1-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (10.3 MB)
   ######################################## 10.3/10.3 MB 47.4 MB/s eta 0:00:00
Using cached scipy-1.15.2-cp311-cp311-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (37.6 MB)
Installing collected packages: scipy, astropy
Successfully installed astropy-7.0.1 scipy-1.15.2

Problems when compiling code

The steps above work when the module is "pure Python", such as the corner package, or needs minimal compilation on the local machine (e.g. AstroPy). There can be problems when installing modules which require access to system-level—or CIAO-provided—code. Please contact the CXC Helpdesk as we may be able to help in these situations.

Have we broken anything?

It is possible to get into a state where incompatible packages have been installed, so it is worth periodically checking your CIAO installation with the following command:

unix% pip3 check
No broken requirements found.

If there is a problem then it may be possible to remove or change the version of the affected package, or packages, or it may require re-installing CIAO.

About Python

Python is a dynamic object-oriented programming language that can be used for many kinds of software development. It offers strong support for integration with other languages and tools and comes with extensive standard libraries. A list of key software features is available on the About Python webpage.

/ ciao / scripting /