Return to Threads Page: Top | All | Intro

Installing CIAO with conda

CIAO 4.16 Science Threads

Overview

Synopsis:

This thread shows how to install CIAO using the conda package manager. Users learn how to perform the basic steps and then can learn about the additional options they can use to customize their installation.

Related Links:

See the Conda FAQs for some common questions about using conda.
Installing CIAO 4.16 Using the ciao-install Script

Last Update: 17 Jul 2024 - Removed references to commercial software vendor.

Quick Start

The following commands are all that is needed to install CIAO using an existing conda installation and verify that everything is working:

$ conda create -n ciao-4.16 -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
  ciao pyciao sherpa ds9 ciao-contrib caldb_main marx python=3.11
$ conda activate ciao-4.16
$ conda env config vars set MARX_ROOT=$CONDA_PREFIX
$ cd $ASCDS_INSTALL
$ bash test/smoke/bin/run_smoke_tests.sh

CIAO 4.16 requires using the conda-forge channel which provides a newer version of libgfortran which is required for XSpec.

The rest of this thread discusses these steps in more detail and presents the user with additional options.

Detailed Instructions

Introduction to conda

CIAO 4.16 can be installed using conda.

conda is a package management and environment management system. It is similar to other package managers (such as yum, apt, macports, dnf) but allows for users to create multiple environments with multiple versions of packages which can co-exist on the same system. Users can easily install pre-compiled and pre-configured packages from curated channels (repositories) into various environments. Users then activate the environment to run the applications.

With conda , Linux and Mac users can choose to install CIAO 4.16 using either Python 3.11, 3.10, or 3.9. The default version is subject to change based on the availability of other packages on the conda-forge channel. Using the conda package manager users can also install additional third-party packages such as astropy, scipy, and jupyter notebook.

Install conda

Users must already have conda installed on their systems. The conda package management software is available from several different sources. It is the users' responsibility to read and adhere to the License and Terms of Service for any and all third party software installed. The miniforge conda package is compatible with CIAO.CIAO 4.16 is available for Linux and MacOSX (both Intel/x86 and ARM/M1/M2/M3). CIAO does not work with Windows, not even when running Windows Subsystem for Linux.

Minimum conda version?

Some users, macOSX in particular, have reported problems with using old versions of conda. CIAO has successfully be installed using conda 4.7.6 and newer. Users should consider upgrading their base conda installation before installing CIAO.

Users can check if they have conda and the version using

$ conda --version
conda: Command not found.

In this case, conda is not installed or otherwise not setup in the user's PATH.

Help installing conda

The CXC does not provide support for installing conda. Users experiencing problems installing conda should contact they support system for the version of conda they are trying to install.

Below are the basic instructions for installing miniforge.

Linux$ curl -L -o get_conda.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-Linux-x86_64.sh
  -or- 
macOS-intel/x86$ curl -L -o get_conda.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-x86_64.sh
  -or-
macOS-arm/m1$ curl -L -o get_conda.sh https://github.com/conda-forge/miniforge/releases/latest/download/Miniforge3-MacOSX-arm64.sh

Users can then run the interactive installation script

$ bash get_conda.sh 

... accept license and answer prompts ... 

installation finished.
Do you wish to update your shell profile to automatically initialize conda?
This will activate conda on startup and change the command prompt when activated.
If you'd prefer that conda's base environment not be activated on startup,
   run the following command when conda is activated:

conda config --set auto_activate_base false

You can undo this by running `conda init --reverse $SHELL`? [yes|no]
[no] >>> no

You have chosen to not have conda modify your shell scripts at all.
To activate conda's base environment in your current shell session:

eval "$(/install_dir/bin/conda shell.YOUR_SHELL_NAME hook)" 

To install conda's shell functions for easier access, first activate, then:

conda init

Thank you for installing Miniforge3!

After the installer has finished users will want to be sure that the conda command is setup in their PATH environment.

bash$ PATH="${PATH}:<<directory where conda was installed>>/bin"

or

tcsh% set path = ( ${path} <<directory where conda was installed>>/bin )

To add the necessary setup commands to your startup files run the following command:

$ conda init

Users need to exit the terminal or source the startup script for the changes to take effect.

Install CIAO into a new conda environment

The CXC is hosting the CIAO 4.16 packages on our own conda channel. This is due to the size of the packages.

To create a new conda environment with CIAO installed users run the following command:

$ conda create -n ciao-4.16 -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge ciao pyciao sherpa ds9 ciao-contrib caldb_main marx python=3.11

This will create an environment named ciao-4.16 and perform a standard CIAO installation including the standard CALDB.

Users can choose to install CIAO with Python 3.11, 3.10 , or 3.9.

Tip

Timeouts during download are frequent. Users are encouraged to simply try again.

Blank sky background files

To install the ACIS and HRC blank sky background files users should replace caldb_main with caldb.

$ conda create -n ciao-4.16 \
  -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
  ciao pyciao sherpa ds9 ciao-contrib caldb marx python=3.11

Problem with downloading XSPEC models or CALDB

Users experiencing problems downloading the CALDB files and/or the XSPEC model files can follow the Alternative download instructions.

Alternative: fixing the installed package versions to avoid possible conflicts

The default conda installation instructions above will select the latest version of the packages, including the packages that CIAO relies on, such as NumPy and matplotlib. As new versions of these packages are made available, it is possible for their to be problems. An alternative approach is to install a fixed set of versions, which fits under the umbrella of "reproducible research", and can be performed with conda specification (aka "spec") files.

Users can select a fixed set of packages using one of the two commands below - the difference is whether the CALDB packages (it selects the caldb_main packages) are included or not:

Linux

$ conda create -n ciao-4.16 --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/ciao-Linux.txt 
  -or-
$ conda create -n ciao-4.16 --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/ciao-Linux.txt \
    --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/caldb-spec.txt

macOS Intel/x86_64

$ conda create -n ciao-4.16 --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/ciao-macOS.txt 
  -or-
$ conda create -n ciao-4.16 --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/ciao-macOS.txt \
    --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/caldb-spec.txt

macOS ARM/M1/M2/M3

$ conda create -n ciao-4.16 --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/ciao-macOS-ARM.txt 
  -or-
$ conda create -n ciao-4.16 --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/ciao-macOS-ARM.txt \
    --file https://cxc.cfa.harvard.edu/ciao-install/ciao-4.16/env_files/caldb-spec.txt

Adding extra CALDB packages

Users can then later include the blank sky background files with:

$ conda install -n ciao-4.16 -c https://cxc.cfa.harvard.edu/conda/ciao/ \
  -c conda-forge acis_bkg_evt hrc_bkg_evt

Activate CIAO environment

Once the CIAO environment is created, it has to be activated. This is equivalent to source'ing the CIAO setup scripts.

$ conda activate ciao-4.16
  -or-
$ source activate ciao-4.16

where ciao-4.16 is the name of the environment you just created. Whether you use conda or source is dependent on how conda was install and configured.

By default, users will notice the shell prompt has changed to indicate the current conda environment.

$ source activate ciao-4.16
(ciao-4.16) $

MARX setup

If you have installed marx and are going to be using the simulate_psf script, the you should setup the MARX_ROOT environment variable in your conda environment

$ conda env config vars set MARX_ROOT=$CONDA_PREFIX

You should deactivate and then activate the CIAO environment to pickup the new environment variable. Then check it is set using plist

$ plist simulate_psf

Parameters for /export/miniconda/envs/ciao-4.16/param/simulate_psf.par

...
    (marx_root = ${MARX_ROOT} -> /export/miniconda/envs/ciao-4.16) Directory where MARX is installed

If you plan to run marx by itself, then you will need to copy the parameter file from the installation directory: $CONDA_PREFIX/share/marx/pfiles/

Run the smoke tests

Users are encouraged to run the CIAO smoke tests to ensure everything is installed and working correctly.

(ciao-4.16)$ cd $ASCDS_INSTALL/
(ciao-4.16)$ bash test/smoke/bin/run_smoke_tests.sh

If users choose to omit packages (eg sherpa or ds9), then smoke tests associated with those packages will fail.

Install additional 3rd party packages

Additional packages can then be installed into the CIAO environment using the conda installer.

(ciao-4.16)$ conda install -n ciao-4.16 -c conda-forge jupyter astropy scipy

Due to the constraints on python version and numpy version, users may not be able to install certain packages, or possibly the latest versions of certain packages such as astropy or scipy. Similarly, users may not be able to install CIAO into an existing environment with incompatible versions of python or numpy.

If installing additional CIAO packages, users need to remember to include the CXC conda channel. For example to add the ACIS blank sky background files to an existing CIAO installation the command would look like:

(ciao-4.16)$ conda install -n ciao-4.16 -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge acis_bkg_evt

Users can also install packages using pip or other package managers

(ciao-4.16)$ pip3 install bash_kernel

Leave CIAO environment

Use the deactivate command to leave the conda environment:

(ciao-4.16)$ conda deactivate

or simply exit the terminal window. deactivate can be useful when users need to install additional package which may conflict with the CIAO environment or when updating CIAO components.

Updating CIAO environment

Install CIAO 4.16 in a new environment

CIAO 4.16 requires using the conda-forge channel to get a version of libgfortran that is compatible with XSpec 12.12. Because of this, users are encourage to create a new conda environment for CIAO 4.16 rather than attempting to update an existing environment. This section is provided as a reference for updating CIAO 4.16 components, such as a new CALDB release, a new contributed scripts release, or a CIAO 4.16 patch.

Users can install the latest CIAO and CALDB updates using the conda update command

$ conda update -n ciao-4.16 \
  -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
  ciao pyciao sherpa ds9 ciao-contrib caldb_main marx --freeze-installed

Only the packages listed will be updated. To update all the packages in the environment use the --all option. We recommend using the --freeze-installed flag to prevent other packages such as numpy from updating which may could potentially cause a conflict with CIAO.

After updating, users may want to run the conda clean command to completely remove the previous versions. This is especially true when updating the CALDB since each CALDB update is a full release which each requires 5-12Gb of disk space.

$ conda clean --all

Alternative Download Options

Users on slow internet or wireless connections may experience problems downloading the large data files associated with the Chandra CALDB and the XSPEC models. Often users will be able to simply retry again successfully; however, sometimes the server timeouts are persistent.

CALDB alternatives

Partial CALDB download

Users who only need to analyze a limited number of Chandra datasets (OBS_IDs), can use the download_obsid_caldb script to download only the calibrations files associated with individual observations (including optional blank sky background files).

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge \
  ciao pyciao sherpa ciao-contrib ds9 python=3.11
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% download_chandra_obsid 4425
...
(ciao_no_caldb) unix% download_obsid_caldb 4425
Output CALDB directory (${ASCDS_INSTALL}/CALDB -> /soft/miniconda/envs/ciao_sans_caldb/CALDB): 
download_obsid_caldb
          infile = 4425
          outdir = /soft/miniconda/envs/ciao_sans_caldb/CALDB
      background = no
         missing = no
         clobber = no
         verbose = 1
            mode = ql

Retrieving files for CALDB_VER = 4.9.6
Retrieving CALDB index files
Processing infile=4425/primary/acisf04425N004_evt2.fits.gz
Retrieving CALDB data files
    Filename:                               0------------------1
    telD1999-07-23geomN0006.fits            ####################
    telD1999-07-23aimptsN0002.fits          ####################
    telD1999-07-23tdetN0001.fits            ####################
    telD1999-07-23skyN0002.fits             ####################
    telD1999-07-23sgeomN0001.fits           ####################
    hrmaD1996-12-20axeffaN0008.fits         ####################
    hrmaD1996-12-20vignetN0003.fits         ####################
    acisD1997-04-17qeN0006.fits             ####################
    acisD2002-02-01qeuN0007.fits            ####################
    acisD2000-11-28badpixN0004.fits         ####################
    acisD1999-08-13contamN0014.fits         ####################
    acisD1996-11-01gradeN0004.fits          ####################
    acisD2000-01-29grdimgN0001.fits         ####################
    acisD2000-01-29gain_ctiN0008.fits       ####################
    acisD1996-11-01evtspltN0002.fits        ####################
    acisD2002-08-01ctiN0009.fits            ####################
    acisD2003-02-01t_gainN0008.fits         ####################
    acisD1999-07-22subpixN0001.fits         ####################
    acisD2000-01-29fef_pha_ctiN0004.fits    ####################
    acisD1999-09-16dead_areaN0001.fits      ####################
    hrmaD1996-12-20reefN0001.fits           ####################
    acisD2000-01-29p2_respN0008.fits        ####################
    hrmaD1996-11-01wpsfN0001.fits           ####################
    acisD2000-01-29osip_ctiN0006.fits       ####################
    acisD2002-11-15gtilimN0004.fits         ####################

Be sure to source the new setup scripts to
use these CALDB files.

(t)csh:
source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.unix

bash:
source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.sh

(ciao_no_caldb) unix% source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.unix

Setup CALDB environment variables

To avoid needing to setup these environment variable each time the conda environment is activated, users can permanently set them using the conda env config command:

$ source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.sh
$ conda env config vars set CALDB=$CALDB
$ conda env config vars set CALDBCONFIG=$CALDBCONFIG
$ conda env config vars set CALDBALIAS=$CALDBALIAS

This only needs to be done once. Users will need to deactivate and then activate their CIAO environment to pick up the new environment variables.

The partial CALDB can be installed anywhere; the default is the same location as the full CALDB. When run again, the script will skip any CALDB files that have already been downloaded and will only download the missing files.

Install separately using the ciao-install tar files

Similar to above, users can download the entire CALDB and install it manually. We have a copy of the CALDB files in our Google Drive in the CALDB folder. The Google Drive downloads are less likely to time out.

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge ciao pyciao sherpa ciao-contrib ds9
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% cd $ASCDS_INSTALL
(ciao_no_caldb) unix% mkdir CALDB
(ciao_no_caldb) unix% cd CALDB
(ciao_no_caldb) unix% tar xfz ~/Downloads/caldb_4.9.6_main.tar.gz
(ciao_no_caldb) unix% tar xfz ~/Downloads/acis_bkgrnd_4.9.2.1.tar.gz
(ciao_no_caldb) unix% tar xfz ~/Downloads/hrc_bkgrnd_4.7.7.tar.gz

  -- For bash/dash/zsh users
(ciao_no_caldb) unix% export CALDB=$ASCDS_INSTALL/CALDB
(ciao_no_caldb) unix% export CALDBCONFIG=${CALDB}/software/tools/caldb.config
(ciao_no_caldb) unix% export CALDBALIAS=${CALDB}/software/tools/alias_config.fits

  -- For tcsh users
(ciao_no_caldb) unix% setenv CALDB $ASCDS_INSTALL/CALDB
(ciao_no_caldb) unix% setenv CALDBCONFIG ${CALDB}/software/tools/caldb.config
(ciao_no_caldb) unix% setenv CALDBALIAS ${CALDB}/software/tools/alias_config.fits

Setup CALDB environment variables

To avoid needing to setup these environment variable each time the conda environment is activated, users can permanently set them using the conda env config command:

$ source /soft/miniconda/envs/ciao_sans_caldb/CALDB/software/tools/caldbinit.sh
$ conda env config vars set CALDB=$CALDB
$ conda env config vars set CALDBCONFIG=$CALDBCONFIG
$ conda env config vars set CALDBALIAS=$CALDBALIAS

This only needs to be done once. Users will need to deactivate and then activate their CIAO environment to pick up the new environment variables.

Install individual conda tar files

Users can also install the individual conda tar files from our Google Drive in the conda/ciao/noarch folder. The Google Drive downloads are less likely to time out.

unix% conda create -n ciao_no_caldb -c https://cxc.cfa.harvard.edu/conda/ciao \
  -c conda-forge ciao pyciao sherpa ciao-contrib ds9
...
unix% conda activate ciao_no_caldb
(ciao_no_caldb) unix% conda install ~/Downloads/caldb_main-4.9.6-0.tar.bz2
(ciao_no_caldb) unix% conda install ~/Downloads/acis_bkg_evt-4.9.2.1-1.tar.bz2   (optional)
(ciao_no_caldb) unix% conda install ~/Downloads/hrc_bkg_evt-4.7.7.1-1.tar.bz2    (optional)

The advantage of this method is that it will also install the activation scripts; however, when updates are requested conda may try to download the entire CALDB files again.

Install from a local channel

Users who have downloaded the conda tar file from the Google Drive (see above) can also setup a local conda repository and install from there. The advantage is that installing using the individual tar files does not check for dependencies; whereas installing from a local repository does resolve dependencies.

unix% conda install conda-build
...
unix% mkdir -p /tmp/my_conda_dir/noarch
unix% cp ~/Downloads/caldb_main-4.9.6-0.tar.bz2 /tmp/my_conda_dir/noarch
unix% cp ~/Downloads/acis_bkg_evt-4.9.2.1-1.tar.bz2 /tmp/my_conda_dir/noarch   (optional)
unix% cp ~/Downloads/hrc_bkg_evt-4.7.7.1-1.tar.bz2 /tmp/my_conda_dir/noarch    (optional)

unix% conda index /tmp/my_conda_dir

unix% conda create -n ciao_my_caldb -c /tmp/my_conda_dir/ \
  -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge ciao pyciao sherpa ciao-contrib ds9 caldb_main
...
  ca-certificates    conda-forge/linux-64::ca-certificates-2021.10.8-ha878542_0
  caldb_main         my_conda_dir/noarch::caldb_main-4.9.6-0
  cfitsio            ciao/linux-64::cfitsio-4.0.0-h9bf148f_2
...
unix% conda activate ciao_my_caldb

XSPEC alternatives

Given the platform specific dependencies, the only alternative method to download the sherpa XSPEC model files is to download the correct tar file from our Google Drive and install it using the local repository method described above

unix% conda install conda-build
...

 -- For Linux
unix% mkdir -p /tmp/my_conda_dir/linux-64
unix% cp ~/Downloads/xspec-modelsonly-12.12.0-h6f05748_1.tar.bz2 /tmp/my_conda_dir/linux-64

 -- For macOS
unix% mkdir -p /tmp/my_conda_dir/osx-64
unix% cp ~/Downloads/xspec-modelsonly-12.12.0-hd4aef4f_1.tar.bz2 /tmp/my_conda_dir/osx-64


unix% conda index /tmp/my_conda_dir

unix% conda create -n ciao_my_xspec -c /tmp/my_conda_dir/ -c https://cxc.cfa.harvard.edu/conda/ciao -c conda-forge ciao pyciao sherpa ciao-contrib ds9 caldb_main
...
  xerces-c           conda-forge/linux-64::xerces-c-3.2.3-hfe33f54_1
  xpa                ciao/linux-64::xpa-2.1.20-h9bf148f_0
  xspec-modelsonly   my_conda_dir/linux-64::xspec-modelsonly-12.12.0-h6f05748_1
  xz                 conda-forge/linux-64::xz-5.2.5-h516909a_1
  zlib               conda-forge/linux-64::zlib-1.2.11-h36c2ea0_1013
...
unix% conda activate ciao_my_xspec

Caveats with Alternative Downloads

Please keep the following caveats in mind

Using a local channel will list the package's channel (when running "conda list") as the directory location
Using an explicit package will list the package's channel (when running "conda list") as "unknown"
If you later try to install from the official channel, you will get messages of the package getting superseded by a higher-priority channel.

Known Problems

Users should be aware of these known issues before using the conda ciao installation.

The conda installation will seem to hang during the installation process when retrieving the largest segments: the XSPEC models and CALDB files. Do not kill the installation. If it times out then it will be able resume and only install the failed packages.
Users who keep the CALDB separate from the CIAO installation must setup the following environment variables independently from the conda setup.
```
$ export CALDB=/Some/Other/Directory/CALDB
$ export CALDBCONFIG=${CALDB}/software/tools/caldb.config
$ export CALDBALIAS=${CALDB}/software/tools/alias_config.fits
```
These can be setup in the users' login script ($HOME/.bash_profile), or setup each time the environment is activated. They can be setup either before or after the conda environment is activated.

The CALDB version will not be reported by the ciaover command if it is not installed by conda
Due to the constraints on python version and numpy version, users may not be able to install CIAO in the same environment as existing packages. Known incompatibilities include:
- CIAO and StSCI's astroconda which fail due to version incompatibilities in the CFITSIO library.
- There is a conflict between CIAO and HEASARC's XSPEC package. Users must
  1. First, initialize their HEASARC environment
```
$ export HEADAS=/soft/lheasoft/headas/x86_64-pc-linux 
$ source $HEADAS/headas-init.sh
```
  2. Then activate their CIAO environment
```
$ conda activate ciao-4.16
```
  3. Finally, users have to restore the HEADAS environment variable back to it's original value
```
$ export HEADAS=/soft/lheasoft/headas/x86_64-pc-linux 
```
  Note: this may cause problems using XSpec models in sherpa. Users must not source the headas-init.sh script after activating the CIAO environment as it will cause incompatibilities in the suite of parameters tools: pset, pget, and plist.
Users must not activate CIAO conda environment in the same terminal (shell) where the classic CIAO setup script (eg ciao.sh) has already been sourced. Similarly, users must not source the CIAO setup scripts in an activate CIAO conda environment. The results in both cases are corrupted environments which can lead to unexpected failures.

History

12 Nov 2020	Moved detailed conda instructions into own thread.
01 Mar 2021	Added new section on Alternative download options.
07 Mar 2021	Added some FAQs
08 Dec 2021	CIAO 4.14 updates: CIAO now requires using conda-forge channel.
10 Jan 2022	Added MARX information. Also added info about setting CALDB environment variables.
13 Apr 2022	The default version of python is subject to change without notice based on the availability of other packages on the conda-forge channel.
27 Jan 2023	Updated for CIAO 4.15.1 patch.
07 Mar 2023	Tweak for force python 3.10. Added CONDA_SUBDIR info.
30 Nov 2023	Updated for CIAO 4.16. ARM native support.
17 Jul 2024	Removed references to commercial software vendor.