CIAO 2.0 Release Notes
Contents:
- Tools
- GUIs
- Sherpa
- DataModel
- Other Libraries
Tools
* acis_detect_afterglow * (*new*) Summary identify and flag anaomlies where residual charges on the ccd detectors appear as events in consecutive exposures in acis event files, aka "Flaring Pixels". It sets specific bits in the status column to indicate whether or not an event is considered to most probably be valid or a flaring pixel. The criteria used in determining this status are user configurable and include- chip position (required), fltgrade degradation (optional), and pulse height amplitude [PHA] degradation (optional). Known Bugs - Input event list must already have a 32bit status word. Re-run acis_process_events on file if needed. * acis_process_events * -- Produce/update ACIS coordinates, PI, GRADE Interface & Algorithm Changes - added support of sky_1d computations for continuous clocking data - bad pixel check moved from acis_format_events to acis_process_events - added additional header keywords- BPIXFILE, GRADESYS, NOMCHIPY, FSW_VER S - calibration database mods - status column changed from 16 bit to 32 bit to support new bits such as flaring pixels - no longer read instrume parameter file to set chipsys,tdetsys,fpsys - wcs for det coords changed from TAN to TAN- P - aspect correction code modified to use any of the following files- asol, aoff, or soff. - ra_targ, dec_targ, and roll_nom now read from obsfile - Changed default event defintions to be consistent with reprocessed data including: floating point cooridate are now stored in single percision and the status column expanded to 32bits. - Removed obsolete qp* parameters Bug Fixes - pi bins now start at 1 instead of zero - subspaces (gti extensions) maintained when rerunning acis_process_events on a level 1 event filt - write out tlmin/max for grade and node_id columns * aconvolve * -- Convolve an N-dimensional image with a kernel * acrosscorr * (*beta*) * ahelp * * apowerspectrum * (*beta*) * asp_apply_sim * -- Add the SIM offsets to the optical axis offset Bug Fixes - fix the problem with the sign of ROLL_OFFSETS being opposite * asp_calc_offsets * -- Apply offsets to L1 aspect solution Interface & Algorithm Changes - Full parameter history added to header Known Bugs - Given a stack of files with a blank line at the end of the stack will result in an output file that does not close properly and cannot be used. Users can check their stack's with the"stk_build" tool and remove any trailing blank lines. * asphist * -- Bin the aspect into a 3D histogram Interface & Algorithm Changes - create an empty file when no gti is found - rearrange the parameters and read all of them in at once Bug Fixes - fix the problem with bins off by 1 res. element - fix SEGV problem if GTI extname is not specified in 'gtifile' * axbary * (*new*) Summary: applies barycenter corrections to all times using the orbit ephemeris file orbitFile, ra, dec, and refFrame "FK5" (DE200) or "ICRS" (default; DE405). * celldetect * -- Uses a "sliding cell" to search for sources Interface & Algorithm Changes - Now accepts an exposure map (parameter "expstk"). - New parameter "ellsigma" controls size of ascii ellipse regions. - New parameter "regfile" specifies name of ascii regions file. -New source output column "expo_ratio" gives ratio of source's detect region exposure to background region exposure. - New parameter "expratio" suppresses ascii regions of sources whose expo_ratio is below specified value. - Slightly different binning for recursive blocking - source positions will be slightly different. - Source list has columns suitable for DM spatial filtering (COMPONENT, SRC_POS, ROTANG, etc). - Source SNR test is now SNR > = threshold instead of SNR > threshold. - Source position columns are now double precision. - "stall" parameter has been deleted. - Default encircled energy is now .9 (which is actually .8) Bug Fixes - A bug which would cause celldetect to fail on HRC-S was fixed. - Bug in net_counts_error and bkg_counts_error has been fixed. - Bug in cell sizes for HRC-S fixed. - Fixed bug in source list FITS Header. * csmooth * -- Adaptively smooth a 2-D image * dmappend * -- Append multiple blocks/extension Bug Fixes - add messages to remind users appending different extensions - negative signifigances can be reported when the maximum smoothing scale is reached. * dmarfadd * -- Add multiple ARF files together weighting by exposure - Interface & Algorithm Changes The tool now allows an arbitrary number of ARF to be added. The former restriction of 8 files is removed. - Minor changes to the documentation to clarify what happens in particular situations. - allow a non grating file to go through without requiring the TG_(M,PART,SCRID) keywords. * dmcontour * (*new*) Known Bugs - The region file created by dmcontour only works as a filter for simple regions. If the region has "holes" then the filter created will not work. Users can display the contours with ds9 to ensure that they have a "simple" region. * dmcoords * (*new*) * dmcopy * -- Filter and bin tables and images * dmextract * -- Make a histogram table file * dmgroup * (*new*) Summary: Tool to group a specified column in a table, with various options: number of bins, number of counts, SNR selection, or an adaptive approach. Known Bugs - Will have problems when the number of bins exceeds the number of rows in the file. This will typically not be a problem for users working in 'channel' space. - the grp_stat_err column is incorrect when the following grouping options are used: NUM_CTS, NUM_BINS, and SNR. * dmgti * -- create a Good Time Interval (mtl_build_gti) Interface & Algorithm Changes - added support for ulong/ushort. - added status column to identify which limits were exceeded. - added second extension with limits used. Bug Fixes - detected/fixed bug in handling of last row. Known Bugs - Spaces on the command line may not get interpreted correctly. Try allowing the tool to prompt. - Error messages will be reported for 'mtl_build_gti'. * dmhedit * -- edit file headers Infrastrucutre & Algorithm Changes - Added parameters 'unit' and 'comment' for use on the command line (in addition to the previous functionality in which the unit and comment were included within the value ) - Removed 'kernel' parameter - Made more robust regarding whitespace within a filelist . - Type and bounds checking when updating a keyword. Bug Fixes - Bug fix -file access checking enhanced. - Bug fix - keywords containing characters and digits could not be added. - any *new* numerical value keyword will get stored as a floating point number. Existing keywords will retain their datatype. - "/"'s in the "value" parameter will be treated as separators for the comment, so things like full path names cannot be written. * dmimg2jpg * (*new*) Summary: Generates a JPEG and an optional encapsulated level 1 postscript file from a FITS file. The tool can read in CIAO region files and place overlays in either an overlapping mode or a merged region mode. Grid and coordinate markings can be added as well as source labeling with numbers. The font marking is primitive, but allows for quick looks at the image on the command line. Given three inputs, dmimg2jpg can also create a three color image. Known Bugs - Printing the postscript images is known to take a long time. * dmimgcalc * -- perform arithmetic on images Known Bugs - Needs to have exact numerical match to preserve WCS. There is a logical match of WCS from the output of dmcopy and mkexpmap, but the exact values don't match, so the WCS is lost. * dmimghist * (*new*) Known Bugs - puts all pixels above max and below min into max and min bins respectively. * dmkeypar * (*new*) Used to retrieve header keyword information. When the user supplies a filename and a keyword, dmkeypar will write the value, datatype, unit, and comment to its parameter file. COMMENT and HISTORY are invalid keywords. * dmlist * -- list contents or structure of a file * dmmakepar * -- Write header keywords to a parameter file Interface & Algorithm Changes - The verbose parameter has changed to an integer value between 0 and 5. * dmmakereg * (*new*) * dmmerge * -- Merge two or more compatible tables into one * dmpaste * (*new*) Known Bugs - will not report an error if column with same name already exists, and will not paste the new column. * dmreadpar * -- Add parameters to a file header Interface & Algorithm Changes - The verbose parameter has changed to an integer value between 0 and 5 * dmrega2fits * -- Convert a grating ASCII region into a FITS format * dmregrid * -- rebin a stack of 2 dimensional images Interface & Algorithm Changes - Speed enhancements for both the 'exact' and 'approximate' algorithms. Bug Fixes - updating the previous 'exact' algorithm also fixed a bug which occasionally caused a segmentation fault for certain combinations of parameters . - if the binning specification is given by a stack in which the second (or later) item results in a greater number of output pixels than the first binning specification, a segmentation fault resulted. The tool now uses the number of pixels resulting from binning the first image, and gives a warning message. * dmsort * -- Limited capability table sort Known Bugs - should only be used in the defined threads. Its is known to have serious problems (and serious limitations) in general analysis sessions. * dmstat * -- Computes common statistics for table and images * dmtcalc * (*new*) Summary: Very powerful table calculator. Works with all data types, array and vector columns, as well as a number of operations and functions. It creates a new file from a stack of input files, creating and populating any new columns that the user defines in the series of arbitrarily complex arithmetic expressions input via the expression parameter. Supported operations include arithmetic, exponentiation, boolean, and smoothing. Supported functions include those in the standard math library and conditional expressions. Known Bugs - Currently will create a TLMIN/MAX == 0 if the input columns do not already have mins and maxes attached to them. * dmtype2split * (*new*) Summary: Create a type 1-like output file for each row specified of a type 2-like input file. Each of the specified rows in the input file is copied to a separate output file. If the input file is a pha file, keywords required by the OGIP format are also written; otherwise, generic keywords are written. For each scalar column in the input, the coordinate keyword information is written, and the datatype, value, and unit are written as a keyword. For each input column that is an array, a column is created from the datatype, value, and unit; the row of input data is read and written to the output column. * dmwritefef * -- create a fits embedded function file * get_src_region * (*new*) Summary: determines a background threshold value and then output regions that have counts higher than the background threshold value. Known Bugs - The tool can only use the x, y physical coordinates of the source regions. So if the input is an event file, it must have the x and y columns, and the wcs infomation in the header. * hrc_process_events * -- Correct HRC event positions, times, PHA, etc Interface & Algorithm Changes - added support for float data type for det and sky columns - updated TLMIN/TLMAX for sky and det columns - add the tap ring correction - added event "flatness" test - added event saturation test - added event "hyperbolic" test - display the badpixfile warning before processing events. - Keep the amplitudes as 'double' for all corrections - Saturation coeffs are a function of amp_sf - set pi=pha if no gain map is used. - bugfix to the gainmap off by one - remove the hidden parameters (chipsys, tdetsys, fpsys) from the code and use the pixlib calls to set these values for different detector. * lightcurve * -- Bin source and background events by specified time Bug Fixes - a non-time-sorted file will not break the tool as it does before. The tool will give out warning and list the row numbers of the data that are not in time order. * mkarf * -- Generate an ARF for Chandra images Interface & Algorithm Changes - Support for ACIS Windows (maskfile) - clobber, maskfile parameters added. - Energy grid specified via `engrid' parameter. This gives the user more flexibility in choosing an energy grid. Bug Fixes - avoid coredump if no bad-pixel file was found * mkexpmap * -- Generate a Chandra imaging effective exposure map Interface & Algorithm Changes - The exposure map be normalized by the exposure time to produce a cm^2 map, or left unnormalized (cm^2-sec). - gridding specification changed to match other tools (engrid) - clobber and normalize parameters added. Bug Fixes - sign error involving RA CDELTA corrected. - axis-group name changed from XXX to SKY. * mkgarf * -- Generate a Chandra Grating ARF for one detector element Interface & Algorithm Changes - New parameters: clobber, osipfile, maskfile - Energy grid specified via `engrid' parameter. This gives the user more flexibility in choosing an energy grid. - Support for ACIS Windows (maskfile) - Support for order sorting tables (osipfile) Bug Fixes - typo affecting the grating ARF for HRC fixed * mkinstmap * -- Generate a Chandra instrument map Interface & Algorithm Changes - Support for ACIS Windows (maskfile) - New parameters: clobber, osipfile, maskfile - Support zeroth order grating instrument maps (grating) Bug Fixes - avoid coredump if no bad-pixel file was found * mkoif * - Create an observation index file (oif.fits) used by the /Firstlook/ GUI. Known Bugs - does not work on Linux platforms. Users can only use the observation index file that came with their data distribution. * mkpsf * -- Extract a point spread function * mkrmf * -- create response matrix file * punlearn * -- Resets tool parameters to their default values Known Bugs - reports of problems when running on parameter files with blank lines, eg ardlib.par. If needed, run paccess to determine the path to the parameter file and then just remove the file. * reproject_events * (*new*) * stk_build * (*new*) Used to build a stack from a string, a file, or a combination of the two. Output is to stdout, stderr, or a file. Useful for scripts. * stk_count * (*new*) Used to count the number of elements in a stack. Output is to the stk_count parameter file. The stack can be defined by a string, a file, or a combination of the two. * stk_read_num * (*new*) Used to read the N'th element of a stack. Output is to the stk_read_num parameter file. The stack can be defined by a string, a file, or a combination of the two. * tg_create_mask * -- Create a region file to define spectrum sky boundaries Interface & Algorithm Changes - The value of the ROLL_PNT header keyword instead of the value of the ROLL_NOM header keyword will be used to calculate the angle of the grating mask. If the ROLL_PNT header keyword is absent, the tool will give out a warning message, and the value of the ROLL_NOM header keyword will be used. - The value of the DEFOCUS header keyowrd instead of the value of the SIM_X header keyword will be used to calculate the radius of the mask of zeroth-order source. If the DEFOCUS header keyword is absent, the tool will give out a warning message, and the value of the SIM_X header keyword will be used. - The LESF (Drake Flat) parameters (s1_drk1_x, s1_drk1_y, s1_width_drk1, s1_length_drk1, s1_angle_drk1....) are removed, since they are not used by the tool. - The default value for the input_psf_tab parameter is now set to CALDB. Bug Fixes - Fixed a bug that occured when the input data have the LETG and CC MODE combination. - Fixed a bug that occured when an observation has a small roll angle. * tg_resolve_events * -- Assign events to spectral orders Interface & Algorithm Changes - Can accept a new type of response matrix (rm) file. The new rm file's energy data are spatially dependent (i.e depends on chip coordinates). Also, the new rm file's energy limits are given absolutely, instead of as differences from the peak. The old type of rm file can still be used. - The wcs information for detector coordinates are written out using a TAN-P transform instead of a TAN wcs transform. - Out put a new column, Sky_1D, if the data are ACIS continuous clocking data. - Added support to propagate TLMIN/TLMAX from input evt file. - Centralized usage of constants to use DS lib value. - The acaofffile and alignmentfile parameters have been changed from hidden type to auto type parameters. - The default value for the rmfile parameter is now set to CALDB. This means that the tool will look at the header of the input data and then use the appropriate response matrix file to process the data. - The qp_internals, qp_pagesize, and qp_bucketlen parameters have been removed. - The stdlev1 parameter has been replaced by two new parameters (stdlev1_ACIS and stdlev1_HRC). stdlev1_ACIS holds the ACIS event format definition string, while stdlev1_HRC holds the HRC event format definition string. There are significant changes in the default event format definition string. For example, the data type of tg_r, tg_d, det, sky x, sky y, tg_m, tg_lam, tg_mlam have been changed from double to float. - The default value of the eventdef parameter has been changed from ")stdlev1" to ")stdlev1_ACIS". Bug Fixes - Bug fix for ACIS Graded data (CORN_PHA column not getting propagated). Known Bugs - The output of tg_resolve_events doesn't have all the ONTIMEn & related header keywords. * tgdetect * -- Locate 0-order image (tgmatchsrc and tgidselectsrc) Interface & Algorithm Changes - Updated with new celldetect parameters. - tgmatchsrc max_separation parameter changed from 1 to 3 arcsec. Known Bugs - No filter can be applied to the input or reference soruce files. * tgextract * -- Bin grating wavelengths column Interface & Algorithm Changes - Added some new background region binning parameters, so background up and background down regions for binning the spectra no longer have to be symmetric to each other. - Removed the dependency of HEG grid on MEG grid. So HEG grid is no longer scaled according to the ratio of MEG and HEG preiod. New parameters are added for the user to specify the HEG, MEG and LEG grid. - Added function to log all the tgextract user input parameters in the output file. - Added the inregion_file parameter. So user can specify the source region, background up region and background down region in the input region file. Thus the tool can bin non-retangular regions. For example, a bow-tie shape region can now be defined for extracting HRC-S/LETG spectra. - The value of the output EXPOSURE keyword is now the average value of the input EXPOSURn keywords. - Added the TG_PART column to the output region extension. - Added the BACKSCUP and BACKSCDN keywords. If the output spectra have different BACKSCAL value, the BACKSCAL keyword will be promoted to a BACKSCAL column in the output SPECTRUM extension. Ditto for BACKSCUP and BACKSCDN. - Changed the POISSERR keyword from string type to boolean type. Also set its value to FALSE. - User cannot enter HESF as the value of the tg_part_list parameter, since this instrument is not supported. - The bin_units parameter can only be set to angstrom, eV, or keV. Degrees, pixel and mm are not supported. - The tg_d_units can only be set to degrees. Arcsec, mm and pixel are not supported. - The min_bkg_tg_d and max_bkg_tg_d parameters have been replaced by the min_upbkg_tg_d, max_upbkg_tg_d, min_downbkg_tg_d, and max_downbkg_tg_d parameters. - The min_bin, max_bin, bin_size and num_bins parameters have been replaced by the min_bin_leg, max_bin_leg, bin_size_leg, num_bins_leg, min_bin_meg, max_bin_meg, bin_size_meg, num_bins_meg, min_bin_heg, max_bin_heg, bin_size_heg, and num_bins_heg parameters. Bug Fixes - Fixed bugs in ACIS + LETG extractions. - Fixed bugs in background binning. - Fixed bugs in the output region extension. - Fixed the HEG binning bug. - Fixed the tg_d limits bug. So now if user's input min_tg_d is larger than max_tg_d, the tool will use the default min_tg_d and max_tg_d. Known Bugs - Command line filter cannot be used with the input file. * vtpdetect * -- Voronoi Tessellation and Percolation (VTP) source detection Interface & Algorithm Changes - Added WCS outputs - New parameter "ellsigma" controls size of ascii region outputs. - New parameter "regfile" specifies name of ascii region output file. - Debug file outputs are no longer generated. - Source list has columns suitable for DM spatial filtering (COMPONENT, SRC_POS, ROTANG, etc). Known Bugs - need to keep using "[cols x,y]" syntax for event lists * wavdetect * -- Mexican-Hat Wavelet source detection (wrecon & wtransform) Interface & Algorithm Changes -The parameter "bkgerrinput" is now a boolean rather than a file name. If true, the error image is expected in the [2] extension of the input background image. - The "mask" parameter has been removed. - New parameter "ellsigma" controls size of ascii region outputs. - New parameter "regfile" specifies name of ascii region output file. - Source list has columns suitable for DM spatial filtering (COMPONENT, SRC_POS, ROTANG, etc). - Source position columns are now double precision. - Now accepts a Chandra exposure map. - A precomputed background map will now be accepted (parameter "bkginput"). - "stall" parameter has been deleted. - A new background extrapolator has been added, which reduces the number of false detections at large wavelet scales near detector edges. - Units have been eliminated from RATE outputs. Bug Fixes - Incorrect source ellipse inclinations fixed. Slight algorithmic changes in computation of default normalized background. - A bug with non-symmetric exposure maps has been fixed. - Bugs have been fixed which affected background error computations when 1 iteration was specified. - A bug in PSF lookup for gratings data was fixed. - Bug is source property computation fixed. Bug occured when PSF size for source was greater than the largest wavelet scale size specified.
GUIs
========
Overview
========
In addition to numerous bugfixes and generally greater stability, the primary
UI contributions to CIAO2.0 are:
o The integration of an interpeted language (S-Lang) into CIAO, which
greatly increases the analysis capabilities of sherpa/chips/etc by
providing an interactive, IDL-like interface at the commandline.
o This work also formed the basis for the introduction of spectroscopy
analysis into CIAO (current tool name: GRASP, modeled in part on the
ISIS tool developed at the MIT/CSR branch of the CXC)
o Considerably improved interaction between our analysis applications, by
encapsulating the notion of an "analysis session" into a class library
that is linked into each of our applications.
o Switched from directly using the SM executable for plotting to using the
callable SM library instead.
o Considerably improved configuration of portable end-user distributions.
o GUIs now utilize parameter files to store persistent preferences.
o Several prism improvements, including 2 new XPA commands for GRASP,
solidified keyword editing (using dmtlib now instead of dmlist), support
for 2D slices (along z=1) of 3D images, providing structural info for ND
images (N>3)
o Several FiltWin improvements, including new menu entry and commandline
switch for loading data files, undo/redo fixes, and better ciao.par usage.
o Improved interaction of GUIs through Analysis Menu
o Private colormaps in GUIs
o Faster chips performance and improved configuration of runtime files
o RMF and image I/O in varmm; better chips parsing of SLang scripts, BUT
until the sherp parser can be generalized more the scripts passed to
"evalfile()" may contain ONLY SLang and chips commands.
o Many minor tweaks to remove compiler warnings and improve port stability.
Greater detail on these and other improvements is given below.
======================
Infrastructure Details
======================
Varmm/Grasp
-----
The Varmm [(Var)iables, (M)ath, and (M)acros] library is a moderately thin
wrapper around S-Lang, and with it provides:
o Access to a powerful IDL-like interpreted language, with a wide range
of arithmetic and programming capabilities.
o The ability for chips to execute scripts containing a mix of chips and
SLang commands, and for sherpa to mix sherpa, chips, and SLang.
o A C++ object interface to interpreted variables created interactively
during analysis, which permits applications like chips and sherpa to
easily utilize those variables for plotting/fitting.
o A rewrite of most of the sherpa I/O routines to enhance performance and
modularity (using DM where appropriate). These routines can be used to
create either interpreted SLang variables or straight C++ objects, and
currently support:
- ASCII
- FITS/IRAF bintable
- FITS/IRAF images
- ARF/RMF/PHA
The full range of DM filtering is supported, as well the ability to
skip arbitrary numbers of table rows when reading any file format.
o A replacement for GNU readline, which is less functional but free of
restrictions which formerly prohibited callable SM from being linked
directly into CIAO analysis applications.
o RMF and image I/O
o Faster and more robust SLang script reading in chips (but as noted above,
sherpa still needs more work)
o Better portability to Dec/Alpha and Linux
o See below for related prism work (new XPA commands)
o Bug fix to PHA I/O (aneta bug report, 6/16)
o Changed behavior of varmmGet so that variables that may be resized and
still be referenced in chips by their original name (dburke, 6/16)
o writeascii() I/O function added
o nominal/imperfect arrow key support added to VI mode
o tab completion added, and command history mechanism enhanced
o added ability to retrieve varmms (for plotting/fitting/etc) by
namespace ID (eg: chips> curve x myNSpace->x y myNSpace->y)
o Editor/Key binding preference for varmm readline taken from
editor parameter in ciao.par
These new capabilities are relatively vast in scope, and thus will
need some time to document completely. Until then, users can look
in ~mnoble/varmm.emails for more detail and explicit examples.
Session
-------
The interaction of our analysis GUIs/apps has been enhanced by encapsulating
the notion of an "analysis session" into a class library that is linked
into each of our applications. This class object manages a shared memory
segment and provides/supports:
o Multiple instances of chips running concurrently
o Imager exits ONLY when the application which invoked it is terminated or
when a 'session' is exited (thus non-CIAO invocations of ds9 or SAOtng
are NOT terminated when the user exits a CIAO applications)
o Ability to save the state of a session from filtwin or prism
o Ability to restore a session provided the user can access all of the
data files
o Better communication between the GUIs and toolagent (eg, tools invoked
from prism will default to running with the file currently being
viewed within the prism window)
o The ability for multiple users to login to a single machine under a single
login (eg, dutysci) without interfering with one another, provided each
uses a distinct DISPLAY.
o Introduced a ciao.par parameter file, which allows the user to set
various session-wide options like: the number of filter states to save
or the maximum number of applications to run in a session, etc.
o Updated session_save option to utilize ciao.par clobber parameter
if output file already exists
o Updated session_save menu to generate a warning dialog if unable to
save the session info
o Some internal enhancements for shared memory segments / session IDs.
o Session management utilities:
cxc_shmutility renamed to ciaoshmem; -f option added to display full
ciao session configuraton.
session_save renamed to ciaosave
session_start renamed to ciaostart
o See parameter files section below.
GUI library
-----------
o CIAO guis can now utilize a private colormap. The $HOME/.CXCdefaults
forcePrivateColormap entry may be set to True, False, or dynamic (the
default), which respectively allow the the GUIs (eg, prism) to always
use a private colormap, never use a private colormap, or switch to a
private colormap at runtime if necessary. This should address most
of the "GUIs are invisible/unusable" problems that have historically
occurred when the X colormap has been hogged by other applications
(like Netscape).
NOTE: This has been tested in mwm, twm, fvwm, and CDE, but the "right
thing to do" is ultimately a subjective question best decided by
individual preference. In essence we have simply traded one ugliness
(whitewashed GUIs from empty colormaps) for another (psychedelic flashing
when moving the mouse from one app to another), although majority
opinion deems the latter less unsavory.
o Fixed the Analysis menu pulldown's 'launch filtwin' option so
that the application gets launched and an appropriate status
message is reported
o Updated the Analysis Pulldown menu's launch commands so that the
imager, browser (prism), and filtwin are launched with the data
that is loaded in the invoking application
o File selection dialog behavior for initial directories and seach patterns
is now dependent on the settings of parameters in ciao.par
o Information/Error dialogs updated to display application name.
o Old parameter files automatically detected and replaced with latest
system version (previous is copied to a .bak file).
o Exiting from the window manager (for example using the window's close
menu option or {alt}{f4}) now performs the exit cleanup that occurs
when an application is exited from the app's file pulldown menu exit
option. Previously, the application terminated but the application's
cleanup procedures were not invoked.
Ahelp
-----
CIAO2.0 ahelp commandline option synopsis:
-s [subject][..context] :
-m [subject][..context] :
-l [subject][..context] : short,medium,long versions of help output
-w [subject][..context] : use web browser to display information
-k keyword : like appropros (searches for subjects related
to specified keyword)
-s -f file :
-m -f file :
-l -f file : processes file and display as formatted ascii
text in short, medium, and long formats
-f file : equivalent to -l -f
-r : used to maintain the help database; creates
index files necessary for search/retrieval
* Format of xml files changed - ENTRY entity has only four attributes:
key, context,refkeywords and seealsogroups.
* New entities have been added to dtd:
ADESC - like DESC but appended after PARAM block ,
HREF - used to keep information for hyperlinks in xml docs
translated to HTML
VERBATIM - used to suppress formating of text within this block,
XMLONLY - text used together with PASSTHRU to specify alternative
text for PASSTHRU to be displayed only in ascii mode.
* PARAMINFO entity removed as it was duplicating functionality of PARAM
* ADESC and PARA can have titles via title attribute
* ahelp keeps a database of subjects and references, which fosters
efficient dynamic help search/retrieval
* maintenance of xml files has been reduced via ahelp database reindexing
facilities (-r option)
* ahelp automatically creates intermediate web pages while reindexing
database if mulitple ADDRESS blocks are encountered
* extensive checking for compliance with the current dtd
* Some changes to the public C++ ahelp interface (see the code)
=====================
Data Analysis Details
=====================
Filtwin
-------
o Integrated session library:
- ability to receive updates from multiple instances of chips/prism
- dynamically updates list of currently running chips/prism processes
- ability to save/restore state of the entire analysis session
o Now displays a variable number (scrollable) of table columns, rather
than simply the static Temporal/Spatial/Housekeeping filter fields.
o When displaying a FITS table: column data type, tlmin, tlmax, and
units displayed through tooltips
o Better interaction with ToolAgent, feeding it the current file and
last filter which was applied.
o Ability to restore sessions via a filtwin commandline option
o Only one instance of filtwin may be invoked per analysis session
o Persistent user preferences are saved across invocations to filtwin.par.
o Uses ciao.par filtwin_init parameter to determine from where initial
data will be loaded {browser, imager, plotter, none}. Filtwin will try
to use the lowest available instance of the specified application.
o Updated to allow the user to specify an initial data file or a session
resume file as a commandline option. Starts a restore if the file is
a session file, otherwise the filename is put in the file text field
o Update filtwin instantiation from chips so that undo/redo work
correctly
o Update filtwin to prevent the filter field scroll window from flashing
all of the columns of a file when an undo/redo is done on a file for
plotting (only the axes columns should be displayed)
o Update so that an 'apply' command to a table file attempts to image the
data instead of automatically assuming that the data should be plotted.
o Renamed 'Plotter' in the options toggle to 'Plot Zoom' to better indicate
that it performs a chips pickpoints operation.
o Feedback provided when a user performs a save data command.
o If three axes are provided in the 'Cols' field, hitting [APPLY] will
plot col1 as X vs col2 as Y with col3 as the errs.
Chips
-----
o Switched from using IPC-based communication with background SM process
to using callable SM library. This makes plots from multiple apps
independent of one another, should provide a substantial memory savings
for large files, generally increase performance, and simplifies the
software, configuration, and distribution.
o Integrated Varmm interface to S-Lang interpeted language into Chips.
This allows users to plot directly from interpreted variables created
during interactive analysis, write powerful functions/scripts, and
provides a broad amount of other arithmetic and programmatic capability.
o no longer need to write an .sm.cxcds to $HOME
o Engine and Enginemode commands disabled.
o API now supports curve/contour/surface creation from float*, double*,
short*, and long* arrays (previously only float* was supported)
o Improved performance of contour plotting.
o Several bugfixes:
- ds9 would display blank screen if an absolute filepath not provided
- at exit time threads no longer generate "killed" message on Linux
- minor fixes to axes width, width, ln width, and errs commands
- small parsing fixes
o Info command has been updated to display more information
o SAO/DS9 2.0 is bundled with CIAO2.0, and SAOtng has been removed
from the package. Users may still utilize SAOtng (1.9.1 or later) if
they have it installed on their systems and edit $HOME/.CXCdefaults to
swap the default imager entry from ds9 to saotng.
Firstlook
---------
o Now exits when it receives a close request from the window manager.
o Generated lightcurves now display err bars.
o The name and path of generated lightcurve/spectrum files is echoed.
Online Help
-----------
o Online help output is automatically sized to fit the screen width.
Prism
-----
o Now utilizes dmlist for header keyword display
o DataModel-visible header keywords may now be edited.
o Easier and smoother navigation:
- added Navigate menu and more useful keyboard accelerators
- no longer pops up annoying "working ..." window in average case,
but only when large # of rows is being scrolled
o Attempting to open files that cannot be read by DM no longer leaves
prism in an undefined, semi-unusable state.
o Removed references to status, error, and quickhelp "external message
files," to simplify the codebase and make it easier to debug/maintain
o Bulk column/row deselection
o Persistent user preferences are saved across invocations to prism.par.
o Added "moveto {block#}" XPA command to enable switching the current
block being displayed via remote control.
o Added "selected rows|cols" XPA command to retrieve (via remote control)
the rows/cols currently selection-highlighted
o Support 3D image reading by taking 2D slice along z=1
o Correctly reports structure of 3D, 4D, etc images, instead of issuing error
(images > 3D, however, will still not be read).
o Also, image info is presented more naturally/correctly as dim1 x dim2 x ...
instead of treating all images as 2D tables of "dim1 rows, dim2 cols"
o Directly uses dmtlib to display keys (instead of dmlist executable) to
to ensure header display is always in-sync with file contents.
o Clear out the data matrix when a block is either empty or cannot be read.
Previous behavior left the old data intact
o Small improvement to image I/O performance.
o Several bugfixes:
- first cell of an expanded vector column no longer 'selected'.
- eliminated XmScrollBar warning messages when changing to files
with fewer columns than the currently displayed file.
- fixed segv seen when toggling between two extensions in FITS file
ToolAgent
----------
o Toolagent behavior now configurable via ciao.par 'toolagent' parameter.
If set to DISPLAY, a GUI taskmonitor is launced to display the stderr
and stdout of the launched tool. If set to 'BACKGROUND', only the tool
exit status is displayed, in an small popup window.
=======================
Portable System Details
=======================
Installation
------------
o This release will eliminate the admittedly clumsy convention of using
CXCDS_ROOT_{system} as the base CIAO install directory, and instead
offers the user the ability to name this directory whatever they
please. This will also simplify the installation by reducing the number
of times the user must change their current working directory, as all
files can now be untarred from the same level in the tree.
o More intelligent searching of X and Motif/LessTif libraries, which
also helps reduce the size of our Solaris OTS.
o Supports multiple types of distributions (CIAO analysis / NRA3), while
making it easy to add more.
Building from source:
---------------------
We now utilize more of the power of Autoconf/Configure scripts. The
user can now customize the configuration by using any of the following:
--x-includes=DIR X include files are in DIR
--x-libraries=DIR X library files are in DIR
Note: The location of the X libraries is determined automatically
however if you want to use X from a non-standard location or if
configure has difficulty finding the X libraries or headers you
might want to use these switches.
--with-comp Compile or just configure. Required for compilation.
--with-top=/top Explicitly specify top of distribution
This switch is used if you are installing ciao on a read-only file
system such that the physical location is different from where the
users will see it.
--with-irafdir=DIR Indicate where the IRAF directory is. The default
is /iraf
--with-fits Include FITS kernel
--with-iraf Include IRAF kernel (Solaris only)
--with-iraflibs Use IRAF libraries provided in package (Solaris only)
--with-nothreads Disable threads
This switch should only be used on older systems that do not fully
support threads.
--with-debug If the users are compiling the software, this
switch will setup the make scripts to compile with the
-g switch by default.
--with-opt=-O3 If the users are compiling the software, this
switch will setup the make scripts to compile with the
select optimization switch. -O3 is the default level if
no argument is provided.
--with-tcltk={base tcldir} If the users are compiling the
software. This sets the directory where the make scripts will
lib and include directories that contain the tcl/tk libraries
and includes. If this switch is omitted the default is
/usr (this should be correct for most linux distributions).
--with-tclver=8.0 If the users are compiling the software, this
switch select the version of tcl/tk to use. Currently 8.0
is the only version that has been tested. The default is 8.0.
--with-lesstif={base dir for lesstif / Mofit} If the users are
compiling the software, this switch sets the base directory
to look for the Motif lib and include directories. The
default location is /usr/X11R6/LessTif
=========================
Environment/Configuration
=========================
o SM resource files are GNU-configured during CIAO installation
o Removed XPA_SHORT_TIMEOUT and added XPA_METHOD=local to the
$HOME/.cxcds.(c)sh configuration, the latter of which helps avoid
problems on machines (like laptops) not attached to networks.
===================================
Runtime Configuration/Customization
===================================
'ciao' alias (ciao.csh and ciao.sh)
-----------------------------------
o New switches
-v Version switch: Prints the versions of all CXC software
installed in $ASCDS_INSTALL tree.
-i CIAO info switch: This switch activates a script that
dumps you system configuration information the screen and
the file $HOME/ciao_info.report This file can then be
used to help debug users problems.
-o Force setup switch: This switch allows the users to over-ride
their current ciao setup and setup a new version of ciao.
This switch will also remove the old setup for the users
path and LD_LIBRARY_PATH to avoid conflicts and path bloat.
Parameter Files
---------------
o new parameters in ciao.par to customize file selection dialogs:
pattern - specifies the default filter pattern to use the first time
a file selection dialog box is invoked from a GUI.
dir_option - specifies whether the default directory for a file dialog
should be based on the last directory used by the application or the
last directory referenced by any application in the session. This
value may be set to either APPLICATION or SESSION.
workdir - if dir_option is set to SESSION, this value specifies the
default directory to list when a file selection dialog is invoked.
o new parameter in ciao.par to customize readline key binding preference
editor - allows user to select emacs or vi type keybindings for readline
o $HOME/.CXCdefaults ehanced to provide better GUI colormap support
(background/foreground colors) under CDE. Colormap problems are mostly
eliminated under other window mangers.
UI CAVEATS
==========
* Retrieval of region specifications from ds9 in filtwin is currently
performed in physical coordinates. Unfortunately, there is an inconstency
between filtwin and dmcopy where dmcopy expects logical coordinates when
working on iamge files. As a result, when attempting to apply a region
filter from filtwin to an image in ds9, the user currently needs to
convert the region descriptor into logical coordinates.
* The region filtering interface in ds9 was changed in ds9 v1.9.4 and
was changed back insubsequent versions of the imager. As a result,
filtwin's region handling code works with various versions of ds9
except for v.1.9.4.
Sherpa
* New Commands *
* ANALYSIS: allows the user to specify space for analysis: energy,
wavelength, channel
* BACKERRORS: allows the user to specify errors for the background
dataset read in with "BACK." (Can be used with READ to read in
errors from a file.)
* COVARIANCE: calculation of the covariance matrix and error estimates.
* DEMINIT: a command to be used with the GUIDE package only.
Prepares an intermediate file with flux and line positions from
each model, and temperature and emissivity grids culled from the
APED line list file.
* FEFFILE: specifies a FITS Embedded Function file for Sherpa to
read in.
* FEFPLOT: allows the user to plot the line-spread function for a
given photon energy in keV, based on information read in with
FEFFILE.
* INTERVAL-PROJECTION: creates a 1D plot showing the fit statistic
as a function of the value of the specified parameter, with all
other thawed parameter values allowed to float.
* INTERVAL-UNCERTAINTY: creates a 1D plot showing the fit statistic
as a function of the value of the specified parameter, with all
other thawed parameter values held to their best-fit values.
* LPLOT: formerly the Sherpa PLOT command. Uses all the same options
formerly understood by the PLOT command; may be abbreviated LP.
* PRECISION: allows the user to change the number of significant
digits in some values output to the screen with SHOW.
* PROJECTION: calculation of error estimates.
* REGION-PROJECTION: creates a 2D plot showing fit statistic
contours as a function of the values of the two specified
parameters, with all other thawed parameter values allowed to float.
* REGION-UNCERTAINTY: creates a 2D plot showing fit statistic
contours as a function of the values of the two specified
parameters, with all other thawed parameter values held to their
best-fit values.
* SETBACK: allows the user to set the time, backscal, or areascal
for a background dataset read in with the BACK (or READ BACK)
command.
* SETDATA: allows the user to set the time, backscal, or areascal
for a dataset read in with the DATA (or READ DATA) command.
* TRUNCATE: resets negative model values to zero. Automatically set
if the statistic is CASH or BAYES.
* UPDATE: stores relevant information about a fit in an S-Lang
variable called "fitResults".
* WCS: allows the user to use WCS coordinates, if available, when
fitting two-dimensional models.
* GUIDE * -- Grating User Interacitve Data Extension
* A new package for analyzing grating data, called GUIDE (Grating
User Interactive Data Extension) has been created. This package
can be accessed from Sherpa and consists of a small number of
Sherpa commands and a large number of S-Lang scripts. Please see
the GUIDE Manual for more information.
* Access to S-Lang has been added to Sherpa. S-Lang is an
interpreted language that may be embedded into an application to
make it extensible. Thus it is now possible for the user to write
their own S-Lang scripts, which incorporate both S-Lang and Sherpa
commands, and run them inside Sherpa.
* Data from S-Lang variables (which were read in via the VARMMIO
read functions) can be assigned to Sherpa data sets.
* A new format for storing information relevant to a fit, called the
Model Descriptor List (MDL), has been created. The MDL stores all
relevant information about a fit (e.g., names of data files,
source and instrument models, and model parameter values). The MDL
can be written to a FITS file, or stored in an S-Lang variable
called "fitResults". This variable is automatically updated after
every fit, and can also be updated manually via the UPDATE
command. The MDL is used by GUIDE, but may also be used for
non-GUIDE analysis.
* Changes to I/O *
* The user can now read in PHA datasets from PHA Type II files.
* The user can now read in/analyze double (real*8) data.
* Support for a Sherpa preference file, .sherparc, has been added.
* SAVE I/O has been altered; vestigial parameters DELTA and EPSILON
are no longer output; errors are specified by err_min, err_max,
and err_met, where err_met = c (covariance), p (projection), or u
(uncertainty).
* The command record now places the file search.dat.[process ID]
into the ASCDS_WORK_PATH directory, normally /tmp.
* Users can now update multiple parameter values using GETX, GETY,
instead of just one.
* Users may save a Model Descriptor List (MDL) to a FITS file with
the WRITE command.
* Users may read in from an MDL file with the READ command. This has
the effect of reading in all specified data files, defining all
source and instrument models, and setting all parameter values.
* Changes to General Fitting *
* The means by which one specifies datasets in commands has been
generalized. Commands that are affected: ANALYSIS, BACKGROUND,
COVARIANCE, FIT, GOODNESS, INSTRUMENT, IGNORE,
INTERVAL-PROJECTION, INTERVAL-UNCERTAINTY, NOTICE, PROJECTION,
REGION-PROJECTION, REGION-UNCERTAINTY, SOURCE, SUBTRACT, TRUNCATE,
UNCERTAINTY, UNSUBTRACT, WEIGHT, WCS
* The user can now jointly fit data specified with DATA and BACK
(before, all BACK data could be used for was background
subtraction). Two backgrounds are allowed per dataset
(specifically, BACK UP and BACK DOWN; the names reflect the tie-in
with grating analysis). Also, the background can be fit alone.
* The user can now specify background errors (BACKERRORS), either on
the command line or from a file. (Equivalent to ERRORS.)
* The user can specify the time, backscal, or areascal of an input
dataset (SETDATA) or background dataset (SETBACK).
* The user can change the analysis space (e.g. from energy-space to
wavelength-space), or determine the current analysis space, using
the command ANALYSIS. Note that ANALYSIS is reset automatically
when PHA files are read in.
* Sherpa can now fit arbitrarily filtered histogram ASCII input.
* Fluxes can now be determined for individual model components.
* Counts fluxes can now be determined for individual model components.
* The user may filter data using NOTICE/IGNORE ENERGY/WAVE.
* Parameter value guesses for background data fits are now based on
the background data, not the source data.
* Changes to Responses *
* The user can now specify an "encircled energy ARF" (EEARF) as the
third parameter of the model RSP. Sherpa also looks for the EEARF
in the RMF fits file; if there, it need not be specified separately.
* The user-specified ARF file no longer needs to have the same
binning as the user-specified RMF file. (However, the EEARF
binning must currently match the ARF binning.)
* The user can read in a FEF file using the command FEFFILE and
display line-spread functions for a given photon energy using the
command FEFPLOT. Note that the user cannot yet *use* the displayed
line-spread function as a response.
* Sherpa can now apply PSFs to filtered 2D data.
* Changes to Optimization Methods *
* The LEVENBERG-MARQUARDT method has been fixed, allowing it to find
the actual local minimum in parameter space in fewer steps.
* The implementation of the SIMPLEX method is improved, such that it
is less likely now that it will stop short of the local minimum.
* GRID & GRID-POWELL have been changed so that the user may specify
the number of grid points on each axis separately.
* The method parameter SHOW interface has been improved; notably,
parameters which do not actually used by the method are not listed
(e.g. the delta parameter for Sherpa 1.0).
* Changes to Error Analysis Methods *
* The uncertainty algorithm has been updated to allow the user to
find uncertainties for subsets of the thawed parameters; it also
can find uncertainty bounds beyond the min and max parameter
values stipulated by the user. Also, one can apply uncertainty to
a subset of the entered data (e.g. to one specific dataset when N
datasets have been entered).
* Two new error analysis algorithms have been added: projection and
covariance. The former determines error estimates numerically by
varying each chosen thawed parameter in turn, and allowing the
other parameter values to drift to new best-fit values. The latter
estimates errors using the covariance matrix, the inverse of the
matrix of parameter space second-derivatives at the best-fit point.
* Four new error analysis/visualization routines have been added:
interval-projection, interval-uncertainty, region-projection, and
region-uncertainty. The former two allow the user to display a 1D
plot showing the statistic value as a function of the value of the
specified parameter (using either the projection or uncertainty
algorithms). The latter two allow the user to display 2D contour
plots showing the statistic value as a function of the values of
the two specified parameters (using either the projection or
uncertainty algorithms).
* New Sherpa Models *
* BPL/BPL1D: broken power law
* CONST/CONST1D: a positive constant, one dimension
* CONST2D: a positive constant, two dimensions
* DELTA/DELTA1D: a delta function, one dimension
* DELTA2D: a delta function, two dimensions
* EDGE: absorption edge as a function of energy or wavelength, one
dimension
* LINEBROAD: a line-broadening model, one dimension
* NGAUSS/NGAUSS1D: a normalized Gaussian model, one dimension
* Altered Sherpa Models *
* BB/BBODY: now includes amplitude parameter (ampl)
* LORENTZ/LORENTZ1D: functional form changed, now normalized
* PSF/PSFFROMTCD: parameter size changed to parameters xsize/ysize
(i.e., Sherpa supports asymmetric kernels); nsigma changed from
two to a user-specifiable parameter
* RSP: new parameter eearf (encircled-energy ARF file)
* SHEXP: the former Sherpa EXP model; functional form altered
* SHEXP10: the former Sherpa EXP10 model; functional form altered
* SHLOG/SHLOGE: the former Sherpa LOG/LOGE model; functional form
altered
* SHLOG10: the former Sherpa LOG10 model; functional form altered
* SQRT: functional form altered
* APEC: APEC models xsapec and xsvapec have been added. These are
models to calculate emission from a plasma with a DEM. Data from
either the APEC plasma emission file or a Raymond-Smith file may
be used (the APEC file is used by default). The model xsvapec
allows certain elemental abundances to be varied.
* General Model Changes *
* The evaluation of all two-dimensional models now includes
(numerical/analytical) integration (if desired).
* All two-dimensional models may now be evaluated in "WCS coordinate
space," i.e. as a function of RA and Dec.
* A new command, TRUNCATE, has been added. This resets model values
that are less than zero to zero, allowing the statistics BAYES and
CASH to be used without difficulty. Note that when those
statistics are specified, TRUNCATE is automatically turned on.
* Sherpa's algorithms for guessing parameter values were overhauled:
(1) better guesses are made for some parameters (such as fwhm of
gauss); (2) time and arf information is taken into account for
guesses in the analysis of PHA data; (3) if the source model
expression and the dataset number are specified in the same
command (e.g. source 4 = pow[p]), then that dataset is used for
guessing, and not the first dataset.
* Changes to Statistics *
* For model variance chi^2 (CHI MVAR), error propagation in
background subtraction has been fixed, and post-fit plots show
properly sized errorbars (instead of using a default errorbar size
of one).
* For Primini chi^2 (CHI PRIMINI), error propagation in background
subtraction has been fixed, and post-fit plots and goodness
computations use the correct error values.
* Plots of fits done using the statistics BAYES and CASH now show
errorbars estimated using the Chi Gehrels variance (rather than
showing no errorbars at all).
* Error propagation has been generalized for background subtraction:
(1) errors are properly propagated if the errors for either the
data or the background or both are read in rather than computed;
(2) errors are properly propagated if two background files are
specified (UP and DOWN for Chandra grating analysis).
* Plotting *
* Improved labeling for plots, by checking for exposure times, etc.
* One can now create composite image filters using NOTICE/IGNORE
IMAGE and the include/exclude region feature of ds9.
* Miscellaneous *
* Added function to wrapper class, to switch model integration on
and off from tools which use ascfitlib.
* RESET command changed so that only model parameters are reset, and
not the method parameters as well.
* Bug Fixes since v. 1.1 (not including items mentioned above) *
* I/O *
* A bug which caused improper fit results after the entry of
two-dimensional ASCII data entry was fixed.
* Fixed problem where chips was being started from Sherpa when the
user issued a help command, or made a typographical error.
* Fixed bug in WRITE ERRORS so that the correct, filtered errors are
actually output to the screen.
* Fixed bug causing the uncertainty minimum to be written sometimes
with "--" in SAVE ALL.
* Sherpa now checks PHA data for the column STAT_ERR_RATE if the
column STAT_ERR is not found (bug fix for Einstein data).
* The command RESET FILTER now immediately causes the data to be
refiltered; before, if RESET FILTER was followed by, e.g., FIT,
the fit was done using the old filter.
* Typing "q" while in GETX (setting a parameter value from a plot)
no longer leads to the parameter value being reset to zero.
* Plotting *
* The user can apply IGNORE/NOTICE PLOT to a log plot to get filter
boundaries.
* One can LPLOT/CPLOT/SPLOT/IMAGE negative data without having to
set ERRORS or READ ERRORS.
* Fixed parser bug so that commands like LPLOT DATA will open the
plotter automatically if chips has not yet been started.
* Attempts to LPLOT, SPLOT, CPLOT or IMAGE an instrument when no
instrument is loaded now cause an error message to be shown,
rather than cause a Seg.V.
* Fixed bugs relating to SPLOT/CPLOT SOURCE/BG.
* Contour plots of rectangular filtered regions now work correctly
(row-major intermediate information was changed to column-major).
* IMAGE [model name] now works properly for arbitrarily filtered
data; the imager no longer displays a 1 x N image.
* IMAGE ERRORS now shows the errors rather than the input data.
* SPLOT/CPLOT FILTER improved to give better zmin/zmax or contour
levels.
* Fitting *
* Bugs relating to timescales and data grouping in FAKEIT were fixed.
* A bug which prevented Sherpa from fitting a (small) subset of
filtered PHA datasets has been fixed for CIAO 1.1-1.1.1.
* Fixed bug in the computation of residuals for 2D data.
* An error which caused the number of degrees of freedom to be
sometimes incorrectly tabulated when background-subtracted data
was analyzed is fixed.
* PSF now keeps FFT of the kernel in memory--speeding the convolution.
* Defining a SOURCE for an empty dataset (made empty by IGNORE ALL)
no longer results in badly set (NaN) model parameter values.
* Parameter links can now be reversed after the link is broken (e.g.
m1 => m2+5 ; ... ; m1 = 20 ; ... ; m2 => m1-5 is now allowed).
* Using GUESS to reset model parameter values after a link has been
set no longer causes the link to be broken.
* Methods *
* A type-conversion error which caused garbage numbers to be written
to the search.dat file was fixed.
* Fixed bug in opt.C that caused the user's choice for random number
seed in some optimization algorithms to be overwritten with "14391."
* Models *
* A bug which caused Sherpa to Seg.V. when nested models were
defined was fixed.
* Increased the maximum number of model components per source and
background models to be 512 (was 32).
* Bugs in model dered fixed.
* Bug causing bbodyfreq to output zero amplitude fixed.
* Known Bugs and Limitations *
* Relating to plotting/imaging *
* The command CLOSE PLOT now longer deletes the plot window from the
screen. (This is not a bug, but rather related to the fact that
ChIPS is no longer run within Sherpa, but as a separate process.)
* The user may not use SPLOT or CPLOT to visualize an arbitrarily
filtered (i.e. non-rectangularly filtered) two-dimensional dataset.
* ChIPS curves are plotted through regions that have been filtered out.
* A Sherpa user may not overplot several datasets, although
overplotting is allowed when using ChIPS itself.
* Multiple plots in one ChIPS window that show different datasets
that have been filtered in the same manner may be shifted with
respect to one another.
* One cannot create images which show only those data which lie
within a NOTICE filter region.
* The ChIPS UNDO command cannot be used properly from Sherpa; labels
and axes are restored, but not the curves themselves. This is due
to the fact that in ChIPS, storing the state of a plot works when
the data were read from file, but not when read from S-Lang
variables. Sherpa sends ChIPS data via variables, instead of
dumping to temporary files, and hence UNDO (as well as STORE and
RESTORE) do not work well for plots generated via sherap LPLOT
commands. This fix for this will be for ChIPS to store plots even
when data came from S_Lang variables.
* Relating to filtering *
* A user may not read in a filter from a file, and then add to that
filter by specifying more filters from the command line
* In the analysis of PHA data, the user may not combine IGNORE BAD
with other filter commands; the desired filter will not be created.
* Read-in files related to a single dataset must have the same
precision. For instance, improper results will be obtained if the
user reads in single-precision data and a double-precision filter
for those data.
* Filter bounds that are given in double precision are truncated to
single precision, regardless of whether the data are expressed in
single or double precision. This can lead to data being improperly
left in or out of a filter region.
* Extra characters added to the end of otherwise valid NOTICE/IGNORE
commands will (i) cause S-Lang error messages to be shown,
although Sherpa *will* properly interpret the filter, and (ii) be
shown when the SHOW command is issued
* Some attempts to filter FITS table (FITSBIN) data using the column
name will fail because the Sherpa parser mistakes the column name
for a parser token. Similar errors will be seen if the column name
is channel. An exhaustive list of bad column names has not been
compiled.
* Relating to GUIDE *
* When the MDL is updated, the flux of each model component is
determined. Currently, the error on the flux is determined only if
the model component is a normalized Gaussian, a Lorentzian, or a
delta function. (For these models, the error on the flux is
equivalent to the error on the amplitude, which is simple to
estimate using COVARIANCE.)
* Currently, the error on the flux can only be determined for the
models listed above in (a) if the amplitude parameter is thawed,
i.e., not linked to another model's amplitude parameter.
* Miscellaneous *
* BUG: for the Linux version of Sherpa (compiled with gcc, v.
egcs-2.91.66), an attempt to read in non-existent files will cause
a core dump.
* After the command JOURNAL [filename] is given, all commands issued
to Sherpa are recorded, including mis-typed/wrong commands that
cause error messages to appear. This can lead to problems if the
output file is used as a script for a subsequent Sherpa session.
* Issuing the DATA or READ DATA commands causes Sherpa to first
delete all information that may already exist for the numbered
dataset.
* If one uses STATISTIC CHI MVAR or CHI PRIMINI, and issues the
command SHOW ERRORS after the fit, the errors shown will not be
correct. Do SHOW FILTER ERRORS instead.
Data Model
* "ahelp dm" gives high level on line help.
* Support for new DS9 region formats.
* dmlist now gives higher precision for double precision
quantities.
* dmcopy binning on a table which is filtered on the binning
axes now shrinks the image to surround the filtered region.
dmcopy evt.fits[sky=circle(4300,4200,100)][bin sky]
will make a 100x100 image. The command [bin/f sky]
will make the full image replicating the older behaviour.
* Multiple component ("logical OR") command line filtering:
You can now do
"m33.fits[(tdetx=4740:4750,5608:5650||
((detx,dety)=circle(5786,5010,2)&&(node_id=3||pi=58))||pi=122)
&&time=:63950001][cols time,tdet,det,pi,ccd_id,node_id]"
or
"m33.fits[@clean.filter]"
where clean.filter is a file containing
(ccd_id=0:6,8:9,chipx=256,512,768,fltgrade=8)||
(ccd_id=0:6,8:9,chipx=257,513,769,fltgrade=16)||
(ccd_id=0:6,8:9,chipx=512,fltgrade=104)||
(ccd_id=0:6,8:9,chipx=513,fltgrade=208)||
(ccd_id=7,chipx=512:513,fltgrade=2,64)||
(ccd_id=0,chipx=81)||(ccd_id=6,chipx=58,585)||(ccd_id=2,chipx=559,571)
This can make the data subspace very complicated; adding the string
[opt -ssupdate] will suppress the updating of the data subspace if you
find this is giving you trouble.
* "Exclude" syntax
You can do
dmcopy "evt1.fts[exclude pha=2,time=4]" bad.fits
dmcopy "evt1.fts[exclude @flt.fits]" bad.fits
or
dmcopy "evt1.fts[@!flt.fits]" bad.fits
and get the stuff that doesn't pass the flt.fits filter.
(Be careful with the @! notation which is quite limited,
you can't combine an
exclude and an include so [pha=2,@!flt.fits] is an error,
it in fact also excludes pha=2.)
* LIVETIME/ONTIME keywords now updated by filtering
If you do
dmcopy "evt1.fits[@flt1.fits]" evt2.fits
the GTIs are updated as before, but now the LIVETIME
and ONTIME (and LIVTIMn ONTIMEn) keywords are updated.
ONTIME is the sum of the GTI intervals, and LIVETIME is
ONTIME times DTCOR, (with DTCOR assumed 1 if not present).
The keywords are only updated if they are present in the
input file.
* There have been some improvements to the copying of GTI
extensions. GTIs are part of the file's data subspace and
dmcopy recreates them rather than copies them, but now
it preserves a few crucial keywords that got lost before.
* Bin image to image;
We finally implemented a simple image rebinning option.
You can rebin an image by an integer factor.
dmcopy image.fits[bin 4] blocked_image.fits
dmcopy image.fits[bin x=2:20:4,y=1:1000:4] blocked_image.fits
* String filtering: (FITS only)
dmcopy bpix1.fits[shape=point:rectangle]
If a table has a string column, you can filter on ASCII-ordered
ranges. The above example passes rows whose value of SHAPE
lies between "point" and "rectangle" in alphabetical order.
The range is a closed interval,
i.e. both "point" and "rectangle" pass the filter.
* Bit filtering (FITS only)
dmcopy bpix1.fits[status=001x0x0]
dmcopy bpix1.fits[status!=001x0x0]
This filter matches 001000, 0011010, 0011000, etc:
the 'x' means accept either 1 or 0. Leading zeroes may be omitted
so that 1x0x0 and 001x0x0 are the same.
Currently you can only do this on columns of bit datatype.
Note that XMM data uses an integer column for its FLAG bitmask;
we plan that a later DM release will let you use bit filters
on integer columns too, but CIAO2.0 can't do this.
* Bin to size:
dmcopy evt.fits[bin sky=1:1024:#200]
bin from 1 to 1024 generating a total of 200 bins, as an alternative
to specifying the step size directly.
* The syntax [cols #1,#2] now works and selects columns number 1 and 2.
* Multiple syntax sections:
dmcopy bpix1.fits[shape=point][cols status][status=001x0x0]
is now interpreted as identical to
dmcopy bpix1.fits[shape=point,status=001x0x0][cols status]
This will be helpful for programmers wishing to append extra
filters to an input DM spec. The block name, if present,
must still be the first syntax section:
dmcopy bpix1.fits[events][shape=point]
not
dmcopy bpix1.fits[shape=point][events]
* Control option syntax:
[opt option-list]
dmcopy evt.fits[bin time,pha][opt mem=200]
The supported options are:
mem=n Max memory is n megabytes. Used to control how
big an image you can make when binning.
-ssupdate Suppress updating the data subspace.
type=i2 Control data type of output binning image.
type=i4
type=r4
type=r8
Subroutine interface enhancements
---------------------------------
These are relevant to anyone programming directly with the DM interface.
* WCS enhancements: CROTAn, EQUIn, LONPn, LATPn recognized.
New routine dmCoordSetParam allows you to set LONPn.
* Support for null subspaces. If you filter e.g.
dmcopy evt.fits[time<0] flt.fits
this gives a data subspace which is empty.
DM didn't use to handle this; now it writes a
DSVALn = "NULL"
keyword and filtering recognizes this and acts accordingly
(so filtering on a file with any such keyword will give no
output rows).
Bug fixes:
* Binning xmin:xmax:step generates bins from xmin to xmin + n * step
where n is such that xmax <= xmin + n * step
A bug in binning created an extra bin in the common case where the step
size divided into the range exactly (in other words, the eqn above had
a < instead of a <=). This is now fixed so your images may end
up 1 pixel smaller than they used to.
* Column selection/filtering bug: specs of the form
evt.fits[pha=20:40][cols time,grade]
in some circumstances failed to apply the pha filter if it was not
in the list of selected columns. This is fixed properly now - I hope!
* Compound areas now calculated correctly (e.g. BACKSCAL in dmextract)
* Multiple component filtering support.
You can now do
dmcopy "evt1.fits[@flt.fits]" evt2.fits
and get the correct answer! Earlier implementations were flawed
when multiple GTIs were present.
* Data subspace failed to handle default filters in multiple-component
subspaces correctly.
* The FITS kernel occasionally got confused about which HDU
it was meant to be writing to. I think this is fixed now.
* The QPOE kernel was inconsistent about block names, sometimes
appending ".qp" to them; I fixed this, the block name will
not include ".qp".
Outstanding bugs
-----------------
There are still problems with inserting header
keys; still no support for writing variable length arrays or
filtering on row number. Support for filtering regions from
images is still primitive, with logical and physical coordinates
not being well supported; using world coords (RA and Dec) is safer.
Other Libraries
Pixlib
------
- The various pix_*.par parameter files are now gone. They have
been replaced with a set of CALDB indexed FITS files. The files
used by pixlib are set by the "geom.par" file.
TCD
---
- The transformations/convolutions/ and deconvolutions library has
been upgraded to use the "FFTW" algorithm. It has also been
upgraded for futher efficiency for 2D FFTs. Tools such as
wavdetect, csmooth, and aconvolve should experience great
improvements in speed.
CALDB (*new*)
-------------
- Based on version 4.2 of FTOOLs, the CALDB lib allows some selected
tools to access the calibration database (CALDB) directly. Each
tool has to know what kinds of files it is looking for (a choice
made by the developer+scientist) and then when the string "CALDB"
is used for the calibration file name the tool will automatically
find the file that matches the current observation. Tools that
have been CALDB-enabled include: acis_process_events,
hrc_process_events, tg_resolve_events, tg_create_mask, tgextract,
mkgarf, mkarf, mkinstmap, and pixlib.
Parameter Interface
-------------------
- Will try to detect if an out-of-date parameter file is present
in the user's home directory and will move it out of the way
so the tool can run without any problems.
- Uses a different command line interface library so things like
tab completion, back-arrows when the line wraps, emacs & vi
style key bindings, etc may not either look or function as in
previous versions of CIAO.
DSLIB
-----
- Various physical constants were updated with the most recently
accepted values.
REGION
------
- Bug was fixed in polygon filtering where two adjacent points
shared the same x-coordinate.