| No. 18 - 1st July 2005 | Edited by Marc Türler |
 |
 |
 |
| << Previous issue | Next issue >> |
| |||
| |||
|
Here it is, the special issue for the release of OSA 5.0, the new version of the Off-line Scientific Analysis (OSA) software! This newsletter is entirely devoted to this event and therefore does not contain the usual Science News and Community News sections.
As you will read below and hopefully experience yourself, OSA 5.0 constitutes a major step for the INTEGRAL data analysis. Significant improvements have been achieved both on the scientific results and on the user friendliness of the software. The many changes in the software directly impacting on the quality of the scientific results are listed in the various articles (one per instrument) below. Besides this, much effort has also been put in updating the documentation (user manuals, cookbooks, FAQs) to better guide INTEGRAL users in their analysis. The lifting of the ISDC WWW site should also allow now an easier access to data analysis resources.
OSA 5.0 is definitely not the ultimate INTEGRAL data analysis software, but it constitutes an important milestone. Some two and a half years after the start of the mission the INTEGRAL community has now a mature data analysis package for all instruments.
The ISDC Staff and the Instrument Teams wish you much success with your INTEGRAL analysis and a lot of new and interesting scientific results !
 |
 |
|
| General Improvements of the User Friendliness of OSA 5.0 | ||
|
|
I. Lecoeur (ISDC) |
|
The ISDC Team is pleased to ship the version 5.0 of the Off-line Scientific Analysis (OSA 5.0), the fifth major release of data analysis package since the launch of INTEGRAL almost three years ago.
Of note in this release is a significant improvement of the scientific analysis capabilities and particularly that of the ISGRI imager. The documentation was extensively rewritten and the general usability of the software was improved through e.g. a general revision of the graphical interfaces and a simplification of the analysis scripts.
OSA 5.0 is now available, not only on Linux and SUN Solaris platforms, but also on Mac OS X. We strongly recommend you to download and install the provided binary packages which are compatible with these operating systems. You can find them on the software download page. Nevertheless, if for any reason you do not succeed with the binary packages, an OSA 5.0 installation from the source code is possible.
We have removed commercial third-party packages from OSA 5.0 which is now completely self-contained. It includes ROOT 4.03/02 and several other mathematical packages. Note that the use of ROOT is optional but is necessary for the display of Graphical User Interfaces (GUIs).
After software installation, you no longer need to download the full Instrument Characteristics (IC) file which occupies much disk space. A download script is now available allowing you to install just the IC files of a given instrument.
The Installation Guide was rewritten and provides help with the software, IC and catalogue installations. The four instrument User Manuals were also rewritten and are meant both for beginners and advanced users. PDF and HTML versions are available for each one. Simple examples - from the download of INTEGRAL archived data to the generation of images, spectra and light curves - are presented in the different cookbooks. The GUIs and their parameters are explained in detail and recipes, tips and tricks have been added to ease your analyses. We also offer a revised FAQ web page, with a new high-level interface and extended contents.
We wish you will enjoy this new version on which all the INTEGRAL contributors
participated. Your comments are welcome through the INTEGRAL Helpdesk be they related to the
software, the documentation or the web interface.
| IBIS/ISGRI Changes & Improvements in OSA 5.0 | ||
|
|
R. Walter & N. Produit (ISDC) |
|
ISGRI pixels regularly become noisy. Noisy pixels are detected
on board and the transmission of affected events is stopped
automatically until the pixel comes back to a normal state.
A small fraction of affected events are however transmitted
and they could have a dramatic effect on the image quality if
not properly flagged. Two new algorithms have been implemented.
The first and most powerful one flags events as soon as the statistic of the time difference between events of an individual pixel is not as expected. On average few hundred pixels are killed in every pointing which improves dramatically the image and count rate reconstruction (see Fig. 1).
Fig. 1: Crab count rate vs. time. The blue curve with large variations was derived from OSA 4.2. The red curve derived from OSA 5 is much more constant thanks to the new noisy pixel event rejection algorithm.
The second algorithm flags events that come from pixels with abnormal event spectral distribution. This kills few hundred pixels in addition and mainly improves the source spectra.
Only few percent of the detector pixels are affected which does not change significantly the instrument sensitivity. As a result the count rate of a bright source is now stable within 3% (deviations up to 20% were not unusual with OSA 4.2).
The OSA 4.2 dead time correction was affected by a possible on-board counter overflow in case of very bright source or background conditions. This is now correctly taken into account.
The off-axis correction and background maps were not properly interpolated with energy in OSA 4.2. This could generate artificial spectral features. These interpolations are now performed by a new executable weighting and interpolating the maps taking the Ancillary Response Function (ARF) and the average source spectral shape into acount.
The ISGRI energy calibration is a convolution of the rise-time correction (LUT1), of the electronic calibration (LUT2) and of the gain calibration.
The gain calibration takes now into account temperature effects
and the degradation of the detector with time and after solar
flares. This has been calibrated within 1%.
The on-board pixel low energy thresholds have been decreased by about 2 keV. The analysis software has been modified to take the values of those low energy threshold better into account. The on-board modification and the improvement of the software together increases the low energy count rate of the instrument by a factor of 5 between 13 and 22 keV (see Fig. 2).
Fig. 2: Spectra illustrating the gain in low energy sensitivity obtained by a better handling of the detector low thresholds (blue: old settings; black: new settings).
A new and better Look-Up Table 2 (LUT2) has been derived using ground calibration data. A validated Redistribution Matrix File (RMF) taking the new LUT2 into acount is however not yet available so the new LUT2 was not included in OSA 5. However the ARF was modified to provide a reasonable response down to 17 keV and to take into account all the OSA 5 software improvements affecting the source count spectra. 1% systematic errors should be taken into account when fitting spectra.
The image cleaning has been improved very significantly and several options have been added to fix known source positions to improve ghost cleaning, to force cleaning of faint sources even if found negative.
Spectral and lightcurve extractions benefit now from a new fitting
procedure that works much better for faint sources and remove the
significant hard tails that were found with previous versions of
the software.
New background maps (see Fig. 3) have been built from empty fields. As these are defined in narrow energy bands, they could be used for spectral extraction.
Fig. 3: Example of an IBIS/ISGRI detector background map in a narrow energy band.
Besides this the analysis script parameters and graphical user interface (GUI) where improved. Several improvements important for bright sources are still being worked out and could not be included in the current OSA release. Those should be made available later this summer in the form of new calibration files:
| IBIS/PICsIT Changes & Improvements in OSA 5.0 | ||
|
|
L. Foschini (IASF/CNR, Bologna, Italy) |
|
The OSA for PICsIT has been almost completely revised for the release 5. Some
bugs have been fixed, in order to have a more robust pipeline (e.g. overflow
error in lightcurves), while some user's request have been added, like the use
of a variance shadowgram not corrected for background. Indeed, in the
previous releases of OSA, the shadowgram of the variance is updated for the
background subtraction after its creation. This means that the significance in
output from OSA is already corrected for most systematic effects of background.
With OSA 5, it will be possible to select the corrected variance or the raw
variance: in the latter case, the significance map will not be corrected and
therefore it is necessary to analyse the signal-to-noise (S/N) distribution in
the pixels in order to evaluate the background level.
OSA 5 also provides a series of 5 new sets of background maps prepared by Piotr Lubinski (ISDC) by using the public data at different epochs. The default set of maps used in OSA 5 will remain the set with 1.7 Ms of exposure, built with revolutions from 49 to 67, but the user will have in the IC files more maps, in order to try different background subtractions.
Figure: IBIS/PICsIT mosaic image of the Crab. The Crab is clearly detected at the centre of the intensity image (left). Its 8 ghosts aligned on a square around the source are also visible. The exposure map is shown on the right. The maximal exposure time (in the white area) is of 670 ksec.
A spectral extraction prototype module for point sources, based on PIF (Pixel Illumination Factor), is also included in OSA 5. It should, however, be used with extreme care, since PICsIT has not too many counts in a single science window when compared to the background (we remind that the Crab rates is about 10 c/s to be compared with a background rate of ~2500 c/s over all the detector). It is worth mentioning that it is always possible to extract the spectrum also from the imaging results (there is also a script to do this made by M. Chernyakova and available in the contributed scripts page).
Last, but not least, the spectral timing mode settings have been updated with
the change of the energy boundaries of the channels. The number of channels is
still 4, with time resolution of 4 ms, but the boundaries have been changed to
this set: 260-364, 364-676, 676-1196, 1196-2600 keV. These new settings would
be useful in the studies of Gamma=Ray Bursts at high energy, since PICsIT is
currently the only in-orbit instrument able to explore the MeV energy range.
| SPI Changes & Improvements in OSA 5.0 | ||
|
|
P. Dubath (ISDC) |
|
The spi_science_analysis pipeline script has been fully
redesigned. The "alternative" part was removed and the interface
simplified. The Graphic User Interface is now clearer and it is also
much easier to launch the script without the GUI, either from
command-line analyses, or from scripts.
Figure: Main GUI window for the redesigned SPI science analysis in OSA 5.0.
The script runs an improved version of the image reconstruction program spiros (version 9.2). Much efforts have been made to obtained a more robust software, with additional functionalities in "timing" mode, while the "imaging" and "spectral" modes are basically unchanged.
A new script spi_grb_analysis, including two new executables, has been developed to ease analyses of Gamma-ray bursts. The main parameters are the start and the stop of the GRB, either in UT or in IJD, and they are entered as command-line arguments.
A new set of Instrument Response Functions (IRFs) is part of the package. It include three independent responses for before, between, and after the detector 2 and 17 failures. The responses have been fully recomputed after the discovery that a part of the JEM-X mask was not properly modeled in the GEANT simulations. This problem was affecting however only analysis of sources with very large offset angles towards the JEMX instruments.
A large effort has been made to improve the documentation. The User Manual has been re-worked heavily, and a SPI data analysis WWW site has been redesigned for public access. It now includes a full user oriented documentation tree.
Scientific validation of the SPI data analysis system is going on at the
ISDC and in different instrument team sites. A paper dedicated to the
evaluation of the performance of point-source data analysis has been
published in MNRAS and is available from our OSA documentation page.
| JEM-X Changes & Improvements in OSA 5.0 | ||
|
|
N. Lund & N. J. Westergaard (DNSC, Copenhagen) |
|
In parallel with the development of OSA software for public use a number of "stand-alone" packages are developed within the JEM-X team to test methods for improving the JEM-X imaging and spectral extraction. The stand alone imaging software has now reached a stage where it is useful to make it public as part of OSA 5. In addition a special mosaicing tool adapted for the new JEM-X images has been developed.
Hence the imaging software has been completely reworked. The set of three components for image making, source finding and intensity correction has been replaced by a single component: j_ima_iros. This produces better images and makes better estimates of the source fluxes. As the name `IROS' suggests an `Iterative Removal Of Sources' mechanism is used for the flux determination and improved sensitivity. Also a more detailed description of the detector behavior, and instrument geometry has been included.
The images are formed by backprojection of the shadowgram counts using a pre-calculated aperture response function, which are stored as IC files with data structure name JMXi-BPL.-GRP (where `BPL' stands for `BackProjection List'). A cleaning process takes out the large scale structures of the image as well as systematic ripples arising from instrument features.
During the IROS process several images are formed and it is possible to specify through the parameter ("skyImagesOut") which images to output. Two keywords, IMATYPE and IMADESCR, report on the image type. The table below gives the possible combinations:
| IMATYPE | IMADESCR | Unit | Description |
|---|---|---|---|
| INTENSITY | RAW_INT | counts | "Raw intensity" no cleaning |
| INTENSITY | RAW_RECT | cts/cm2/s | "Raw rectified" with cleaning |
| RECONSTRUCTED | RES+SRC | cts/cm2/s | Residual image with sources as found in the RAW_RECT image 1) |
| VARIANCE | VARIANCE | (cts/cm2/s)2 | "Variance map" |
| RESIDUAL | RESIDUAL | cts/cm2/s | "Residual image" Strong sources removed |
Fig. 1 gives an example of a set of images for the energy range 6 to 20 keV. The basic source finding criterion is that a candidate source should be located at nearly the same position in at least two energy bands. A sufficiently strong source will also be accepted even if it is only found in just a single energy bands. If only a single energy band is analyzed then a high DETSIG (detection significance) value is used as acceptance criterion (the default limit for DETSIG is 12).
Fig. 1:
A set of JEM-X images from SWID 005100180010 where several
sources were found. Upper left: The RAW_INT image with large scale
structure due to the presence of sources. Upper right: The RAW_RECT
image where the large scale structures have been removed in the
cleaning process. Lower left: The RESIDUAL image (not cleaned) after
source subtraction. There are relics of the sources because the
modelling is not perfect. Lower right: The RES+SRC image with the
best presentation of the sources.
Fig. 2 is a zoom-in of the previous figure except that here four RES+SRC images corresponding to four energy bands are shown. The energy ranges from left to right: 3-6, 5-15, 6-20 and 10-25 keV. The color scale has been optimized to bring out the sources.
Fig. 2: Zoomed JEM-X images corresponding to the four energy bands: 3-6, 5-15, 6-20 and 10-25 keV.
Another new component is the mosaicing tool, j_ima_mosaic, adapted to the images produced by j_ima_iros. It should be used in place the ISDC tool, "image_mosaic", which for the moment does not make the best of the JEM-X images. One of the major purposes of a mosaicing tool is to reveal faint sources not visible in the individual images. In order to maximize the signal/noise ratio of each pixel in the mosaic image j_ima_mosaic makes use of the variance maps produced by j_ima_iros to calculate the weighted contribution of each pixel from each input image to be mosaiced.
The input images to j_ima_mosaic (from a given observation group) can
be either "Raw rectified" intensity images or "Reconstructed" images
generated by j_ima_iros. Three kinds of mosaic images can then be output
in a single FITS file: an intensity map of the combined weighted intensity
images, a weighted variance map, and the corresponding significance map.
Besides, it is also possible to make two kinds of exposure maps output
together in an independent FITS file: a raw observation time map and an
effective exposure map that takes account of the characteristics of the
JEM-X instruments, such as vignetting, dead time and grey filter effects.
One interesting possibility of j_ima_mosaic is its "zooming" function around a given sky position (inside the observed region) by use of the parameters "diameter" (of the mosaic image) and "radiusSelect" (of the input images). This makes possible to only consider the pixel contributions from the selected region. An example of a JEM-X mosaic image is given in Fig. 3.
Fig. 3: This JEM-X mosaic image shows the significance map in the 6-10 keV energy band obtained from eight individual pointings of a GCDE observation during INTEGRAL revolution 179. The strongest source detected is GX 5-1 at about 650 sigmas, while the faintest source GRS 1741-312 is detected at 19 sigmas.
The spectrum and light-curve extraction have also been updated with a more precise detector modelling and collimator description. It is now also possible to save a list with the Pixel Illumination Function values for each event by setting a parameter.
The gain correction algorithm has been updated to cope with the higher frequency of the so called 'glitches' - sudden drops in gain under the calibration sources that take up to half an hour to restabilize.
It has been noticed that due to the heavy irradiation of the small parts of the detector under the calibration source the gain development is a little different from the rest of the detector which leads to an apparent 'drift' of the Xe fluorescence line in energy with time. This has been included in the gain corrections to first order.
j_ima_iros requires two large (400 MB) IC files: The backprojection
lists (BPL, one for each instrument: jmx1_bpl_grp_0001.fits and
jmx2_bpl_grp_0001.fits) represent the "Aperture Response Function" i.e.
the description of the transparency between each pair of detector pixel
and sky pixel.
| OMC Changes & Improvements in OSA 5.0 | ||
|
|
A. Domingo Garau (LAEFF-INTA, Madrid) |
|
Considerable effort has been made since OSA 4 to improve the algorithms used to compute the fluxes. At the same time, the new algorithms allow the calculation of an accurate astrometric solution, compared with the OMC pixel size (17.504 arcsec).
We should notice that the implementation of the new functionality and improvements summarized here, has been a step by step process from the release of OSA 4. So, part of this new functionality and algorithms were implemented in OSA 4.1 and OSA 4.2.
It has been found that the old algorithm to compute the sky background produced poor results in the case of crowded fields and noisy shots. The new sky background is computed by using the 11x11 exterior pixel rim. The 12 brightest and the 8 faintest pixels are rejected to avoid cosmic rays, contamination by other sources and noisy pixels. This method has been tested to give good results in most cases: clean fields, crowded fields, noisy shots, high number of cosmic rays...
To be able to improve the flux calculation method, one important requirement is the knowledge of the source centroid with an accuracy as good as possible. This has been achieved by means of an iterative procedure, that minimize the residuals in each pixel according to a Point Spread Function (PSF) with Gaussian profile.
The PSF width is also computed. When fitting a PSF profile to estimate its FWHM, one should select stars with a high number of counts but not saturated, and of course, isolated stars. In the case of OMC data, these conditions are true for most of the photometric reference stars. So, after applying some few restrictions to select the best available candidates, the PSF width is estimated by using the photometric reference stars and performing an iterative procedure similar to that used in the computation of the centroids.
In previous OSA releases, the fluxes were computed by summing the flux in 1x1, 3x3 and 5x5 pixel squared areas. Since OSA 4.1, in the 3x3 and 5x5 apertures, partial pixels are used, dividing each real pixel in 4 sub-pixels. The areas have been circularized, removing the corners in 3x3 and 5x5, giving effective apertures of 8 and 19 pixel squared, respectively. These effective apertures are centered on the computed centroids. In Fig. 1 we can see the important improvement when using these apertures in the presence of close companions, as in Cyg X-1. On the right panel, the real variability of Cyg X-1 can be appreciated clearly.
Fig. 1: Computed light curve for Cyg X-1. Left panel: using neither partial pixels nor circularized photometric apertures. Right panel: using partial pixels and circularized photometric apertures. The OMC sub-window for Cyg X-1 is shown on the upper right corner of the graph.
Thanks to the good estimate of the PSF FWHM and the calculation of the centroids that are now available, photometric aperture corrections are calculated and applied to each one of the three flux estimates (1x1, 3x3 and 5x5). These corrections are performed individually for each source.
The accuracy reached when calculating the source centroids allow us the determination of a good astrometric solution. This astrometric solution has been implemented through a World Coordinate System (WCS) support: new columns (RA_FIN, DEC_FIN) for each source in the table of results of o_src_get_fluxes, and WCS header keywords in the output of the imaging tool (o_ima_build).
The WCS support is derived by fitting the best astrometric solution to the
faint photometric reference stars. A new solution is computed for each
effective integration. This allows to correct the inaccuracy due to the
thermoelastic deformations, which affect the alignment of the OMC optical
axis with the S/C attitude reference.
Thanks to the improvement achieved in the astrometric solution, a new OMC misalignment matrix has been calculated. By using this new misalignment matrix, the accuracy of the computed celestial coordinates is better than 2 arcsec in most cases, as it can be seen in Fig. 2.
Fig. 2: Distribution of the off-sets (in pixel units) between computed and real sky positions for all photometric reference stars observed in revolutions 161-170. The new misalignment matrix was used. Left panel: off-set in the X CCD coordinate (sigma = 0.08 pix = 1.4 arcsec). Right panel: off-set in the Y CCD coordinate (sigma = 0.07 pix = 1.2 arcsec).
This tool has been substantially updated to compute the WCS support as well as to make it more user friendly by adding new parameters. Now the user can specify the minimum and maximum values for the shot integration time, as well as the OMC_ID of the source he/she is interested in. If a given source is selected, small images are built containing only the OMC sub-windows corresponding to the selected source. An example of the output of this tool is shown in Fig. 3.
Fig. 3:
Image created with o_ima_build for IOMC 0466000051. It corresponds to the High
Mass X-ray Binary 4U 1901+03, which generates a mosaic of 5x5 OMC sub-windows.
The green cross and circle are the position and error of 4U 1901+03 in the
IBIS catalogue, respectively.
| Announcement of the 2nd INTEGRAL Data Analysis Workshop | ||
|
|
A. Neronov (ISDC) |
|
Three years after the launch and one and a half year after the first public data release, the "INTEGRAL sky" becomes more and more populated: new sources are found each month, both in the "real-time" and "off-line" regime (i.e. using the data from the INTEGRAL Archive). The INTEGRAL archive becomes a more interesting place for "data mining": for almost any direction in the sky the accumulated exposure time of the publicly available data grows each month. Even if you have little or no experience in working with INTEGRAL data, but you expect your source of interest to be active in the 10 keV - 1 MeV energy band, you may visit the 2nd INTEGRAL Data Analysis Workshop to learn how to use the Off-line Scientific Analysis (OSA) software package which provides easy-to-use methods for imaging, spectral and timing analysis of INTEGRAL data.
The Workshop will take place at the ISDC on October 12-15, 2005. The main goals of the workshop will be to introduce basic data analysis techniques and new developments in the 5th version of OSA. Special attention will be paied to the tools which enable quick extraction of results (mosaic images, source spectra) from the whole set of publicly available data in the INTEGRAL Archive.
The organization of the Workshop will be on a 2 + 1.5 days scheme. October 12-13 (main session) will be devoted to the presentation of the software and analysis techniques. October 14-15 (hands-on session) allowing participants who want to get practical experience in the data analysis to have the opportunity to use the computer facilities at ISDC and/or to get support for OSA software installation and set-up. For more details visit the web-page of the Workshop.
|
|
 |
 |