Integral
INTEGRAL Scripts
The Remeis INTEGRAL scripts can be found under
/software/Science/satscripts/intscripts/
on Remeis machines. If you have any questions on how to use them, read the following text and talk to Ingo or Philipp.
General remarks on Integral data extraction at Remeis:
Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis. The resulting spectra are then combined using spe_pick while the ISGRI/IBIS images are merged using INT_mosa_ibis. Generally it is a good idea, if you have not already to have a look at the OSA cookbook, e.g., to check how the paths in the data object locator (DOL) files is supposed to look like or how to prepare a suitable catalog for spectral extraction: https://www.isdc.unige.ch/integral/analysis
Before any OSA analysis can be run the corresponding module has to be load with:
ml integral-osa
Before running any data extraction you need to set up a few soft links:
ln -sf /home/X-ray/INTEGRAL/archive/aux ln -sf /home/X-ray/INTEGRAL/archive/ic ln -sf /home/X-ray/INTEGRAL/archive/idx ln -sf /home/X-ray/INTEGRAL/archive/scw
Typical IBIS spectral analysis
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
2) Start the ScW based analysis:
master_sbs --job=SPE --scwlist=scws_all.txt --source="Cyg X-1" --verbose --clobber=yes --path=$PWD --catalog=specat.fits
You can check whether your jobs are running or scheduled by SLURM by calling "squeue". 3) Generate the required input file for spe_pick. Check that that all objects in the DOL file exist.
txt2idx element="scws_dols.txt" index="cygx1"
4) Run spe_pick to generate the final spectra
spe_pick group="cygx1.fits" source="Cyg X-1" rootname="cygX1_all" instrument="ISGRI"
Typical IBIS image/mosaic
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
2) Start the ScW based analysis:
master_sbs --job=IMA --scwlist=scws_all.txt --energies="30,50,200" --verbose --clobber=yes --path=$PWD
3) Generate the required input file for spe_pick
txt2idx element="scws_dols.txt" index="cygx1"
4) Run INT_mosa_ibis to generate the final image
INT_mosa_ibis --obsgrp=$PWD --dol=scws_dols.txt --destin=$PWD/obs/mosa_cygx1
5) Check the images for artifacts and sort out affected ScWs through trial-and-error:
ds9 isgri_mosa_ima.fits[2] -region found.reg -cmap b -scale sqrt -scale limits 0 60 -zoom 2
Typical JEMX spectral analysis
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt
2) Generate a RMF with the required binning, e.g., 32 bins:
j_rebin_rmf binlist=STD_032
(Depending on your fitting software the resulting RMF might give you troubles and needs to be rewritte. ) 3) Start the ScW based analysis with 32=2^5 logarithmic energy bins:
master_sbs --path=$PWD --scwlist=scws_jemx.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5
4) Generate the required input file for spe_pick
txt2idx element="scws_jmx_dols.txt" index="cygx1"
5) Run spe_pick to generate the final spectra.
spe_pick group=cygx1.fits source="Cyg X-1" rootname="cygx1" instrument="JMX1"
JEMX images
To generate JEMX images there are no dedicated scripts at Remeis, and you have to fall back to the OSA scripts, e.g.:
og_create idxSwg=scw.lst ogid=oph2 baseDir="./" instrument=JMX1 jemx_science_analysis startLevel="IMA2" endLevel="IMA2" jemxNum= COR_gainModel=2
On catalogs
Adapted from the documentation: While for generally image generation a user generated catalog is not required, if supplied, it needs to keeps the same format of the general reference catalog from which it was built, i.e. it has the extension of the type ”GNRL-REFR-CAT”. This is the correct extension to be used as input for the Imaging step. In Section 7.2 of the OSA manual, you are been shown how to build a catalog for the lightcurve (LCR) and spectral (SPE) step. In that case the catalog, you obtained, specat.fits, was of the type of ”ISGR-SRCLRES” which is the correct format for the LCR and SPE steps. The two formats cannot be switched, meaning that you cannot use a ”GNRL-REFR-CAT” type catalog for SPE and LCR or an ”ISGR-SRCL-RES” one for IMA.
In practice one prepares the catalog used for spectral extraction from the results of the IMA step, taking only into account actually detected sources.
Quick Look Analysis
NRT data can also be analysed at Remeis with the usual scripts, require, however, an additional --nrt parameter in most scripts. Also have a look at http://isdc.unige.ch/heavens/
Documentation
Useful to know: http://www.isdc.unige.ch/integral/download/osa/doc/10.0/osa_um_ibis/node27.html And for the cook-book and general documentation. Reading the IBIS Analysis User Manual is recommend: https://www.isdc.unige.ch/integral/analysis Each INTEGTAL script also has a help
master_sbs --help cat $ISDC ENV/help/ibis_science_analysis.txt
Abbreviations
http://www.isdc.unige.ch/integral/support/glossary
- DAL The Data Access Layer (DAL) is a software library used to manipulate (create, write, read and manage) scientific data.
- DOL The Data Object Location (DOL) specifies the location of a DAL object, both by the FITS file name and its extension. For instance, /isdc/dev/xxx.fits[2] is a DOL, while xxx.fits is simply the name of a FITS file.
Known Issues
- In general: check first http://www.isdc.unige.ch/integral/support/faq for the error code you get
- The resulting spectra will not be strictly OGIP compliant, which e.g. ISIS might care about, but can be ignored.
Errors mosaicing using custom DOLs
If you want to create a mosaic using DOL (Data Object Location) files, sometimes the building interrupts with errors like a Fortran Runtime error (http://www.isdc.unige.ch/integral/support/faq.cgi?SOFT-051) or errors like:
Using and copying first SCW >>/scratch1/beuchert/integral/scw_extr/obs/125400400010/scw/125400400010.001/swg_ibis.fits << Log_0 : 100 elements opened. Log_0 : 100 elements attached. Error_1: Can not update the index.
or
Error_1: Can not close the element
or
Error_2: DAL_READ_ERROR : error reading from FITS file
etc.
- Somehow corrupted scws? Collect sets of two scws and create Mosaics based on those. Like that it is easy to enclose corrupted scws.
- An error -2004 (DAL_FILE_NOT_ACCESSIBLE) often results because one of your groups is corrupted. Before restarting the analysis, make a ``dal_clean, which should be able to do an automatic repair. If it does not work try a ``dal_verify to see what is corrupted. Or you can try to open your group (or child group) with fv and look for a file pointed by the group that does not exists. Most of the time removing the line with this wrong file in the group table is fixing your problem.
The latter: first create an observation group out of your dol-list of scws. Creating observation groups is also the canonical way of extracting INTEGRAL/IBIS data. Before doing that, the soft links as noted above have to be set in the directory. Then:
cd path/to/dir/with/softlinks/ og_create idxSwg=dol.txt ogid=isgri_gc baseDir="./" instrument=IBIS ls obs/isgri_gc/
Repair the observation group and included indices/links with
dal_clean inDOL="obs/isgri_gc/og_ibis.fits" chatty="4" checkExt="1"
Now INT_mosa_ibis should work again on the scws in dol.txt
How to create mosaics (outdated)
First make a list of scws you want to add to a mosaic, i.e., a "dol-file" dol.txt:
/scratch1/beuchert/integral/scw_extr/obs/066200340010/scw/066200340010.001/swg_ibis.fits /scratch1/beuchert/integral/scw_extr/obs/066200350010/scw/066200350010.001/swg_ibis.fits /scratch1/beuchert/integral/scw_extr/obs/066200360010/scw/066200360010.001/swg_ibis.fits
Then prepare the directory where the mosaic shall be built in, i.e., set the env. variables to allow INT_mosa_ibis to find indices etc.:
cd /some/path/ ln -s /virgo/b/INTEGRAL/aux/ ln -s /virgo/b/INTEGRAL/idx/ ln -s /virgo/b/INTEGRAL/ic/ ln -s /virgo/b/INTEGRAL/scw/ ln -s /virgo/b/INTEGRAL/cat/ mkdir /some/path/obs/ INT_mosa_ibis --obsgrp=/path/to/all/extracted/scws/ --destin=/path/to/the/prepared/directory/obs/mosa_og/ --dol="dol.txt"