Integral
INTEGRAL Scripts
The Remeis INTEGRAL scripts can be found under
/software/Science/satscripts/intscripts/
on Remeis machines. If you have any questions on how to use them, read the following text and talk to Ingo or Philipp.
General remarks on Integral data extraction at Remeis:
Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis. The resulting spectra a then combined using spe_pick while the ISGRI/IBIS images are merged using INT_mose_ibis. Generally it is a good idea, if you have not already to have a look at the OSA cookbook, e.g., to check how the paths in the data object locator (DOL) files is supposed to look like or how to prepare a suitable catalog for spectral extraction: https://www.isdc.unige.ch/integral/analysis
Before any OSA analysis can be run the corresponding module has to be load with:
ml integral-osa
Before running any data extraction you need to set up a few soft links:
ln -sf /home/X-ray/INTEGRAL/archive/aux ln -sf /home/X-ray/INTEGRAL/archive/ic ln -sf /home/X-ray/INTEGRAL/archive/idx ln -sf /home/X-ray/INTEGRAL/archive/scw
Typical IBIS spectral analysis
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
2) Start the ScW based analysis:
master_sbs --job=SPE --scwlist=scws_all.txt --source="Cyg X-1" --verbose --clobber=yes --path=$PWD --catalog=specat.fits
You can check whether your jobs are running or scheduled by SLURM by calling "squeue". 3) Generate the required input file for spe_pick. Check that that all objects in the DOL file exist.
txt2idx element="scws_dols.txt" index="cygx1"
4) Run spe_pick to generate the final spectra
spe_pick group="cygx1.fits" source="Cyg X-1" rootname="cygX1_all" instrument="ISGRI"
Typical IBIS image/mosaic
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
2) Start the ScW based analysis:
master_sbs --job=IMA --scwlist=scws_all.txt --energies="30,50,200" --verbose --clobber=yes --path=$PWD
3) Generate the required input file for spe_pick
txt2idx element="scws_dols.txt" index="cygx1"
4) Run INT_mosa_ibis to generate the final image
INT_mosa_ibis --obsgrp=$PWD --dol=scws_dols.txt --destin=$PWD/obs/mosa_cygx1
5) Check the images for artifacts and sort out affected ScWs through trial-and-error:
ds9 isgri_mosa_ima.fits[2] -region found.reg -cmap b -scale sqrt -scale limits 0 60 -zoom 2
Typical JEMX spectral analysis
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt
2) Generate a RMF with the required binning, e.g., 32 bins:
j_rebin_rmf binlist=STD_032
(Depending on your fitting software the resulting RMF might give you troubles and needs to be rewritte. ) 3) Start the ScW based analysis with 32=2^5 logarithmic energy bins:
master_sbs --path=$PWD --scwlist=scws_jemx.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5
4) Generate the required input file for spe_pick
txt2idx element="scws_jmx_dols.txt" index="cygx1"
5) Run spe_pick to generate the final spectra.
spe_pick group=cygx1.fits source="Cyg X-1" rootname="cygx1" instrument="JMX1"
JEMX images
To generate JEMX images there are no dedicated scripts at Remeis, and you have to fall back to the OSA scripts, e.g.:
og_create idxSwg=scw.lst ogid=oph2 baseDir="./" instrument=JMX1 jemx_science_analysis startLevel="IMA2" endLevel="IMA2" jemxNum= COR_gainModel=2
Quick Look Analysis
NRT data can also be analysed at Remeis with the ususal scipts, require, however, an additional --nrt parameter in most scripts. Also have a look at http://isdc.unige.ch/heavens/
Documentation
For common questions: http://www.isdc.unige.ch/integral/download/osa/doc/10.0/osa_um_ibis/node27.html And for the cook-book and general documentation. Reading the IBIS Analysis User Manual is recommend: https://www.isdc.unige.ch/integral/analysis
Abbreviations
http://www.isdc.unige.ch/integral/support/glossary
- DAL The Data Access Layer (DAL) is a software library used to manipulate (create, write, read and manage) scientific data.
- DOL The Data Object Location (DOL) specifies the location of a DAL object, both by the FITS file name and its extension. For instance, /isdc/dev/xxx.fits[2] is a DOL, while xxx.fits is simply the name of a FITS file.
Known Issues
- In general: check first http://www.isdc.unige.ch/integral/support/faq for the error code you get
- The resulting spectra will not be strictly OGIP compliant, which e.g. ISIS might care about, but can be ignored.
Errors mosaicing using custom DOLs
If you want to create a mosaic using DOL (Data Object Location) files, sometimes the building interrupts with errors like a Fortran Runtime error (http://www.isdc.unige.ch/integral/support/faq.cgi?SOFT-051) or errors like:
Using and copying first SCW >>/scratch1/beuchert/integral/scw_extr/obs/125400400010/scw/125400400010.001/swg_ibis.fits << Log_0 : 100 elements opened. Log_0 : 100 elements attached. Error_1: Can not update the index.
or
Error_1: Can not close the element
or
Error_2: DAL_READ_ERROR : error reading from FITS file
etc.
- Somehow corrupted scws? Collect sets of two scws and create Mosaics based on those. Like that it is easy to enclose corrupted scws.
- An error -2004 (DAL_FILE_NOT_ACCESSIBLE) often results because one of your groups is corrupted. Before restarting the analysis, make a ``dal_clean, which should be able to do an automatic repair. If it does not work try a ``dal_verify to see what is corrupted. Or you can try to open your group (or child group) with fv and look for a file pointed by the group that does not exists. Most of the time removing the line with this wrong file in the group table is fixing your problem.
The latter: first create an observation group out of your dol-list of scws. Creating observation groups is also the canonical way of extracting INTEGRAL/IBIS data. Before doing that, the soft links as noted above have to be set in the directory. Then:
cd path/to/dir/with/softlinks/ og_create idxSwg=dol.txt ogid=isgri_gc baseDir="./" instrument=IBIS ls obs/isgri_gc/
Repair the observation group and included indices/links with
dal_clean inDOL="obs/isgri_gc/og_ibis.fits" chatty="4" checkExt="1"
Now INT_mosa_ibis should work again on the scws in dol.txt
How to create mosaics (outdated)
First make a list of scws you want to add to a mosaic, i.e., a "dol-file" dol.txt:
/scratch1/beuchert/integral/scw_extr/obs/066200340010/scw/066200340010.001/swg_ibis.fits /scratch1/beuchert/integral/scw_extr/obs/066200350010/scw/066200350010.001/swg_ibis.fits /scratch1/beuchert/integral/scw_extr/obs/066200360010/scw/066200360010.001/swg_ibis.fits
Then prepare the directory where the mosaic shall be built in, i.e., set the env. variables to allow INT_mosa_ibis to find indices etc.:
cd /some/path/ ln -s /virgo/b/INTEGRAL/aux/ ln -s /virgo/b/INTEGRAL/idx/ ln -s /virgo/b/INTEGRAL/ic/ ln -s /virgo/b/INTEGRAL/scw/ ln -s /virgo/b/INTEGRAL/cat/ mkdir /some/path/obs/ INT_mosa_ibis --obsgrp=/path/to/all/extracted/scws/ --destin=/path/to/the/prepared/directory/obs/mosa_og/ --dol="dol.txt"