Integral

INTEGRAL Scripts

The Remeis INTEGRAL scripts can be found under

 /software/Science/satscripts/intscripts/

on Remeis machines. If you have any questions on how to use them, read the following text and talk to Ingo or Philipp.

General remarks on Integral data extraction at Remeis:

Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis. The resulting spectra a then combined using spe_pick while the ISGRI/IBIS images are merged using INT_mose_ibis. Generally it is a good idea, if you have not already to have a look at the OSA cookbook, e.g., to check how the paths in the data object locator (DOL) files is supposed to look like or how to prepare a suitable catalog for spectral extraction: https://www.isdc.unige.ch/integral/analysis

Before any OSA analysis can be run the corresponding module has to be load with:

 ml integral-osa

Before running any data extraction you need to set up a few soft links:

 ln -sf /home/X-ray/INTEGRAL/archive/aux
 ln -sf /home/X-ray/INTEGRAL/archive/ic
 ln -sf /home/X-ray/INTEGRAL/archive/idx
 ln -sf /home/X-ray/INTEGRAL/archive/scw

Typical IBIS spectral analysis

1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:

 objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt

2) Start the ScW based analysis:

 master_sbs  --job=SPE --scwlist=scws_all.txt --source="Cyg X-1"  --verbose --clobber=yes  --path=$PWD --catalog=specat.fits

You can check whether your jobs are running or scheduled by SLURM by calling "squeue". 3) Generate the required input file for spe_pick. Check that that all objects in the DOL file exist.

 txt2idx element="scws_dols.txt" index="cygx1"

4) Run spe_pick to generate the final spectra

 spe_pick group="cygx1.fits" source="Cyg X-1" rootname="cygX1_all" instrument="ISGRI"

Typical IBIS image/mosaic

1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:

 objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt

2) Start the ScW based analysis:

 master_sbs  --job=IMA --scwlist=scws_all.txt --energies="30,50,200" --verbose --clobber=yes  --path=$PWD 

3) Generate the required input file for spe_pick

 txt2idx element="scws_dols.txt" index="cygx1"

4) Run INT_mosa_ibis to generate the final image

 INT_mosa_ibis --obsgrp=$PWD --dol=scws_dols.txt --destin=$PWD/obs/mosa_cygx1

5) Check the images for artifacts and sort out affected ScWs through trial-and-error:

 ds9 isgri_mosa_ima.fits[2] -region found.reg -cmap b -scale sqrt -scale limits 0 60 -zoom 2

Typical JEMX spectral analysis

1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:

 objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt

2) Generate a RMF with the required binning, e.g., 32 bins:

 j_rebin_rmf  binlist=STD_032

(Depending on your fitting software the resulting RMF might give you troubles and needs to be rewritte. ) 3) Start the ScW based analysis with 32=2^5 logarithmic energy bins:

 master_sbs  --path=$PWD --scwlist=scws_jemx.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5

4) Generate the required input file for spe_pick

 txt2idx element="scws_jmx_dols.txt" index="cygx1"

5) Run spe_pick to generate the final spectra.

 spe_pick group=cygx1.fits source="Cyg X-1" rootname="cygx1" instrument="JMX1"

JEMX images

To generate JEMX images there are no dedicated scripts at Remeis, and you have to fall back to the OSA scripts, e.g.:

 og_create idxSwg=scw.lst ogid=oph2 baseDir="./" instrument=JMX1
 jemx_science_analysis startLevel="IMA2" endLevel="IMA2"  jemxNum= COR_gainModel=2

Quick Look Analysis

NRT data can also be analysed at Remeis with the ususal scipts, require, however, an additional --nrt parameter in most scripts. Also have a look at http://isdc.unige.ch/heavens/

Documentation

For common questions: http://www.isdc.unige.ch/integral/download/osa/doc/10.0/osa_um_ibis/node27.html And for the cook-book and general documentation. Reading the IBIS Analysis User Manual is recommend: https://www.isdc.unige.ch/integral/analysis

Abbreviations

http://www.isdc.unige.ch/integral/support/glossary

• DAL The Data Access Layer (DAL) is a software library used to manipulate (create, write, read and manage) scientific data.
• DOL The Data Object Location (DOL) specifies the location of a DAL object, both by the FITS file name and its extension. For instance, /isdc/dev/xxx.fits[2] is a DOL, while xxx.fits is simply the name of a FITS file.

Known Issues

1. In general: check first http://www.isdc.unige.ch/integral/support/faq for the error code you get
2. The resulting spectra will not be strictly OGIP compliant, which e.g. ISIS might care about, but can be ignored.

Errors mosaicing using custom DOLs

If you want to create a mosaic using DOL (Data Object Location) files, sometimes the building interrupts with errors like a Fortran Runtime error (http://www.isdc.unige.ch/integral/support/faq.cgi?SOFT-051) or errors like:

 Using and copying first SCW >>/scratch1/beuchert/integral/scw_extr/obs/125400400010/scw/125400400010.001/swg_ibis.fits
 <<
 Log_0  :  100 elements opened.
 Log_0  :  100 elements attached.
 Error_1: Can not update the index.

or

 Error_1: Can not close the element

or

 Error_2: DAL_READ_ERROR : error reading from FITS file

etc.

1. Somehow corrupted scws? Collect sets of two scws and create Mosaics based on those. Like that it is easy to enclose corrupted scws.
2. An error -2004 (DAL_FILE_NOT_ACCESSIBLE) often results because one of your groups is corrupted. Before restarting the analysis, make a ``dal_clean, which should be able to do an automatic repair. If it does not work try a ``dal_verify to see what is corrupted. Or you can try to open your group (or child group) with fv and look for a file pointed by the group that does not exists. Most of the time removing the line with this wrong file in the group table is fixing your problem.

The latter: first create an observation group out of your dol-list of scws. Creating observation groups is also the canonical way of extracting INTEGRAL/IBIS data. Before doing that, the soft links as noted above have to be set in the directory. Then:

 cd path/to/dir/with/softlinks/
 og_create idxSwg=dol.txt ogid=isgri_gc baseDir="./" instrument=IBIS
 ls obs/isgri_gc/

Repair the observation group and included indices/links with

 dal_clean inDOL="obs/isgri_gc/og_ibis.fits" chatty="4" checkExt="1"

Now INT_mosa_ibis should work again on the scws in dol.txt

How to create mosaics (outdated)

First make a list of scws you want to add to a mosaic, i.e., a "dol-file" dol.txt:

 /scratch1/beuchert/integral/scw_extr/obs/066200340010/scw/066200340010.001/swg_ibis.fits
 /scratch1/beuchert/integral/scw_extr/obs/066200350010/scw/066200350010.001/swg_ibis.fits
 /scratch1/beuchert/integral/scw_extr/obs/066200360010/scw/066200360010.001/swg_ibis.fits

Then prepare the directory where the mosaic shall be built in, i.e., set the env. variables to allow INT_mosa_ibis to find indices etc.:

 cd /some/path/
 ln -s /virgo/b/INTEGRAL/aux/
 ln -s /virgo/b/INTEGRAL/idx/
 ln -s /virgo/b/INTEGRAL/ic/
 ln -s /virgo/b/INTEGRAL/scw/
 ln -s /virgo/b/INTEGRAL/cat/
 mkdir /some/path/obs/
 
 INT_mosa_ibis --obsgrp=/path/to/all/extracted/scws/ --destin=/path/to/the/prepared/directory/obs/mosa_og/ --dol="dol.txt"