Integral

From Remeis-Wiki
Jump to navigation Jump to search

INTEGRAL Scripts

The Remeis INTEGRAL scripts can be found under 

 /software/Science/satscripts/intscripts/

on Remeis machines. If you have any questions on how to use them, read the following text and talk to Ingo or Philipp.


General remarks on Integral data extraction at Remeis:

Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. This allows for the usage of SLURM and leads to significantly faster extraction compared to a combined extraction on a single machine. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis. The resulting spectra are then combined using spe_pick while the ISGRI/IBIS images are merged using INT_mosa_ibis. As the spectral extraction requires a catalog containing the sources for which the spectral extraction should be run, it is advised to first generate images & a mosaic to determine, which sources are bright and detected in the field of view. Generally it is a good idea, if you have not already to have a look at the OSA cookbook, e.g., to check how the paths in the data object locator (DOL) files is supposed to look like or how to prepare a suitable catalog for spectral extraction: https://www.isdc.unige.ch/integral/analysis

Before any OSA analysis can be run the corresponding module has to be load with: 

 ml integral-osa

Before running any data extraction you need to set up a few soft links: 

 ln -sf /home/X-ray/INTEGRAL/archive/aux
 ln -sf /home/X-ray/INTEGRAL/archive/ic
 ln -sf /home/X-ray/INTEGRAL/archive/idx
 ln -sf /home/X-ray/INTEGRAL/archive/scw


Typical IBIS image/mosaic

1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source
 objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt

If you want to use scws_all.txt as input for hte next step remove the header of this file, i.e, the first 6 lines. Importantly you will probalby want to select only a subset of the available observations and therefore perform some selection on the full list of science windows. As a simple example one might image to only use the latest 100 ScWs: 

 tail -n 100 scws_all.txt > scws_cut.txt
2) Start the ScW based analysis
 master_sbs  --job=IMA --scwlist=scws_all.txt --energies="30,50,200" --verbose --clobber=yes  --path=$PWD
3) Generate the required input file for spe_pick
 txt2idx element="scws_dols.txt" index="cygx1"
4) Run INT_mosa_ibis to generate the a combine mosaic
 INT_mosa_ibis --obsgrp=$PWD --dol=scws_dols.txt --destin=$PWD/obs/mosa_cygx1
5) Check the images for artifacts and sort out affected ScWs through trial-and-error
 ds9 isgri_mosa_ima.fits[2] -region found.reg -cmap b -scale sqrt -scale limits 0 60 -zoom 2

The resulting image should look something like this:


Mosa cygx1.png

Details about the detected sources, such as there flux and detection significance can be found in the file isgri_mosa_res.fits.

Typical IBIS spectral analysis

1) Again, find ScWs that cover your source and are of interest to you
2) Start the ScW based analysis
 master_sbs  --job=SPE --scwlist=scws_cut.txt --source="Cyg X-1"  --verbose --clobber=yes  --path=$PWD --catalog=specat.fits
3) Generate the required input file for spe_pick. Check that that all objects in the DOL file exist.
 txt2idx element="scws_dols.txt" index="cygx1"
4) Run spe_pick to generate the final spectra
 spe_pick group="cygx1.fits" source="Cyg X-1" rootname="cygX1_all" instrument="ISGRI"

This will result in background subtracted spectra for the given source. Remember that the source name has exactly the naming in the INTEGRAL reference catalog.

Typical IBIS light-curve generation

1) Quite similar to the spectral extraction. First find ScWs that cover your source, or reuse the previous ScW list
2) Start the ScW based analysis, here for 100s time binning
 master_sbs  --job=LC --scwlist=scws_cut.txt --source="Cyg X-1"  --verbose --clobber=yes  --path=$PWD --catalog=specat.fits --energies=30,120 --deltat=100

This generates a separate light-curve for each ScW at, e.g., obs/250300170010/scw/250300170010.001/250300170010_Cyg_X-1_30-120.lc

3) These can then be merged together e.g. using fmerge or ISIS.
 ls obs/*/scw/*.001/*_Cyg_X-1_30-120.lc >> lcs.txt
 fmerge @lcs.txt Cyg_X1_ibis_lc.fits -

Typical IBIS PIF light-curve generation

The above light-curves are only reliable down to a time-resolution of 1.0-0.1s. For higher timing resolution the deconvolution becomes unstable and one has to generate light-curves from the event-lists and pixel-illumination fraction (PIF).

1) First event lists are generated with the respective Remeis Scripts
 eventlist --path=$PWD --source="Cyg X-1" --scw=$scw --catalog=$cat

This generates a event-list under "obs/266100220010/scw/266100220010.001/events.fits". For each source in the catalog and the FOV this file includes a column indicating whether a source is illuminated by the respective source.

2) Filter the eventfile e.g. with an ftools or an ISIS.
 fcopy "events.fits[2][PIF_1==1]" cyg_pif1.fits
3) Light-curves can then be generated e.g. with ISIS (lc_from_events)
 variable lc = lc_from_events(struct{time=evts.time[n], pi=evts.energy[n]};dt=dt,
 gti = fits_read_table(gtiname + "[14]"), minfracexp=.5);

Typical JEMX spectral analysis

Until revolution 976 (Oct. 10,2010) only one JEM-X unit was operating at any give time. During that time one should therefore run the analysis for both detectors.

1) Find ScWs that cover your source. As JEMX has a smaller field of view, its max distance to the source should also be reduced to ~5 degree
 objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt

And again, you will probably further select in which ScWs you are interested.

2) Generate a RMF with the required binning, e.g., 32 bins
 j_rebin_rmf  binlist=STD_032

The RMF does not change between observations and might be reused as long as binning requirements do not change. (Depending on your fitting software the resulting RMF might give you troubles and needs to be rewritten.)

3) Start the ScW based analysis with 32=2^5 logarithmic energy bins
 master_sbs  --path=$PWD --scwlist=scws_jemx_cut.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5
4) Generate the required input file for spe_pick
 txt2idx element="scws_jmx_dols.txt" index="cygx1"
5) Run spe_pick to generate the final spectra.
 spe_pick group=cygx1.fits source="Cyg X-1" rootname="cygx1" instrument="JMX1"

JEMX images

To generate JEMX images there are no dedicated scripts at Remeis, and you have to fall back to the OSA scripts, e.g.: 

 og_create idxSwg=scw.lst ogid=oph2 baseDir="./" instrument=JMX1
 jemx_science_analysis startLevel="IMA2" endLevel="IMA2" jemxNum=1 COR_gainModel=2

On catalogs

Adapted from the documentation: While for generally image generation a user generated catalog is not required, if supplied, it needs to keeps the same format of the general reference catalog from which it was built, i.e. it has the extension of the type ”GNRL-REFR-CAT”. This is the correct extension to be used as input for the Imaging step. In Section 7.2 of the OSA manual, you are been shown how to build a catalog for the lightcurve (LCR) and spectral (SPE) step. In that case the catalog, you obtained, specat.fits, was of the type of ”ISGR-SRCLRES” which is the correct format for the LCR and SPE steps. The two formats cannot be switched, meaning that you cannot use a ”GNRL-REFR-CAT” type catalog for SPE and LCR or an ”ISGR-SRCL-RES” one for IMA.


In practice one prepares the catalog used for spectral extraction from the results of the IMA step, taking only into account actually detected sources, e.g. by running the following on the results of INT_mosa_ibis: 

fcopy "isgri_srcl_res.fits[1][DETSIG>10]" specat.fits

Time selection

User generated GTI files can be used to perform time or phase resolved spectral analysis. This is done through the --gtiname & --gtiformat keywords. Consult the master_sbs help for more details. To generate GTI file it is recommended to make use of the OSA script gti_user, e.g., as follows: 

 gti_user tstart=8170.5 tstop=8171.8 unit=sec begin=0.0 end=50 step=150 repeat=653 gti=user_gti.fits clobber=yes

Or for a to calculate a reverse bary-corrected GTI file. Not that this requires the coordinated and the DOL file of a ScW or Observation Group. 

 gti_user tstart=8393.16 unit=sec begin=0.0 end=50 step=25 repeat=600 gti=user_gti.fits clobber=yes bary=y group=og_ibis.fits+1

Also note that the light-curves generated by imalc_sbs through master_sbs are already barycenter corrected.


More details can be found in the introduction to INTEGRAL analysis:

https://www.isdc.unige.ch/integral/download/osa/doc/11.2/osa_um_intro.pdf

Quick Look Analysis

NRT data can also be analysed at Remeis with the usual scripts, require, however, an additional --nrt parameter in most scripts. Also have a look at http://isdc.unige.ch/heavens/

Documentation

Useful to know: http://www.isdc.unige.ch/integral/download/osa/doc/10.0/osa_um_ibis/node27.html And for the cook-book and general documentation. Reading the IBIS Analysis User Manual is recommend: https://www.isdc.unige.ch/integral/analysis Each INTEGTAL script also has a help 

 master_sbs --help
 cat $ISDC_ENV/help/ibis_science_analysis.txt

Abbreviations

http://www.isdc.unige.ch/integral/support/glossary

  • DAL The Data Access Layer (DAL) is a software library used to manipulate (create, write, read and manage) scientific data.
  • DOL The Data Object Location (DOL) specifies the location of a DAL object, both by the FITS file name and its extension. For instance, /isdc/dev/xxx.fits[2] is a DOL, while xxx.fits is simply the name of a FITS file.

Known Issues

  1. In general: check first http://www.isdc.unige.ch/integral/support/faq for the error code you get
  2. The resulting spectra will not be strictly OGIP compliant, which e.g. ISIS might care about, but can be ignored.

Errors mosaicing using custom DOLs

If you want to create a mosaic using DOL (Data Object Location) files, sometimes the building interrupts with errors like a Fortran Runtime error (http://www.isdc.unige.ch/integral/support/faq.cgi?SOFT-051) or errors like: 

 Using and copying first SCW >>/scratch1/beuchert/integral/scw_extr/obs/125400400010/scw/125400400010.001/swg_ibis.fits
 <<
 Log_0  :  100 elements opened.
 Log_0  :  100 elements attached.
 Error_1: Can not update the index.

or 

 Error_1: Can not close the element

or 

 Error_2: DAL_READ_ERROR : error reading from FITS file

etc.


  1. Somehow corrupted scws? Collect sets of two scws and create Mosaics based on those. Like that it is easy to enclose corrupted scws.
  2. An error -2004 (DAL_FILE_NOT_ACCESSIBLE) often results because one of your groups is corrupted. Before restarting the analysis, make a ``dal_clean, which should be able to do an automatic repair. If it does not work try a ``dal_verify to see what is corrupted. Or you can try to open your group (or child group) with fv and look for a file pointed by the group that does not exists. Most of the time removing the line with this wrong file in the group table is fixing your problem.

The latter: first create an observation group out of your dol-list of scws. Creating observation groups is also the canonical way of extracting INTEGRAL/IBIS data. Before doing that, the soft links as noted above have to be set in the directory. Then: 

 cd path/to/dir/with/softlinks/
 og_create idxSwg=dol.txt ogid=isgri_gc baseDir="./" instrument=IBIS
 ls obs/isgri_gc/

Repair the observation group and included indices/links with 

 dal_clean inDOL="obs/isgri_gc/og_ibis.fits" chatty="4" checkExt="1"

Now INT_mosa_ibis should work again on the scws in dol.txt

How to create mosaics (outdated)

First make a list of scws you want to add to a mosaic, i.e., a "dol-file" dol.txt: 

 /scratch1/beuchert/integral/scw_extr/obs/066200340010/scw/066200340010.001/swg_ibis.fits
 /scratch1/beuchert/integral/scw_extr/obs/066200350010/scw/066200350010.001/swg_ibis.fits
 /scratch1/beuchert/integral/scw_extr/obs/066200360010/scw/066200360010.001/swg_ibis.fits

Then prepare the directory where the mosaic shall be built in, i.e., set the env. variables to allow INT_mosa_ibis to find indices etc.: 

 cd /some/path/
 ln -s /virgo/b/INTEGRAL/aux/
 ln -s /virgo/b/INTEGRAL/idx/
 ln -s /virgo/b/INTEGRAL/ic/
 ln -s /virgo/b/INTEGRAL/scw/
 ln -s /virgo/b/INTEGRAL/cat/
 mkdir /some/path/obs/
 
 INT_mosa_ibis --obsgrp=/path/to/all/extracted/scws/ --destin=/path/to/the/prepared/directory/obs/mosa_og/ --dol="dol.txt"

Non-OSA INTEGRAL extraction

There are a few alternatives by now to using OSA+Remeis scripts:

  • To run IBIS and JEMX analysis online there is MMODA at https://www.astro.unige.ch/mmoda/help/mmoda/integral-mission
  • If you are interested in SPI data you can do this with the OSA installed here at Remeis, there are however much more reliable and stable pipelines available in form of either SPI-DAI or the SPI-pipeline at MPE organized by Tomas Stiegert.
Retrieved from "https://www.sternwarte.uni-erlangen.de/wiki/index.php?title=Integral&oldid=3386"