Difference between revisions of "Integral"
Ballhausen (talk | contribs) (Created page with " ====== Quick Look Analysis ====== http://isdc.unige.ch/heavens/ How does the INTEGRAL analysis software work in general? http://www.isdc.unige.ch/integral/download/osa/doc/...") |
Thalhammer (talk | contribs) (Update the INTEGRAL data extraction) |
||
Line 1: | Line 1: | ||
− | === | + | === INTEGRAL Scripts === |
− | + | The Remeis INTEGRAL scripts can be found under | |
− | + | /software/Science/satscripts/intscripts/ | |
− | + | ||
+ | on Remeis machines. If you have any questions on how to use them, read the following text and talk to Ingo or Philipp. | ||
+ | |||
+ | |||
+ | === General remarks on Integral data extraction at Remeis: === | ||
+ | |||
+ | Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis. | ||
+ | The resulting spectra a then combined using spe_pick while the ISGRI/IBIS images are merged using INT_mose_ibis. Generally it is a good idea, if you have not already to have a look at the OSA cookbook, e.g., to check how the paths in the data object locator (DOL) files is supposed to look like or how to prepare a suitable catalog for spectral extraction: https://www.isdc.unige.ch/integral/analysis | ||
+ | |||
+ | Before any OSA analysis can be run the corresponding module has to be load with: | ||
+ | ml integral-osa | ||
+ | |||
+ | Before running any data extraction you need to set up a few soft links: | ||
+ | ln -sf /home/X-ray/INTEGRAL/archive/aux | ||
+ | ln -sf /home/X-ray/INTEGRAL/archive/ic | ||
+ | ln -sf /home/X-ray/INTEGRAL/archive/idx | ||
+ | ln -sf /home/X-ray/INTEGRAL/archive/scw | ||
+ | |||
+ | |||
+ | === Typical IBIS spectral analysis === | ||
+ | 1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source: | ||
+ | objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt | ||
+ | 2) Start the ScW based analysis: | ||
+ | master_sbs --job=SPE --scwlist=scws_all.txt --source="Cyg X-1" --verbose --clobber=yes --path=$PWD --catalog=specat.fits | ||
+ | You can check whether your jobs are running or scheduled by SLURM by calling "squeue". | ||
+ | 3) Generate the required input file for spe_pick. Check that that all objects in the DOL file exist. | ||
+ | txt2idx element="scws_dols.txt" index="cygx1" | ||
+ | 4) Run spe_pick to generate the final spectra | ||
+ | spe_pick group="cygx1.fits" source="Cyg X-1" rootname="cygX1_all" instrument="ISGRI" | ||
+ | |||
+ | === Typical IBIS image/mosaic === | ||
+ | 1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source: | ||
+ | objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt | ||
+ | 2) Start the ScW based analysis: | ||
+ | master_sbs --job=IMA --scwlist=scws_all.txt --energies="30,50,200" --verbose --clobber=yes --path=$PWD | ||
+ | 3) Generate the required input file for spe_pick | ||
+ | txt2idx element="scws_dols.txt" index="cygx1" | ||
+ | 4) Run INT_mosa_ibis to generate the final image | ||
+ | INT_mosa_ibis --obsgrp=$PWD --dol=scws_dols.txt --destin=$PWD/obs/mosa_cygx1 | ||
+ | 5) Check the images for artifacts and sort out affected ScWs through trial-and-error: | ||
+ | ds9 isgri_mosa_ima.fits[2] -region found.reg -cmap b -scale sqrt -scale limits 0 60 -zoom 2 | ||
+ | |||
+ | |||
+ | === Typical JEMX spectral analysis === | ||
+ | 1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source: | ||
+ | objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt | ||
+ | 2) Generate a RMF with the required binning, e.g., 32 bins: | ||
+ | j_rebin_rmf binlist=STD_032 | ||
+ | (Depending on your fitting software the resulting RMF might give you troubles and needs to be rewritte. ) | ||
+ | 3) Start the ScW based analysis with 32=2^5 logarithmic energy bins: | ||
+ | master_sbs --path=$PWD --scwlist=scws_jemx.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5 | ||
+ | 4) Generate the required input file for spe_pick | ||
+ | txt2idx element="scws_jmx_dols.txt" index="cygx1" | ||
+ | 5) Run spe_pick to generate the final spectra. | ||
+ | spe_pick group=cygx1.fits source="Cyg X-1" rootname="cygx1" instrument="JMX1" | ||
− | ====== | + | === JEMX images === |
+ | To generate JEMX images there are no dedicated scripts at Remeis, and you have to fall back to the OSA scripts, e.g.: | ||
+ | og_create idxSwg=scw.lst ogid=oph2 baseDir="./" instrument=JMX1 | ||
+ | jemx_science_analysis startLevel="IMA2" endLevel="IMA2" jemxNum= COR_gainModel=2 | ||
− | |||
− | |||
− | + | === Quick Look Analysis === | |
− | + | NRT data can also be analysed at Remeis with the ususal scipts, require, however, an additional --nrt parameter in most scripts. | |
+ | Also have a look at http://isdc.unige.ch/heavens/ | ||
− | + | === Documentation === | |
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
− | |||
+ | For common questions: | ||
+ | http://www.isdc.unige.ch/integral/download/osa/doc/10.0/osa_um_ibis/node27.html | ||
+ | And for the cook-book and general documentation. Reading the IBIS Analysis User Manual is recommend: | ||
+ | https://www.isdc.unige.ch/integral/analysis | ||
− | === | + | === Abbreviations === |
http://www.isdc.unige.ch/integral/support/glossary | http://www.isdc.unige.ch/integral/support/glossary | ||
Line 39: | Line 86: | ||
* '''DOL''' The Data Object Location (DOL) specifies the location of a DAL object, both by the FITS file name and its extension. For instance, /isdc/dev/xxx.fits[2] is a DOL, while xxx.fits is simply the name of a FITS file. | * '''DOL''' The Data Object Location (DOL) specifies the location of a DAL object, both by the FITS file name and its extension. For instance, /isdc/dev/xxx.fits[2] is a DOL, while xxx.fits is simply the name of a FITS file. | ||
− | + | === Known Issues === | |
− | In general: check first http://www.isdc.unige.ch/integral/support/faq for the error code you get | + | # In general: check first http://www.isdc.unige.ch/integral/support/faq for the error code you get |
+ | # The resulting spectra will not be strictly OGIP compliant, which e.g. ISIS might care about, but can be ignored. | ||
− | === | + | ==== Errors mosaicing using custom DOLs ==== |
If you want to create a mosaic using DOL (Data Object Location) files, sometimes the building interrupts with errors like a Fortran Runtime error (http://www.isdc.unige.ch/integral/support/faq.cgi?SOFT-051) or errors like: | If you want to create a mosaic using DOL (Data Object Location) files, sometimes the building interrupts with errors like a Fortran Runtime error (http://www.isdc.unige.ch/integral/support/faq.cgi?SOFT-051) or errors like: | ||
Line 57: | Line 105: | ||
etc. | etc. | ||
− | + | ||
# Somehow corrupted scws? Collect sets of two scws and create Mosaics based on those. Like that it is easy to enclose corrupted scws. | # Somehow corrupted scws? Collect sets of two scws and create Mosaics based on those. Like that it is easy to enclose corrupted scws. | ||
# An error -2004 (DAL_FILE_NOT_ACCESSIBLE) often results because one of your groups is corrupted. Before restarting the analysis, make a ``dal_clean'', which should be able to do an automatic repair. If it does not work try a ``dal_verify'' to see what is corrupted. Or you can try to open your group (or child group) with fv and look for a file pointed by the group that does not exists. Most of the time removing the line with this wrong file in the group table is fixing your problem. | # An error -2004 (DAL_FILE_NOT_ACCESSIBLE) often results because one of your groups is corrupted. Before restarting the analysis, make a ``dal_clean'', which should be able to do an automatic repair. If it does not work try a ``dal_verify'' to see what is corrupted. Or you can try to open your group (or child group) with fv and look for a file pointed by the group that does not exists. Most of the time removing the line with this wrong file in the group table is fixing your problem. | ||
Line 72: | Line 120: | ||
dal_clean inDOL="obs/isgri_gc/og_ibis.fits" chatty="4" checkExt="1" | dal_clean inDOL="obs/isgri_gc/og_ibis.fits" chatty="4" checkExt="1" | ||
Now INT_mosa_ibis should work again on the scws in dol.txt | Now INT_mosa_ibis should work again on the scws in dol.txt | ||
+ | |||
+ | ==== How to create mosaics (outdated) ==== | ||
+ | |||
+ | First make a list of scws you want to add to a mosaic, i.e., a "dol-file" dol.txt: | ||
+ | /scratch1/beuchert/integral/scw_extr/obs/066200340010/scw/066200340010.001/swg_ibis.fits | ||
+ | /scratch1/beuchert/integral/scw_extr/obs/066200350010/scw/066200350010.001/swg_ibis.fits | ||
+ | /scratch1/beuchert/integral/scw_extr/obs/066200360010/scw/066200360010.001/swg_ibis.fits | ||
+ | Then prepare the directory where the mosaic shall be built in, i.e., set the env. variables to allow INT_mosa_ibis to find indices etc.: | ||
+ | cd /some/path/ | ||
+ | ln -s /virgo/b/INTEGRAL/aux/ | ||
+ | ln -s /virgo/b/INTEGRAL/idx/ | ||
+ | ln -s /virgo/b/INTEGRAL/ic/ | ||
+ | ln -s /virgo/b/INTEGRAL/scw/ | ||
+ | ln -s /virgo/b/INTEGRAL/cat/ | ||
+ | mkdir /some/path/obs/ | ||
+ | |||
+ | INT_mosa_ibis --obsgrp=/path/to/all/extracted/scws/ --destin=/path/to/the/prepared/directory/obs/mosa_og/ --dol="dol.txt" | ||
+ | |||
+ | |||
[[Category:Data Extraction]] | [[Category:Data Extraction]] |
Revision as of 11:21, 26 May 2023
INTEGRAL Scripts
The Remeis INTEGRAL scripts can be found under
/software/Science/satscripts/intscripts/
on Remeis machines. If you have any questions on how to use them, read the following text and talk to Ingo or Philipp.
General remarks on Integral data extraction at Remeis:
Initially data is extracted in parallel on a ScWs basis and in a second step combined to the final products. The initial step is launch by the scripts master_sbs. The script can generate spectra and images for ISGRI and spectra for JEMX. No PICsIT, SPI or OMC analysis is run at Remeis. The resulting spectra a then combined using spe_pick while the ISGRI/IBIS images are merged using INT_mose_ibis. Generally it is a good idea, if you have not already to have a look at the OSA cookbook, e.g., to check how the paths in the data object locator (DOL) files is supposed to look like or how to prepare a suitable catalog for spectral extraction: https://www.isdc.unige.ch/integral/analysis
Before any OSA analysis can be run the corresponding module has to be load with:
ml integral-osa
Before running any data extraction you need to set up a few soft links:
ln -sf /home/X-ray/INTEGRAL/archive/aux ln -sf /home/X-ray/INTEGRAL/archive/ic ln -sf /home/X-ray/INTEGRAL/archive/idx ln -sf /home/X-ray/INTEGRAL/archive/scw
Typical IBIS spectral analysis
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
2) Start the ScW based analysis:
master_sbs --job=SPE --scwlist=scws_all.txt --source="Cyg X-1" --verbose --clobber=yes --path=$PWD --catalog=specat.fits
You can check whether your jobs are running or scheduled by SLURM by calling "squeue". 3) Generate the required input file for spe_pick. Check that that all objects in the DOL file exist.
txt2idx element="scws_dols.txt" index="cygx1"
4) Run spe_pick to generate the final spectra
spe_pick group="cygx1.fits" source="Cyg X-1" rootname="cygX1_all" instrument="ISGRI"
Typical IBIS image/mosaic
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=10 > scws_all.txt
2) Start the ScW based analysis:
master_sbs --job=IMA --scwlist=scws_all.txt --energies="30,50,200" --verbose --clobber=yes --path=$PWD
3) Generate the required input file for spe_pick
txt2idx element="scws_dols.txt" index="cygx1"
4) Run INT_mosa_ibis to generate the final image
INT_mosa_ibis --obsgrp=$PWD --dol=scws_dols.txt --destin=$PWD/obs/mosa_cygx1
5) Check the images for artifacts and sort out affected ScWs through trial-and-error:
ds9 isgri_mosa_ima.fits[2] -region found.reg -cmap b -scale sqrt -scale limits 0 60 -zoom 2
Typical JEMX spectral analysis
1) Find ScWs that cover your source, e.g., pointing is < 10 degree away from source:
objscw --scwlist "Cyg X-1" --maxdist=5 > scws_jmx.txt
2) Generate a RMF with the required binning, e.g., 32 bins:
j_rebin_rmf binlist=STD_032
(Depending on your fitting software the resulting RMF might give you troubles and needs to be rewritte. ) 3) Start the ScW based analysis with 32=2^5 logarithmic energy bins:
master_sbs --path=$PWD --scwlist=scws_jemx.txt --job=SPE --source="Cyg X-1" --jemx=1 --clobber=yes --catalog=catalog_jemx.fits --response=jemx_rebinned_rmf.fits --nChanBins=-5
4) Generate the required input file for spe_pick
txt2idx element="scws_jmx_dols.txt" index="cygx1"
5) Run spe_pick to generate the final spectra.
spe_pick group=cygx1.fits source="Cyg X-1" rootname="cygx1" instrument="JMX1"
JEMX images
To generate JEMX images there are no dedicated scripts at Remeis, and you have to fall back to the OSA scripts, e.g.:
og_create idxSwg=scw.lst ogid=oph2 baseDir="./" instrument=JMX1 jemx_science_analysis startLevel="IMA2" endLevel="IMA2" jemxNum= COR_gainModel=2
Quick Look Analysis
NRT data can also be analysed at Remeis with the ususal scipts, require, however, an additional --nrt parameter in most scripts. Also have a look at http://isdc.unige.ch/heavens/
Documentation
For common questions: http://www.isdc.unige.ch/integral/download/osa/doc/10.0/osa_um_ibis/node27.html And for the cook-book and general documentation. Reading the IBIS Analysis User Manual is recommend: https://www.isdc.unige.ch/integral/analysis
Abbreviations
http://www.isdc.unige.ch/integral/support/glossary
- DAL The Data Access Layer (DAL) is a software library used to manipulate (create, write, read and manage) scientific data.
- DOL The Data Object Location (DOL) specifies the location of a DAL object, both by the FITS file name and its extension. For instance, /isdc/dev/xxx.fits[2] is a DOL, while xxx.fits is simply the name of a FITS file.
Known Issues
- In general: check first http://www.isdc.unige.ch/integral/support/faq for the error code you get
- The resulting spectra will not be strictly OGIP compliant, which e.g. ISIS might care about, but can be ignored.
Errors mosaicing using custom DOLs
If you want to create a mosaic using DOL (Data Object Location) files, sometimes the building interrupts with errors like a Fortran Runtime error (http://www.isdc.unige.ch/integral/support/faq.cgi?SOFT-051) or errors like:
Using and copying first SCW >>/scratch1/beuchert/integral/scw_extr/obs/125400400010/scw/125400400010.001/swg_ibis.fits << Log_0 : 100 elements opened. Log_0 : 100 elements attached. Error_1: Can not update the index.
or
Error_1: Can not close the element
or
Error_2: DAL_READ_ERROR : error reading from FITS file
etc.
- Somehow corrupted scws? Collect sets of two scws and create Mosaics based on those. Like that it is easy to enclose corrupted scws.
- An error -2004 (DAL_FILE_NOT_ACCESSIBLE) often results because one of your groups is corrupted. Before restarting the analysis, make a ``dal_clean, which should be able to do an automatic repair. If it does not work try a ``dal_verify to see what is corrupted. Or you can try to open your group (or child group) with fv and look for a file pointed by the group that does not exists. Most of the time removing the line with this wrong file in the group table is fixing your problem.
The latter: first create an observation group out of your dol-list of scws. Creating observation groups is also the canonical way of extracting INTEGRAL/IBIS data. Before doing that, the soft links as noted above have to be set in the directory. Then:
cd path/to/dir/with/softlinks/ og_create idxSwg=dol.txt ogid=isgri_gc baseDir="./" instrument=IBIS ls obs/isgri_gc/
Repair the observation group and included indices/links with
dal_clean inDOL="obs/isgri_gc/og_ibis.fits" chatty="4" checkExt="1"
Now INT_mosa_ibis should work again on the scws in dol.txt
How to create mosaics (outdated)
First make a list of scws you want to add to a mosaic, i.e., a "dol-file" dol.txt:
/scratch1/beuchert/integral/scw_extr/obs/066200340010/scw/066200340010.001/swg_ibis.fits /scratch1/beuchert/integral/scw_extr/obs/066200350010/scw/066200350010.001/swg_ibis.fits /scratch1/beuchert/integral/scw_extr/obs/066200360010/scw/066200360010.001/swg_ibis.fits
Then prepare the directory where the mosaic shall be built in, i.e., set the env. variables to allow INT_mosa_ibis to find indices etc.:
cd /some/path/ ln -s /virgo/b/INTEGRAL/aux/ ln -s /virgo/b/INTEGRAL/idx/ ln -s /virgo/b/INTEGRAL/ic/ ln -s /virgo/b/INTEGRAL/scw/ ln -s /virgo/b/INTEGRAL/cat/ mkdir /some/path/obs/ INT_mosa_ibis --obsgrp=/path/to/all/extracted/scws/ --destin=/path/to/the/prepared/directory/obs/mosa_og/ --dol="dol.txt"