Difference between revisions of "How to use the Remeis cluster"

Latest revision as of 16:17, 14 April 2025

How To use the Remeis Cluster

The Cluster of the Remeis observatory is designed to make all resources available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to admin@sternwarte.uni-erlangen.de.

Performing Computations

You should never log in on servers directly and especially not run any computations on servers without using slurm.
Computations and extractions on the Cluster should be submitted via slurm. Information on how to use it can be found here.
Also interactive sessions can be started via slurm.

The Data-Storage System

/userdata: Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be backed up simply. And there is no quota on /userdata.

/home: This is the place for self-developed software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is backed up every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory for the worst case (e.g., fire). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (5 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the quota command on libra, like

 ssh libra quota -s -f /exports/disk1

/scratch: For temporary or otherwise not very important data. You should not store anything important on /scratch!! The scratch disks are NOT backedup and the scratch disks are not fail safe, i.e. if a scratch disk is broken (which happens regularly), your data is GONE. scratch might be emptied at any time without warning.

Satellite Data: We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to Ingo.

Logging into Remeis

Remote Login

Using ssh

You can log-in via ssh simply by

 ssh -X <username>@<host>.sternwarte.uni-erlangen.de

Using x2go

For remote login with a graphical interface you can use X2Go Remote Desktop

Hosts

Host names are generally the constellation names, e.g., norma or mensa.

Connecting without password and short hostnames

Sick of always typing user@myfavouritehost.sternwarte.uni-erlangen.de and always having to enter your password when switching machines inside inside the Remeis Cluster? The follwing setup will make your life easier.

Config-File

In the file ~/.ssh/config you can set up the configuration for ssh. A few things are really useful.

  # activate compression, might lead to a faster connection
  Compression yes
  # do X11 forwarding by default, i.e., the option "-X" is 
  # always set when using "ssh"
  forwardX11 yes

In order to define a short-cut for a host, you have to add the following information to the ~/.ssh/config

  # the short-name you want to call the  host (most useful is the general name of the machine
  Host=norma
  # now define user and host such that <User>@<Hostname> is the correct adress
  Hostname=norma.sternwarte.uni-erlangen.de
  User=YourUsername

ssh-keys

If you setup the ssh-keys properly, you can login via ssh without entering a password (although you should definitely protect your ssh key with a password). A detailed description can be found here.

Setting the Default Options/Software

Each new account is equipped with a sample file at ~/.cshrc, which is loaded on every start of the terminal. Hence, this defines your options, i.e., the software available, short-cuts, your favourite editor, and so on. Commonly useful options are activated by default. In addition, there are a few additional features, which are commented out by default but still very useful for most users.

  # set your favourite text editor to be used by default; important, e.g., if you want to use GIT
  setenv EDITOR jed 
  
  # activate the X-ray software (HEADAS, satellite extraction scripts, ...)
  source /data/software/cshrc.in

More information on the TC shell and how to configure it can be found here.

@@ Line 1: / Line 1: @@
 == How To use the Remeis Cluster ==
-The Cluster of the Remeis obsveratory is designed to make all ressources available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to [mailto:admins@sternwarte.uni-erlangen.de admins@sternwarte.uni-erlangen.de].
+The Cluster of the Remeis observatory is designed to make all resources available for easy usage. Nevertheless, there are a few important things one has to know in order to use the cluster properly. In the following the most important information will be listed. Other questions or problems with the system or software sould be directed to [mailto:admin@sternwarte.uni-erlangen.de admin@sternwarte.uni-erlangen.de].
 === Performing Computations  ===
-* The servers ''crux'', ''grus'', and ''phoenix'' should not be used for computations at all.
+* You should never log in on servers directly and especially not run any computations on servers without using slurm.
-* Computations and extractions on the Cluster should be submitted via '''torque'''. Information on how to use it can be found [[torque:start|here]].
+* Computations and extractions on the Cluster should be submitted via '''slurm'''. Information on how to use it can be found [[Slurm|here]].
-* Also [[torque:start#interactive_jobs|interactive sessions]] can be started via '''torque'''.
+* Also interactive sessions can be started via '''slurm'''.
 ===  The Data-Storage System ===
-* '''/userdata:''' Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be backuped simply. And there is no quota on /userdata.
+* '''/userdata:''' Extracted data like images or spectra or other space consuming stuff or stuff which can re recreated/downloaded again should be put there. /userdata is a RAID6 system, i.e. two disks can fail simultaneously w/o data loss --> your data is safe on /userdata! However, there is NO backup of userdata, as /userdata is too big to be backed up simply. And there is no quota on /userdata.
-* '''/home:''' This is the place for self-developped software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is backedup every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory  for the worst case (fire etc). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (1 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the ''quota'' command on crux, like
-  ssh crux quota -s -f /home
-* '''/scratch:''' For temporary or otherwise not very important data. You should not store anything important on /scratch!! The scratch disks are NOT backedup and the scratch disks are not fail safe, i.e. if a scratch disk is broken (which happens regularly), your data is GONE.
-* '''Satellite Data:''' We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to the person in charge of a specific mission. You can find out who is responsible by executing <code>showSatelliteContacts</code> in the shell.
+* '''/home:''' This is the place for self-developed software, scripts, documents, TeX files, diploma/phd/... theses, papers, spreadsheets, etc. /home is also a RAID system, i.e. more than one disk may fail w/o data loss. Furthermore, /home is backed up every night to TWO different disks (in two machines) AND once a week on an external disk that is stored outside the observatory  for the worst case (e.g., fire). It therefore does not make sense to create your own private backups on local scratch disks etc. /home, however, is quite small (5 TB for all the institute); we therefore have a quota of about 10GB per user. It is no problem to get more space, if you can't move stuff around anymore, just drop an Email to the admins. On the other hand, you don't have to fill your quota completely! To check your quota you can use the ''quota'' command on libra, like
+  ssh libra quota -s -f /exports/disk1
+* '''/scratch:''' For temporary or otherwise not very important data. You should not store anything important on /scratch!! The scratch disks are NOT backedup and the scratch disks are not fail safe, i.e. if a scratch disk is broken (which happens regularly), your data is GONE. scratch might be emptied at any time without warning.
+* '''Satellite Data:''' We already have archives for most satellite raw data (applies mostly to the X-ray people). So please do not download any raw data yourself, but talk to Ingo.
 ===  Logging into Remeis ===
@@ Line 32: / Line 29: @@
-===== Using the NX-Client =====
+===== Using x2go =====
-If you need a graphics interface, it might be worth to use the <code>nxclient</code> to connect to the Remeis Cluster. The corresponding <code>nxserver</code> is installed on (almost) all machines. In order to use it you have to [http://www.nomachine.com/download.php download] it and follow the Connection-Wizard to connect on a host. Always use the same machine, as only a limited number of total connections is allowed on each machine. This works for Linux, Mac OSX, and even Windows.
+For remote login with a graphical interface you can use [[X2Go Remote Desktop]]
 ===== Hosts =====
-Host names are generally the constellation names, e.g., <code>norma</code> or <code>leo</code>. All available hosts can be found in the shell variable <code>$RemeisHosts</code>. The servers (<code>crux</code>, <code>grus</code>, and <code>phoenix</code>) should not be used.
+Host names are generally the constellation names, e.g., <code>norma</code> or <code>mensa</code>.
 ==== Connecting without password and short hostnames ====
@@ Line 54: / Line 51: @@
 In order to define a short-cut for a host, you have to add the following information to the ''~/.ssh/config''
     # the short-name you want to call the  host (most useful is the general name of the machine
-    Host=pulsar
+    Host=norma
     # now define user and host such that ''<User>@<Hostname>'' is the correct adress
-    Hostname=pulsar.sternwarte.uni-erlangen.de
+    Hostname=norma.sternwarte.uni-erlangen.de
     User=YourUsername
 ===== ssh-keys =====
-If you setup the ssh-keys properly, you can login via ssh without entering a password. This isn't only nice to have, but if you use [[torque:start|Torque]], this is mandatory. A detailed description can be found [[SSH-Keys|here]].
+If you setup the ssh-keys properly, you can login via ssh without entering a password (although you should definitely protect your ssh key with a password). A detailed description can be found [[SSH-Keys|here]].
 === Setting the Default Options/Software ===
@@ Line 71: / Line 68: @@
     # activate the X-ray software (HEADAS, satellite extraction scripts, ...)
-    source $SOFTDIR/softwarescript_Xray.csh
+    source /data/software/cshrc.in
-More information on the TC shell and how to configure it can be found [[csh:start|here]].
+More information on the TC shell and how to configure it can be found [[TcShell|here]].
-[[Category:Current Members]]
+[[Category:Current Members]][[Category:Software]]