STEP-BY-STEP
INSTRUCTION
TO SET UP A ROCKS LINUX CLUSTER with DUAL BOOT (Windows +
Rocks)
NODES
================================================================================
We have built two 20-PC cluster using Rocks 6.1.1 (Sand Boa)
in the
School of Physics,
Universiti Sains Malaysia (USM), Penang. They are
comsics.usm.my and
anicca.usm.my. This document serves to record all the steps
involved
while we
build our cluster.
First, some useful IP setting information we used when
installing the
Rocks Linux
cluster. USMNET is the network of USM. The IP mentioned below
is that
defined
within USMNET.
Primary DNS: 10.202.1.27
Secondary DNS: 10.202.1.6
Gateway: 10.205.19.254
Public Netmask 255.255.255.0 (alternatively 255.255.254.0, or
that
auto-suggested by Rocks installation DVD.)
Private IP: 10.1.1.1
Private netmask: 255.255.0.0.
Public netmask: 255.255.255.0 (alternatively 255.255.254.0, or
that
auto-suggested by Rocks installation DVD.)
Note that these private netmask are default values suggested
by the
Rocks
installation disk.
IP names and address requirement
================================
You got to have an IP address for the cluster you are setting
up, and
register
the name of the cluster in your institute network. For
example, our
comsics's
IP address is 10.205.19.208, and the formal name of the
cluster,
www.comsics.usm.my, is registered in USMNET so that the IP
10.205.19.208 is
associated with the name www.comsics.usm.my. Then ths comsics
cluster
can be
accessed either via comsics.usm.my or the IP 10.205.19.208. We
have to
request
USM PPKT (the network administrator of USM) to have the domain
name
comsics.usm.my registered under the IP 10.205.19.208. In USM,
we
request for
Server Registration online at
http://infodesk.usm.my/infodesk/login.php
You may have to request for a named DNS and IP address from
the network
administrator of your institute.
Installation procedure
=======================
0. Assures that the boot sequence of ALL PCs are set such that
CD/DVDROM is in
the first boot up sequence.
1. A PC used as a frontend must have two network cards: one
built-in
and
another plug-in with a spec of 10/100/1000 Mb/s. Connect from
internet
line a
LAN wire into the built-in LAN port. This is identified a the
eht1
port. The
eth0 port (plug-in LAN card) is connected to a switch that
must support
10/100/1000 Mb/s. Any other PCs used as compute-nodes must
also have
LAN cards
of 10/100/1000 Mb/s speed (usually plug-in ones). All these
plug-in LAN
cards
must also be connected to the switch. No LAN wire shall be
connected
from a
compute node directly to the internet network. All LAN cable
in a
compute node
must only be connected to the switch.
2. Initially, switch on only the frontend PC. Leave the
compute node
power off.
The LAN switch must be on and assure that the LAN cables are
connected
into the
frontend and the switch in the way mentioned in 2. Slot in the
Rocks
Cluster
DVD installation disk, and type 'build' when prompted by the
screen.
3. When prompted, fill in the name, some miscellaneous info of
the
cluster and
the IP details as mentioned above. Choose automatic partition
if do not
wish to
customise the partition. If customised partitioning is
desired, suggest
to use
the following allocation:
SWAP: 10GB (make it large so that it can serve as
temporary
life-saving RAM
in case of critical situation during intensive calculation)
/var: 20 GB (you may not know when /var is going
to blow
up, which happen in
cases of perpertual warning messages being generated by the
apache
server or
whatever protocal in the server due to don't-now-what error
triggered
unexpectedly).
/boot: 100 MB
/: 24 GB (or larger if wished)
/state/partition1: Maximum
The installation will take place automatically once
partitioning
begins.
4. Make sure to take out the DVD when it is ejected after
about 20 - 30
min
when Rocks is first successfully installed. Failure to
retrieve the
installation disk from the CD drive renders repeated
installation of
infinite
times.
5. Rocks will reboot when finish installing for the the first
time. The
first
screen may see black screen with some warning sign if the
frontend PC
does not
has NVDIA GPU installed. Simply press 'enter' when prompted so
that the
frontend fixes the problem automatically by installing a
generic
display driver.
A GUI shall be displayed after a step times pressing enter.
Post-installation of the Rocks Frontend (Stage 1)
=================================================
Once the Rocks frontend is up and running, do the following:
Download two *.xml files from the www2 server. These *.xml
files
contain the
instruction of how the hardisks of the nodes are to be
partitioned. The
*.xml
files first be stored in /share/apps/configrepo. The directory
/share/apps/configrepo will be used to keep all other files
for
post-installation purposes.
Now, instruct the frontend to partition the nodes so that they
could be
used to
host a dual-boot OS (Windows + Rocks). To do so, execute the
following
command
lines in the frontend terminal:
#####################################
mkdir
/share/apps/configrepo
cd
/share/apps/configrepo
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_noformat_windows
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_format_all
cd
/export/rocks/install/site-profiles/6.1.1/nodes
mv
replace-partition.xml
replace-partition.xml.orig
cp
/share/apps/configrepo/replace-partition.xml_noformat_windows
.
cp
/share/apps/configrepo/replace-partition.xml_format_all .
cp
replace-partition.xml_format_all replace-partition.xml
cd
/export/rocks/install
rocks create distro
#####################################
In the above code, the replace-partition.xml (which is copied
from
replace-partition.xml_format_all) file will provide
instruction (during
the
execution of the insert-ethers Compute procedure, see the
following
section) of
how the node should be formatted. Note that for
replace-partition.xml
node-formatting instruction to take effect, you must do the
following:
cd
/export/rocks/install
rocks create distro
If replace-partition.xml_format_all is used as
replace-partition.xml,
all
content of the node's hard disk will be formatted. After
formatting,
each node
will have a 40GB of space in the first partition (to be used
for
Windows
installation), a small /state/partition2 (to be used for KDE
partition
manager)
and other default partitions.
In the event where you are installing a fresh frontend node
where all
the
compute nodes have been partitioned according to
replace-partition.xml_format_all (say by an earlier
installation), you
should
use replace-partition.xml_noformat_windows in place of
replace-partition.xml_format_all
as replace-partition.xml (the xml file to be 'rocks create
distro').
replace-partition.xml_noformat_windows instructs that the all
partitions in the
nodes shall NOT be formatted except the partition of root
directory '/'
in the
nodes.
Installing compute nodes
========================
Once 'rocks create distro' is done, you can proceed to the
next step:
Installing the compute nodes.
1. First and foremost, issue the following commands as root in
terminal:
rocks set appliance
attr
compute x11 true
After the above command, the nodes will be equipped with GUI
(Gnome
desktop)
when they are installed via insert-ethers subsequently (see
later).
2. Connect the frontend and all the compute nodes to the LAN
switch via
their
eth0 ports. Of course, assure the power supply to the LAN
switch is ON.
In the
frontend's terminal, issue the command as root in the frontend
terminal:
insert-ethers
When prompted, choose `Compute'. Manually slot a Rocks Cluster
installation DVD
into the individual PC. They will be detected if the LAN wires
are
connected
properly into the LAN switch via eth0. Warning: insert-ethers
ONLY
after the
command 'rocks set appliance attr compute x11 true' is issued.
3. Boot up the nodes using the Rocks installation disk. These
nodes
shall be
partitioned according to the spec as specified in
replace-partition.xml. A node
usually takes about 15 - 20 mins (or less) to be configured.
YOU MUST
RETRIEVE
THE INSTALLATION DISK FROM A NODE ONCE IT FINISHES TO PREVENT
REPEATED
INSTALLATION. Once a node completes its installation, it will
reboot
into a
usual-looking CENTOS log-in screen.
Post-installation of the Rocks Frontend (Stage 2)
=================================================
We will put some necessary files into the Rocks frontend. This
is done
via wget
and scp into a private server directory from the server
www2.fizik.usm.my
(password required in a scp fetch).
The wget part:
cd
/share/apps/configrepo
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/
cd
/share/apps/configrepo
wget http://www2.fizik.usm.my/tlyoon/Downloads/Apache_OpenOffice_4.0.1_Linux_x86-64_install-rpm_en-US.tar.gz
tar -xzvf
Apache_OpenOffice_4.0.1_Linux_x86-64_install-rpm_en-US.tar.gz
The scp part:
scp -r
tlyoon@www2.fizik.usm.my/home/tlyoon/repo/configrepo/rocks_611/*
.
chmod +x *.sh
*.conf *. *.run
Add users to the Rocks cluster
sh
/share/apps/configrepo/useradd.sh
sh
/share/apps/configrepo/useraddc.sh
sh
/share/apps/configrepo/useradd_human.sh
rocks sync users
cd
/share/apps/configrepo
Install openoffice:
sh
inst_openoffice.sh
Install some necessary applications in the frontend
yum install -y
gftp.x86_64
yum install -y
compat-libstdc++-33.i686
Copy virtual Windows.XP' vdi file to VBtemplate and carbon
capy one to
VBlocal.
This is quite a lengthy process, may take about 30 mins (or
less)
#########################
mkdir
/share/apps/VBtemplate
mv
/share/apps/configrepo/windowsXP.vdi /share/apps/VBtemplate
mkdir
/state/partition1/VBlocal
cp
/share/apps/VBtemplate/windowsXP.vdi /state/partition1/VBlocal
chmod 777
/state/partition1/VBlocal/windowsXP.vdi
cd
/share/apps/configrepo
#########################
Install mathematica (manual attention required)
sh
/share/apps/configrepo/mathematica1.conf
Install mathematica license manager (manual attention
required. Read
the
instruction as contained in addmathlm)
sh
/share/apps/configrepo/addmathlm
To activate mathlm, you may need the following URLs:
https://user.wolfram.com/portal/requestAK/506f8a2585f11524c0d64de6d0589e4f427ba1af
https://user.wolfram.com/portal/passwordrequest.html
Installation of compilers and libraries in /share/apps in the
frontend
(in
sequence)
====================================================================================
List of
compilers and libraries to be installed in /share/apps in
the frontend (icc must be installed first)
============================================================================================================
icc
http://www2.fizik.usm.my/configrepo/howto/intel/inst_icc_11.0.081_sa
ifftw
http://www2.fizik.usm.my/configrepo/howto/intel/inst_ifftw_sa
ifort
http://www2.fizik.usm.my/configrepo/howto/intel/inst_ifort_111072_sa
impi 4.1.0.024 (* may not wrok without valid lisence
file*)
http://www2.fizik.usm.my/configrepo/howto/intel/inst_impi_410024_sa
lapack, blas
(Netlib, non-optimised)
http://www2.fizik.usm.my/configrepo/howto/BLAS_and_LAPACK/inst_blas
http://www2.fizik.usm.my/configrepo/howto/BLAS_and_LAPACK/inst_lapack
mpich2
http://www2.fizik.usm.my/configrepo/howto/mpich2/inst_mpich2_314_intel_sa
fftw3
http://www2.fizik.usm.my/configrepo/howto/fftw/inst_fftw312_sa
gcc-4.7.1 (note
this takes a few hours to complete)
http://www2.fizik.usm.my/configrepo/howto/gcc/inst_gcc_sa
openmpi
ATLAS (tuned and optimised lapack and blas)
install hdf5
install netcdf
install PnetCDF
List of physics software to be installed in
/share/apps in the frontend
=======================================================================
g09
http://www2.fizik.usm.my/configrepo/howto/gaussian/inst_g09
dftb+
http://www2.fizik.usm.my/configrepo/howto/dftb+/inst_dftb+_sa
pthbhga
mkdir
/share/apps/ptmbhga
cd
/share/apps/ptmbhga
scp
-r
tlyoon@www2.fizik.usm.my:/home/tlyoon/repo/ptmbhga/ptmbhga_anicca/*
.
cd VERX
Modify
Makefile (locate -llapack and set mpif90 gnu version)
make
runopt
cd
/share/apps
scp
-r tlyoon@www2.fizik.usm.my:/home/tlyoon/repo/runopt_VER2.0
lammps, gnu version
http://www2.fizik.usm.my/configrepo/howto/mylammps/gnu/inst_lammps_gnu_Rocks611
lammps,intel version
http://www2.fizik.usm.my/configrepo/howto/mylammps/intel/inst_lammps_intel_Rocks611_generic
TestHsu3
wien2k
http://www2.fizik.usm.my/configrepo/howto/wien2k/ver_140611/inst_wien2k_140611_parallel_sa
ptmbhga_lammps
ptmbhga_dftb+
ptmbhga_g09
deMon2k
VASP
Mathlab
USPECK
Post-installation of Rocks compute nodes
=========================================
Right after all nodes are up and running:
1. Install some necessary packages in the nodes
rocks run host
'nohup yum
install -y compat-libstdc++-33.x86_64
compat-libstdc++-33.i686 brasero.x86_64 gftp.x86_64 &'
rocks run host
'nohup sh
/share/apps/configrepo/inst_virtualbox
&'
rocks run host
'nohup sh
/share/apps/configrepo/inst_openoffice &'
rocks run host
'mkdir
/state/partition1/VBlocal/'
3. The following line cp *.vdi to local hard disk. This
process takes a
long
time, a few hours at least.
rocks run host
'nohup cp
/share/apps/VBtemplate/windowsXP.vdi
/state/partition1/VBlocal/
&'
4. Installation of mathematica has to be done locally in each
node. It
can't be
done via rocks run host
sh
/share/apps/configrepo/mathematica1.conf
Setting up dual boot in the compute nodes
=================================================
This stage is to be carried out only when everything described
above
are
completed. Please proceed to
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/howto_set_up_dual_boot_for_nodes.txt
Maintenance
===========
At this stage the Rocks cluster should have already up and
running.
However at
times some possible glitches may occur to the cluster
involving
frontend and
nodes alike. We shall address some possibilities in the
following, and
their
remedy.
In the event of electric blackout or unintended hardware
failure, a few
possibilities may occur.
1. The frontend and all the nodes reboot automatically as
normal,
without the
need of any manual intervention. The cluster as a whole
appears normal
and
working as usual.
2. Auto re-installation of the nodes may be triggered. The
auto-reinstallation
may just proceed automatically and seamlessly without the need
of
manual
intervention. Upon completion of the re-installation, the
nodes appear
normal
and working as usual. All previous applications and software
existed in
them
are not lost.
3. Auto re-installation of the nodes are triggered. The auto
re-installation
may just proceed automatically and seamlessly without the need
of
manual
intervention. Upon completion of the re-installation, the
nodes appear
normal
and working as usual, BUT all previous applications and
software
previously
existed in some of the nodes are lost. In this case, you have
to
reinstall all
these lost applications and software in these lost nodes.
5. The worst case scenario will be: the frontend just get
corrupted and
the
Rocks cluster as a whole could not resume. Re-installation of
the
frontend
(hence the Rocks cluster) is necessary.
Ideally, the customization of the frontend and nodes should be
as
painless as
possible. It is particularly more so in the case where the
re-installation of
nodes or frontend are forced to take place. To make the
customization
as easy
as it could possibly be, the strategy is to prepare a list of
installation
scripts (written in bash script) that, once executed, install
the
intended
program/software package seamlessly. These one-click
installation
scripts are
to be found in /share/apps/configrepo, and are usually named
in the
format of
inst_XXX, e.g., inst_openoffice, inst_virtualbox, etc.
To install a program across all nodes with a one-liner script,
issue
rocks run host "sh
/share/apps/configrepo/inst_XXX"
where XXX is the neme of the program to install. To install
the program
XXX
using the inst_XXX script in a compute node X,
ssh -X -Y compute-0-X
sh
/share/apps/configrepo/inst_XXX,
The script /share/apps/configrepo/mathematica1.conf will
install
Mathematica
(but you have to standby to interact manually during the
installation
process).
Re-installation of the frontend
==============================
This worst-case scenario could be forced upon us in the event
of hard
disk
breakdown, which is not entirely unlikely. In this case, the
re-installation
process have to be repeated by following through the
instructions and
procedures as described in this document. However, since the
nodes may
have
already been pre-installed with Windows and KDE partition
earlier, you
don't
want the re-installation to wipe them off. So, make sure that
when you
re-install the frontend, mind to assure that you are using the
right
replace-partition.xml, i.e, use the
replace-partition.xml_noformat_windows as
the replace-partition.xml when performing the 'rocks create
distro'
act.
#####################################
mkdir
/share/apps/configrepo
cd
/share/apps/configrepo
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_noformat_windows
wget
http://www2.fizik.usm.my/configrepo/howto/RocksClusters_6.1.1/replace-partition.xml_format_all
cd
/export/rocks/install/site-profiles/6.1.1/nodes
mv
replace-partition.xml
replace-partition.xml.orig
cp
/share/apps/configrepo/replace-partition.xml_noformat_windows
.
cp
/share/apps/configrepo/replace-partition.xml_format_all .
cp
replace-partition.xml_noformat_windows replace-partition.xml
cd
/export/rocks/install
rocks create distro
#####################################
Theoretically, the re-installation of frontend and the cluster
as a
whole
should be a seamless process and much easier than in the first
instance
of
installation. Inst_XXX scripts in /share/apps/configrepo make
the
installation
of software and applications in the frontend and nodes an easy
task.
You may
not need to copy the virtual Windows.XP's dvi file into the
nodes
anymore
since, as a legacy from previous installation, they are
already sitting
in the
nodes's /state/partition1. Also due to legacy, there is no
need to
reinstall
the Windows and KDE partition manager when you are doing the
re-installation.
By Yoon Tiem Leong
School of Physics
Universiti Sains Malaysia (USM)
11800 USM
Penang, Malaysia
14 May 2015