Corso Cloud Storage Es1

From Centro Calcolo
Revision as of 15:00, 11 October 2017 by Mirko (talk | contribs) (→‎Installazione sul nodo CentOS gpfs-srv-#)
(diff) ← Older revision | Latest revision (diff) | Newer revision → (diff)
Jump to navigation Jump to search

Esercitazione 1

Nel seguito i nodi su cui eseguire i comandi sono indicati con i nomi

gpfs-srv-0, gpfs-obj1-0, gpfs-obj2-0, stack-group-00.

Ciascun gruppo dovra' sostituire l'ultimo numero con il numero del gruppo di appartenenza (gpfs-srv-#, stack-group-0#, etc.).


Preparazione dei nodi

Preparazione dei nodi che faranno parte del cluster:

gpfs-srv-#: GPFS server

gpfs-obj1-#, gpfs-obj2-#: Protocol nodes (CES nodes)

stack-group-0#: Openstack controller (AllInOne)

Verificare che la macchina stack-group-0# (Ubuntu) consenta l'accesso a root via ssh

adminuser@:stack-group-00:~# sudo grep ^PermitRootLogin /etc/ssh/sshd_config
PermitRootLogin yes

adminuser@:stack-group-00:~# grep ^root /etc/shadow
root:$6$pVAFmNrA$yiUfOPnZNXOT0c1aYYtw9Nh3s2r/HZW.2dEiG/AldUt9sQ97QFzHVI34O9amSHV95m5zoP8MkaA/JE5BOEFI6.:17450:0:99999:7:::

Se cosi' non fosse, modificare la configurazione di sshd come riportato e assegnare una password a root. Configurare i nodi in modo da poter fare ssh senza password e senza richiesta di accettazione di chiavi (known_hosts).

  • Collegarsi su gpfs-srv-#, generare le chiavi ssh, e consentire l'accesso via chiave su tutte le macchine (compreso il server gpfs-srv-#)
  • Riempire il file known_hosts con tutte le chiavi dei quattro nodi (simple name e FQDN):
[root@gpfs-srv-0 .ssh]# ssk-keygen
...
...
[root@gpfs-srv-0 .ssh]# for node in stack-group-00 gpfs-srv-0 gpfs-obj1-0 gpfs-obj2-0
> do
> ssh-copy-id $node
> done
...
...
[root@gpfs-srv-0 .ssh]# cd /root/.ssh
[root@gpfs-srv-0 .ssh]# for node in stack-group-00 gpfs-srv-0 gpfs-obj1-0 gpfs-obj2-0
> do
> ssh -o StrictHostKeyChecking=no ${node}.recas.ba.infn.it hostname
> ssh -o StrictHostKeyChecking=no ${node} hostname
> done
  • Copiare su tutti i nodi i file di chiave privata e il known_hosts
[root@gpfs-srv-0 .ssh]# for node in stack-group-00 gpfs-obj1-0 gpfs-obj2-0
> do
> scp known_hosts id_rsa id_rsa.pub ${node}:~/.ssh/
> done
  • Controllare la configurazione di SELINUX sui nodi CentOS

Editare su gpfs-srv-# il file /etc/sysconfig/selinux e modificare la riga

SELINUX=enforcing

in

SELINUX=disabled

Copiare il file /etc/sysconfig/selinux sui nodi gpfs-obj*

[root@gpfs-srv-0 httpd]# for i in gpfs-obj1-0 gpfs-obj2-0
> do
> scp /etc/sysconfig/selinux ${node}:/etc/sysconfig/
> done

Disabilitare su tutti i tre nodi CentOS selinux

[root@gpfs-srv-0 httpd]# setenforce 0
[root@gpfs-srv-0 httpd]# for node in gpfs-obj1-0 gpfs-obj2-0
> do
> ssh $node setenforce 0
> done

Installazione di GPFS sui nodi gpfs-srv-# e stack-group-0#

  • Sacaricare il software ed i file di setup necessari via wget da group-www
[root@gpfs-srv-0 ~]# wget -O /etc/profile.d/gpfs.sh http://group-www/gpfs/gpfs.sh
[root@gpfs-srv-0 ~]# wget http://group-www/gpfs/Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install
[root@gpfs-srv-0 ~]# wget http://group-www/gpfs/Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install
[root@gpfs-srv-0 ~]# . /etc/profile.d/gpfs.sh
root@stack-group-00:~# wget -O /etc/profile.d/gpfs.sh http://group-www/gpfs/gpfs.sh
root@stack-group-00:~# wget http://group-www/gpfs/Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install
root@stack-group-00:~# wget http://group-www/gpfs/Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install
root@stack-group-00:~# . /etc/profile.d/gpfs.sh 


Installazione sul nodo CentOS gpfs-srv-#

  • Spacchettate il software Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install e Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install (usare le flag --silent --text-only)
  • Installate i pacchetti gpfs.base, gpfs.docs, gpfs.ext, gpfs.gpl, gpfs.gskit, gpfs.msg.en_US che si trovano in /usr/lpp/mmfs/4.2.0.4
  • Eseguite il build del portability layer (con il comando mmbuildgpl) dopo aver definito la variabile di environment LINUX_DISTRIBUTION=REDHAT_AS_LINUX
[root@gpfs-srv-0 ~]# chmod +x Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install
[root@gpfs-srv-0 ~]# chmod +x Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install
[root@gpfs-srv-0 ~]# ./Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install --text-only --silent
...
...
[root@gpfs-srv-0 ~]# ./Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install --text-only --silent
...
...
[root@gpfs-srv-0 ~]# cd /usr/lpp/mmfs/4.2.3.4/gpfs_rpms
[root@gpfs-srv-0 gpfs_rpms]# yum localinstall gpfs.base-4.2.3-4.x86_64.rpm gpfs.docs-4.2.3-4.noarch.rpm gpfs.ext-4.2.3-4.x86_64.rpm  gpfs.gpl-4.2.3-4.noarch.rpm \
> gpfs.gskit-8.0.50-75.x86_64.rpm  gpfs.msg.en_US-4.2.3-4.noarch.rpm gpfs.license.std-4.2.3-4.x86_64.rpm
...
...
[root@gpfs-srv-0 gpfs_rpms]# export LINUX_DISTRIBUTION=REDHAT_AS_LINUX
[root@gpfs-srv-0 gpfs_rpms]# mmbuildgpl
...
...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at Wed Oct 11 16:22:13 CEST 2017.
--------------------------------------------------------

Installazione sul nodo Ubuntu stack-group-0#

  • Spacchettate il software Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install (usare le flag --silent --text-only)
  • Installate i pacchetti gpfs.base, gpfs.docs, gpfs.ext, gpfs.gpl, gpfs.gskit, gpfs.msg.en_US che si trovano in /usr/lpp/mmfs/4.2.0.4
  • Eseguite il build del portability layer (con il comando mmbuildgpl)
root@stack-group-00:~# chmod +x Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install 

root@stack-group-00:~# ./Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install --text-only --silent

root@stack-group-00:/usr/lpp/mmfs/4.2.3-4/gpfs_debs# dpkg -i gpfs.base_4.2.3-4_amd64.deb gpfs.docs_4.2.3-4_all.deb gpfs.ext_4.2.3-4_amd64.deb \
> gpfs.gpl_4.2.3-4_all.deb gpfs.gskit_8.0.50-75_amd64.deb gpfs.msg.en-us_4.2.3-4_all.deb gpfs.license.std_4.2.3-4_amd64.deb

Selecting previously unselected package gpfs.base.
(Reading database ... 142501 files and directories currently installed.)
Preparing to unpack gpfs.base_4.2.3-4_amd64.deb ...
...
dpkg: dependency problems prevent configuration of gpfs.base:
 gpfs.base depends on ksh; however:
  Package ksh is not installed.
 gpfs.base depends on m4; however:
  Package m4 is not installed.
...

root@stack-group-00:/usr/lpp/mmfs/4.2.3-4/gpfs_debs# apt-get install -f
Reading package lists... Done
Building dependency tree       
...
root@stack-group-00:/usr/lpp/mmfs/4.2.3.4/gpfs_debs# apt-get install make gcc g++
...
...
root@stack-group-00:/usr/lpp/mmfs/4.2.3-4/gpfs_debs# mmbuildgpl
...
...
--------------------------------------------------------
mmbuildgpl: Building GPL module completed successfully at Wed Oct 11 16:40:08 CEST 2017.
--------------------------------------------------------

Creazione del cluster e configurazione dei ruoli

Si deve creare un cluster GPFS con le seguenti caratteristiche:

  • nome uguale al nome del nodo gpfs-srv-#
  • abilitare il CCR
  • usare ssh/scp (e' il default)
  • usare il profilo gpfsProtocolDefaults

Inserire i nodi gpfs-srv-0# come quorum node e manager, e stack-group-0# come semplice client senza ruolo di quorum, e configurare conseguentemente la licenza.

Comandi: mmcrcluster, mmchlicense, mmlscluster, mmlslicense, mmaddnonde, mmlsnode.

Creazione del cluster

Create il cluster con il solo gpfs-srv-# come membro

[root@gpfs-srv-0 ~]# mmcrcluster -N gpfs-srv-0:quorum-manager: --ccr-enable -C gpfs-srv-0 --profile gpfsProtocolDefaults
mmcrcluster: Performing preliminary node verification ...
mmcrcluster: Processing quorum and other critical nodes ...
mmcrcluster: Finalizing the cluster data structures ...
/bin/grep: /var/mmfs/gen/mmsdrfs: No such file or directory
mmcrcluster: Command successfully completed
mmcrcluster: Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.
[root@gpfs-srv-0 ~]# mmlscluster

===============================================================================
| Warning:                                                                    |
|   This cluster contains nodes that do not have a proper GPFS license        |
|   designation.  This violates the terms of the GPFS licensing agreement.    |
|   Use the mmchlicense command and assign the appropriate GPFS licenses      |
|   to each of the nodes in the cluster.  For more information about GPFS     |
|   license designation, see the Concepts, Planning, and Installation Guide.  |
===============================================================================


GPFS cluster information
========================
  GPFS cluster name:         gpfs-srv-0.recas.ba.infn.it
  GPFS cluster id:           17561377055771261101
  GPFS UID domain:           gpfs-srv-0.recas.ba.infn.it
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name             IP address   Admin node name              Designation
------------------------------------------------------------------------------------------
   1   gpfs-srv-0.recas.ba.infn.it  172.20.0.95  gpfs-srv-0.recas.ba.infn.it  quorum-manager

[root@gpfs-srv-0 ~]# mmchlicense server --accept -N gpfs-srv-0

The following nodes will be designated as possessing server licenses:
	gpfs-srv-0.recas.ba.infn.it
mmchlicense: Command successfully completed
[root@gpfs-srv-0 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs-srv-0.recas.ba.infn.it
  GPFS cluster id:           17561377055771261101
  GPFS UID domain:           gpfs-srv-0.recas.ba.infn.it
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name             IP address   Admin node name              Designation
------------------------------------------------------------------------------------------
   1   gpfs-srv-0.recas.ba.infn.it  172.20.0.95  gpfs-srv-0.recas.ba.infn.it  quorum-manager

Aggiungete il nodo stack-group-0# come membro del cluster.

[root@gpfs-srv-0 ~]# mmaddnode -N stack-group-00:nonquorum-client:
Sun Oct  2 12:10:37 UTC 2016: mmaddnode: Processing node stack-group-00.recas.ba.infn.it
mmaddnode: Command successfully completed
mmaddnode: Warning: Not all nodes have proper GPFS license designations.
    Use the mmchlicense command to designate licenses as needed.
mmaddnode: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@gpfs-srv-0 ~]# mmchlicense client --accept -N stack-group-00

The following nodes will be designated as possessing client licenses:
	stack-group-00.recas.ba.infn.it
mmchlicense: Command successfully completed
mmchlicense: Propagating the cluster configuration data to all
  affected nodes.  This is an asynchronous process.
[root@gpfs-srv-0 ~]# mmlscluster

GPFS cluster information
========================
  GPFS cluster name:         gpfs-srv-0.recas.ba.infn.it
  GPFS cluster id:           17561377055771261101
  GPFS UID domain:           gpfs-srv-0.recas.ba.infn.it
  Remote shell command:      /usr/bin/ssh
  Remote file copy command:  /usr/bin/scp
  Repository type:           CCR

 Node  Daemon node name                 IP address   Admin node name                  Designation
--------------------------------------------------------------------------------------------------
   1   gpfs-srv-0.recas.ba.infn.it      172.20.0.95  gpfs-srv-0.recas.ba.infn.it      quorum-manager
   2   stack-group-00.recas.ba.infn.it  172.20.0.55  stack-group-00.recas.ba.infn.it

Startup del cluster

Far partire GPFS su tutti i nodi.

Nota: il primo startup richiede qualche minuto prima che i nodi siano attivi.

[root@gpfs-srv-0 ~]# mmstartup -a
Sun Oct  2 12:35:55 UTC 2016: mmstartup: Starting GPFS ...
mmget[root@gpfs-srv-0 ~]# mmgetstate -a

 Node number  Node name        GPFS state 
------------------------------------------
       1      gpfs-srv-0       arbitrating
       2      stack-group-00   arbitrating

[root@gpfs-srv-0 ~]# mmgetstate -a

 Node number  Node name        GPFS state 
------------------------------------------
       1      gpfs-srv-0       active
       2      stack-group-00   active