Corso Cloud Storage Es1
Esercitazione 1
Nel seguito i nodi su cui eseguire i comandi sono indicati con i nomi
gpfs-srv-0, gpfs-obj1-0, gpfs-obj2-0, stack-group-00.
Ciascun gruppo dovra' sostituire l'ultimo numero con il numero del gruppo di appartenenza (gpfs-srv-#, stack-group-0#, etc.).
Preparazione dei nodi
Preparazione dei nodi che faranno parte del cluster:
gpfs-srv-#: GPFS server
gpfs-obj1-#, gpfs-obj2-#: Protocol nodes (CES nodes)
stack-group-0#: Openstack controller (AllInOne)
Verificare che la macchina stack-group-0# (Ubuntu) consenta l'accesso a root via ssh
adminuser@:stack-group-00:~# sudo grep ^PermitRootLogin /etc/ssh/sshd_config PermitRootLogin yes adminuser@:stack-group-00:~# grep ^root /etc/shadow root:$6$pVAFmNrA$yiUfOPnZNXOT0c1aYYtw9Nh3s2r/HZW.2dEiG/AldUt9sQ97QFzHVI34O9amSHV95m5zoP8MkaA/JE5BOEFI6.:17450:0:99999:7:::
Se cosi' non fosse, modificare la configurazione di sshd come riportato e assegnare una password a root. Configurare i nodi in modo da poter fare ssh senza password e senza richiesta di accettazione di chiavi (known_hosts).
- Collegarsi su gpfs-srv-#, generare le chiavi ssh, e consentire l'accesso via chiave su tutte le macchine (compreso il server gpfs-srv-#)
- Riempire il file known_hosts con tutte le chiavi dei quattro nodi (simple name e FQDN):
[root@gpfs-srv-0 .ssh]# ssk-keygen ... ... [root@gpfs-srv-0 .ssh]# for node in stack-group-00 gpfs-srv-0 gpfs-obj1-0 gpfs-obj2-0 > do > ssh-copy-id $node > done ... ... [root@gpfs-srv-0 .ssh]# cd /root/.ssh [root@gpfs-srv-0 .ssh]# for node in stack-group-00 gpfs-srv-0 gpfs-obj1-0 gpfs-obj2-0 > do > ssh -o StrictHostKeyChecking=no ${node}.recas.ba.infn.it hostname > ssh -o StrictHostKeyChecking=no ${node} hostname > done
- Copiare su tutti i nodi i file di chiave privata e il known_hosts
[root@gpfs-srv-0 .ssh]# for node in stack-group-00 gpfs-obj1-0 gpfs-obj2-0 > do > scp known_hosts id_rsa id_rsa.pub ${node}:~/.ssh/ > done
- Controllare la configurazione di SELINUX sui nodi CentOS
Editare su gpfs-srv-# il file /etc/sysconfig/selinux e modificare la riga
SELINUX=enforcing
in
SELINUX=disabled
Copiare il file /etc/sysconfig/selinux sui nodi gpfs-obj*
[root@gpfs-srv-0 httpd]# for i in gpfs-obj1-0 gpfs-obj2-0 > do > scp /etc/sysconfig/selinux ${node}:/etc/sysconfig/ > done
Disabilitare su tutti i tre nodi CentOS selinux
[root@gpfs-srv-0 httpd]# setenforce 0 [root@gpfs-srv-0 httpd]# for node in gpfs-obj1-0 gpfs-obj2-0 > do > ssh $node setenforce 0 > done
Installazione di GPFS sui nodi gpfs-srv-# e stack-group-0#
- Sacaricare il software ed i file di setup necessari via wget da group-www
[root@gpfs-srv-0 ~]# wget -O /etc/profile.d/gpfs.sh http://group-www/gpfs/gpfs.sh [root@gpfs-srv-0 ~]# wget http://group-www/gpfs/Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install [root@gpfs-srv-0 ~]# wget http://group-www/gpfs/Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install [root@gpfs-srv-0 ~]# . /etc/profile.d/gpfs.sh
root@stack-group-00:~# wget -O /etc/profile.d/gpfs.sh http://group-www/gpfs/gpfs.sh root@stack-group-00:~# wget http://group-www/gpfs/Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install root@stack-group-00:~# wget http://group-www/gpfs/Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install root@stack-group-00:~# . /etc/profile.d/gpfs.sh
Installazione sul nodo CentOS gpfs-srv-#
- Spacchettate il software Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install e Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install (usare le flag --silent --text-only)
- Installate i pacchetti gpfs.base, gpfs.docs, gpfs.ext, gpfs.gpl, gpfs.gskit, gpfs.msg.en_US che si trovano in /usr/lpp/mmfs/4.2.0.4
- Eseguite il build del portability layer (con il comando mmbuildgpl) dopo aver definito la variabile di environment LINUX_DISTRIBUTION=REDHAT_AS_LINUX
[root@gpfs-srv-0 ~]# chmod +x Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install [root@gpfs-srv-0 ~]# chmod +x Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install [root@gpfs-srv-0 ~]# ./Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install --text-only --silent ... ... [root@gpfs-srv-0 ~]# ./Spectrum_Scale_Protocols_Standard-4.2.3.4-x86_64-Linux-install --text-only --silent ... ... [root@gpfs-srv-0 ~]# cd /usr/lpp/mmfs/4.2.3.4/gpfs_rpms [root@gpfs-srv-0 gpfs_rpms]# yum localinstall gpfs.base-4.2.3-4.x86_64.rpm gpfs.docs-4.2.3-4.noarch.rpm gpfs.ext-4.2.3-4.x86_64.rpm gpfs.gpl-4.2.3-4.noarch.rpm \ > gpfs.gskit-8.0.50-75.x86_64.rpm gpfs.msg.en_US-4.2.3-4.noarch.rpm gpfs.license.std-4.2.3-4.x86_64.rpm ... ... [root@gpfs-srv-0 gpfs_rpms]# export LINUX_DISTRIBUTION=REDHAT_AS_LINUX [root@gpfs-srv-0 gpfs_rpms]# mmbuildgpl ... ... -------------------------------------------------------- mmbuildgpl: Building GPL module completed successfully at Wed Oct 11 16:22:13 CEST 2017. --------------------------------------------------------
Installazione sul nodo Ubuntu stack-group-0#
- Spacchettate il software Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install (usare le flag --silent --text-only)
- Installate i pacchetti gpfs.base, gpfs.docs, gpfs.ext, gpfs.gpl, gpfs.gskit, gpfs.msg.en_US che si trovano in /usr/lpp/mmfs/4.2.0.4
- Eseguite il build del portability layer (con il comando mmbuildgpl)
root@stack-group-00:~# chmod +x Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install root@stack-group-00:~# ./Spectrum_Scale_Standard-4.2.3.4-x86_64-Linux-install --text-only --silent root@stack-group-00:/usr/lpp/mmfs/4.2.3-4/gpfs_debs# dpkg -i gpfs.base_4.2.3-4_amd64.deb gpfs.docs_4.2.3-4_all.deb gpfs.ext_4.2.3-4_amd64.deb \ > gpfs.gpl_4.2.3-4_all.deb gpfs.gskit_8.0.50-75_amd64.deb gpfs.msg.en-us_4.2.3-4_all.deb gpfs.license.std_4.2.3-4_amd64.deb Selecting previously unselected package gpfs.base. (Reading database ... 142501 files and directories currently installed.) Preparing to unpack gpfs.base_4.2.3-4_amd64.deb ... ... dpkg: dependency problems prevent configuration of gpfs.base: gpfs.base depends on ksh; however: Package ksh is not installed. gpfs.base depends on m4; however: Package m4 is not installed. ... root@stack-group-00:/usr/lpp/mmfs/4.2.3-4/gpfs_debs# apt-get install -f Reading package lists... Done Building dependency tree ...
root@stack-group-00:/usr/lpp/mmfs/4.2.3.4/gpfs_debs# apt-get install make gcc g++ ... ... root@stack-group-00:/usr/lpp/mmfs/4.2.3-4/gpfs_debs# mmbuildgpl ... ... -------------------------------------------------------- mmbuildgpl: Building GPL module completed successfully at Wed Oct 11 16:40:08 CEST 2017. --------------------------------------------------------
Creazione del cluster e configurazione dei ruoli
Si deve creare un cluster GPFS con le seguenti caratteristiche:
- nome uguale al nome del nodo gpfs-srv-#
- abilitare il CCR
- usare ssh/scp (e' il default)
- usare il profilo gpfsProtocolDefaults
Inserire i nodi gpfs-srv-0# come quorum node e manager, e stack-group-0# come semplice client senza ruolo di quorum, e configurare conseguentemente la licenza.
Comandi: mmcrcluster, mmchlicense, mmlscluster, mmlslicense, mmaddnonde, mmlsnode.
Creazione del cluster
Create il cluster con il solo gpfs-srv-# come membro
[root@gpfs-srv-0 ~]# mmcrcluster -N gpfs-srv-0:quorum-manager: --ccr-enable -C gpfs-srv-0 --profile gpfsProtocolDefaults mmcrcluster: Performing preliminary node verification ... mmcrcluster: Processing quorum and other critical nodes ... mmcrcluster: Finalizing the cluster data structures ... /bin/grep: /var/mmfs/gen/mmsdrfs: No such file or directory mmcrcluster: Command successfully completed mmcrcluster: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. [root@gpfs-srv-0 ~]# mmlscluster =============================================================================== | Warning: | | This cluster contains nodes that do not have a proper GPFS license | | designation. This violates the terms of the GPFS licensing agreement. | | Use the mmchlicense command and assign the appropriate GPFS licenses | | to each of the nodes in the cluster. For more information about GPFS | | license designation, see the Concepts, Planning, and Installation Guide. | =============================================================================== GPFS cluster information ======================== GPFS cluster name: gpfs-srv-0.recas.ba.infn.it GPFS cluster id: 17561377055771261101 GPFS UID domain: gpfs-srv-0.recas.ba.infn.it Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp Repository type: CCR Node Daemon node name IP address Admin node name Designation ------------------------------------------------------------------------------------------ 1 gpfs-srv-0.recas.ba.infn.it 172.20.0.95 gpfs-srv-0.recas.ba.infn.it quorum-manager [root@gpfs-srv-0 ~]# mmchlicense server --accept -N gpfs-srv-0 The following nodes will be designated as possessing server licenses: gpfs-srv-0.recas.ba.infn.it mmchlicense: Command successfully completed [root@gpfs-srv-0 ~]# mmlscluster GPFS cluster information ======================== GPFS cluster name: gpfs-srv-0.recas.ba.infn.it GPFS cluster id: 17561377055771261101 GPFS UID domain: gpfs-srv-0.recas.ba.infn.it Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp Repository type: CCR Node Daemon node name IP address Admin node name Designation ------------------------------------------------------------------------------------------ 1 gpfs-srv-0.recas.ba.infn.it 172.20.0.95 gpfs-srv-0.recas.ba.infn.it quorum-manager
Aggiungete il nodo stack-group-0# come membro del cluster.
[root@gpfs-srv-0 ~]# mmaddnode -N stack-group-00:nonquorum-client: Sun Oct 2 12:10:37 UTC 2016: mmaddnode: Processing node stack-group-00.recas.ba.infn.it mmaddnode: Command successfully completed mmaddnode: Warning: Not all nodes have proper GPFS license designations. Use the mmchlicense command to designate licenses as needed. mmaddnode: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. [root@gpfs-srv-0 ~]# mmchlicense client --accept -N stack-group-00 The following nodes will be designated as possessing client licenses: stack-group-00.recas.ba.infn.it mmchlicense: Command successfully completed mmchlicense: Propagating the cluster configuration data to all affected nodes. This is an asynchronous process. [root@gpfs-srv-0 ~]# mmlscluster GPFS cluster information ======================== GPFS cluster name: gpfs-srv-0.recas.ba.infn.it GPFS cluster id: 17561377055771261101 GPFS UID domain: gpfs-srv-0.recas.ba.infn.it Remote shell command: /usr/bin/ssh Remote file copy command: /usr/bin/scp Repository type: CCR Node Daemon node name IP address Admin node name Designation -------------------------------------------------------------------------------------------------- 1 gpfs-srv-0.recas.ba.infn.it 172.20.0.95 gpfs-srv-0.recas.ba.infn.it quorum-manager 2 stack-group-00.recas.ba.infn.it 172.20.0.55 stack-group-00.recas.ba.infn.it
Startup del cluster
Far partire GPFS su tutti i nodi.
Nota: il primo startup richiede qualche minuto prima che i nodi siano attivi.
[root@gpfs-srv-0 ~]# mmstartup -a Sun Oct 2 12:35:55 UTC 2016: mmstartup: Starting GPFS ... mmget[root@gpfs-srv-0 ~]# mmgetstate -a Node number Node name GPFS state ------------------------------------------ 1 gpfs-srv-0 arbitrating 2 stack-group-00 arbitrating [root@gpfs-srv-0 ~]# mmgetstate -a Node number Node name GPFS state ------------------------------------------ 1 gpfs-srv-0 active 2 stack-group-00 active