Running BellaDati in Cluster - Linux

BellaDati is not depending on the selected cluster environment, you can run it on GlassFish, Websphere or JBoss, but there are several parameters contained in application.properties (see BellaDati WAR Configuration) which need to be changed in order to run BellaDati in the cluster):

application.actor.server.port=2335
tapestry.clustered-sessions=true
application.servers=172.31.38.23:2335,172.31.13.180:2335,172.31.45.195:2335,172.31.2.41:2335

The example above defines parameters for 4 machines in a cluster. The server's IPs are   172.31.38.23, 172.31.13.180, 172.31.45.195 and 172.31.2.41

 Cluster overview page

BellaDati administrators have access to cluster overview page , which provides useful information regarding each of clustered servers.

Glassfish cluster on Linux

Installation process example & description 

The whole  installation process is divided into several blocks

• Environment Setting up

• Cluster Installation

• Setting up Cluster Configuration

• Starting Cluster 

• Modifying BellaDati License
  

• Updating BellaDati 
  

Environment Setting up

Root runs the bellow mentioned commands

• Adding ssh user  "cluster" on each of the servers;  home directory is /home/cluster, user is in group wheel
• Test ssh user "cluster" login to each of the servers ;

User password authentication is allowed.

 This example installation utilize internal, Firewall protected network; connections from outbound are strictly restricted for limited set of IP adresses; password authentication is sufficient from security point of view.

Initial table describing working environment 

• Setting hostname (each server )

Run command :  hostnamectl set-hostname     Fully Qualified Domain Name           

    e.g    hostnamectl set-hostname ip-172-31-2-41.eu-central-1.compute.internal      ( FQDM for server belladati-main in private network)

  Command  hostnamectl status yields to :

• Setting hosts table for every server

Run command:   vim  /etc/hosts          Modify the file

   Modified Hosts table   ( /etc/hosts file content on all servers)

• Modifying PostgreSQL setting on dbs server

Run editor command     vim

• Testing connectivity to database

Database engine is located/running on server dbs

Run next command on every server:       psql -h 172.31.38.23 -p 5432 -U belladati_dbuser  -d  belladati_db   

                                                            run additional psql commands (\l+, \connect, \conninfo etc... \q)

• Testing ssh across servers communication and java commands availability for account     cluster  (login name)

Run next command on  belladati-main server testing every server:                       ssh -l login name  server identification 

e.g. : ssh -l cluster dbs,  ssh -l cluster node1, ssh -l cluster node2  etc (or IPs or FQDNs)

Run java testing commands on belladati-main server (for each of the servers)       ssh -l login name server identification 'java command'

e.g. :  ssh -l cluster node1 'java -version' ;  ssh -l cluster dbs 'jar' etc.

 SSH acaount cluster is going to spread GlassFish AS environment from belladati-main server to every cluster server.This is reason for the communication.

• Setting Application Properties parameters on server belladati-main

BellaDati application is running on belladati-main server, installation directory is /usr/local/BellaDati. It yields to next location for the properties file.

Run command:  vim /usr/local/BellaDati/glassfish5/glassfish/domains/domain1/applications/belladati/WEB-INF/classes/conf/application.properties

The file application.properties must  be modified to cover new cluster communication requirements.

Add  lines  :

application.actor.server.port=2335

tapestry.clustered-sessions=true

application.servers=belladati-main:2335,node1:2335,node2:2335,dbs:2335

  instead of application servers alias names,  IP adress or FQDN (see /etc/hosts) could be used.

Cluster Installation

The processes described here are carried out by root on belladati-main machine.

   GlassFish asadmin utility subcommands are prefered but GlassFish Admin Console GUI is used for illustration as well.

• Setting account(user) cluster password

The role of the user cluster is to spread up glassfish environment from belladat-main to all machines devored for clustering. To use the account in next commands, its password alias is created:

run command : asadmin create-password-alias clusterpassword    

insert cluster account password

run vim /home/cluster/clusterpassword     

add line:    AS_ADMIN_SSHPASSWORD=${ALIAS=clusterpassword}

  asadmin list-password-aliases   shows the name, asadmin delete-password-alias  XXXXX  delete the password alias

• Node creation

             Machine belladati-main directory /usr/local/BellaDati/glassfish5 will be copied to the cluster machine (node1,node2 and dbs), to directory home directory of the user cluster.

             The user cluster has ssh connection to the machines. Its password information is in /home/password/clusterpassword file.

             The name of GlassFish nodes are defined (node1,node2 and node3 for host "dbs").

        run command step by step for hostname node1,node2 and dbs  changing operand adequately (node1,node2 and node3):  

       asadmin create-node-ssh  --nodehost node1 --installdir /home/cluster/glassfish --install=true --sshuser cluster --passwordfile  /home/cluster/clusterpassword  node1 

Set of commands to nodes creation:

•  Node verification - testing connection to node (s)

      run command  asadmin ping-node-ssh node1(node2,node3)

                            asadmin list-nodes-ssh    ssh nodes created are listed

                            asadmin list-nodes            this command list all nodes, including original node localhost-domain1 on localhost i.e. on belladati-main

     Have a look at node1, node2 or dbs user cluster home directory. Now there are additional files there - directory glassfish and its files/directories. GlassFish environment is now on all cluster servers.

GlassFiish GUI illustration:

•  cluster creation

         The base environment/configuration for cluster  itself is created now.Name of the cluster is going to be "belladati-cluster" . The step is necessary  next GlassFish cluster jvm instances.

     run command  asadmin create-cluster belladati-cluster

       New configuration is added to GlassFish configuration. Its name is belladati-cluster-config. It is necessary to modify it . See next block of the the installation process.

       helpfull additional commands:   asadmin list-clusters     ;     asadmin delete-cluster belladati-cluster 

• Creating belladati-cluster instances

        now jvm instances of GlassFish AS will be added/created for the cluster

       first cluster instance on belladati-main; its name is instance-local

       run command asadmin create-local-instance  --cluster belladati-cluster instance-local

  see /usr/local/BellaDati/glassfish5/glassfish    additional files hierarchy appear  nodes/localhost-domain1/...

repeatedly run next command for all ssh nodes:

vim /home/cluster/glassfish/config/asenv.conf   and   insert # before line defining AS_JAVA

reason for doing it is:  on the ssh node servers is different PATH for JAVA environment; not the path copying from belladati-main in asenv.conf file

run command (per each of the ssh i.e. node1,node2 and node3); names for the created instances will be instance-node1, instance-node2 and instance-node3

asadmin create-instance --node node1   --cluster belladati-cluster instance-node1

helpfull commands:  for visibility asadmin list-instances, for deleting asadmin delete-instance  NAME of the instance to delete

Set of commands creating jvm instances:

GlassFish GUI illustration:               

Setting up Cluster  Configuration

    Configuration file for belladati-cluster is named belladati-cluster-config. Untill now, there is no information regarding resources for the cluster, no information about application to run in cluster. Parameters

   defining e.g. thread-pools, heap space for jvm etc. are not valid for running application BellaDati in cluster environment. Next steps will modify the configuration file belladati-cluster-config to change the status.

• Setting up resources (JDBC)

        For belladati-cluster ("target") reference to jdbc/belladati_db i.e. definition of database for application will be created.

         run command      asadmin create-resource-ref   --target  belladati-cluster    jdbc/belladati_db

           helpfull commands - to see resources  :  asadmin list-resource-refs  belladati-cluster  ;   to delete resource jdbc/belladati_db:    asadmin delete-resource-ref --target belladati-cluster jdbc/belladati_db  

GlassFish GUI - resource targeting

• Setting up application reference

       run command:     asadmin create-application-ref   --target belladati-cluster  belladati

         to see the application references    asadmin list-application-refs  belladati-cluster   ; to delete belladati reference   asadmin delete-application-ref  --target belladati-cluster belladati

GlassFish GUI - application belladati targeting

   Cluster "belladati-cluster" is able to use database belladati_db and application belladati could be deployed on the cluster environment;

   For BellaDati application some parameters are recommended ( see https://support.belladati.com/techdoc/Installing+BellaDati+on+Linux).  

      The parameters need to be set up for cluster too; The fact yields to the following modifications:

• Setting up thread pools

        Taking into account recommendation for belladati installation max-thread-pool-size will be changed.

        run command     asadmin list-threadpools belladati-cluster  to see list of threadpools;

        modifications will be done threadpools: http-thread-pool   and   thread-pool-1

       http-thread-pool:

       run command  asadmin get belladati-cluster-config.thread-pools.thread-pool.http-thread-pool*         parameters values  before setting up

            and run      asadmin set belladati-cluster-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=512

      thread-pool-1:

       run command  asadmin get belladati-cluster-config.thread-pools.thread-pool.thread-pool-1*         parameters values  before setting up

            and run      asadmin set belladati-cluster-config.thread-pools.thread-pool.thread-pool-1.max-thread-pool-size=512

GlassFish GUI - thread pools

• JVM options setting

        Dependently on machine HW specifications there are recommendation to modify so called "heap-space" for running jvm. The parameter regarding is -Xmxnnnnnm. nnnnn is value of RAM size for jvm.

       run next command to see all JVM options as defined before modification for belladati-cluster:

                           asadmin list-jvm-options    --target  belladati-cluster

       run command to delete -Xmx value not valid for cluster:

                           asadmin delete-jvm-options  --target belladati-cluster -Xmx512m                 the "-Xmx512m" is default value, i.e. just 512 MB for running jvm

      run command to setup new, valid value for -Xmx:

                           asadmin create-jvm-options  --target belladati-cluster  -Xmx2048m             the "-Xmx2048m" i.e. 2048 MB for jvm is valid for this cluster. The value is based on given machines HW specification;

GlassFish GUI  screen representing JVM options :

Run command:                                                      asadmin restart-domain

Starting Cluster

  helpfull commands to check the cluster environment before starting cluster:

     asadmin list-domains ;   asadmin list-nodes ;   asadmin list-instances ;  asadmin list-resource-refs belladati-cluster ; asadmin list-application-refs belladati-cluster

       asadmin list-instances  belladati-cluster command lists all instances available for cluster belladati-cluster, namely: instance-local, instance-node1, instance-node2 and instance-node3. 

Set of checking commands, the last one is for starting the cluster belladati-cluster.

The process of starting the cluster can take minutes.  Two base ways can be used to start:

 run command          asadmin start-cluster  --verbose belladati-cluster      one command to start all

or

 run command          asadmin start-instance   NAME   (instance-local, instance-node1,instance-node2 and instance-node3)  

command for the first instance start cluster environment with one instance; the next commands will add instances to the cluster.

This command is very illustrative with BellaDati application Monitoring cluster screen. The screen "cluster overview" shows cluster members activity.

Visibility:           asadmin list-clusters       ;         asadmin list-instances beladati-cluster

Nodes:

 Cluster -general information:

Cluster -application,instances,resources and properties:

Have a look at .../glassfish/nodes/"node"/instance-XXXX directory:   applications  config  docroot  generated  lib  logs  osgi-cache;  now, there is belladati directory in applications  and application.properties file mentioned is in

   /home/cluster/glassfish/glassfish/nodes/node1/instance-node1/applications/belladati/WEB-INF/classes/conf;       application.properties file  was taken during the process from machine belladati-main from /usr/local/BellaDati/glassfish5/glassfish/domains/domain1/applications/belladati/WEB-INF/classes/conf/.  This is nice to know. The knowledge is useful for "Updating BellaDati".

Note properties of cluster belladati-cluster (last screen).  INSTEAD of http port 8080 is port 28080. The port is for http://Public IP:28080 to login to application.

Modifying BellaDati License

• log in

Application BellaDati is operating in cluster environment. To log into use  for machine   "belladati-main"  port 8080, for  "node1" port 28080, for "node2" port 28080 and for machine "dbs" port 28080.

   BellaDati license screen for node1,node2 and dbs machine indicate :  license is not valid 

BellaDati application license is hw machine dependent; hw has been changed - instead of starting position with BellaDati running on machine belladati-main, there are four machines where BellaDati is deployed now. 

• new license

To modify license, copying of  Server IDs of the machines (belladati-main, node1,node2 and dbs) and sending the set to BellaDati support is required.

BellaDati support will generate new, valid license. The new license must be implemented on the every cluster machine.

GUI illustration:

:

And for illustration again the very first picture of the document - BellaDati screen cluster: