Page History

BellaDati is not depending on the selected cluster environment, you can run it on GlassFish, Websphere or JBoss, but there are several parameters contained in application.properties (see BellaDati WAR Configuration) which need to be changed in order to run BellaDati in the cluster):

application.actor.server.port=2335
tapestry.clustered-sessions=true
application.servers=172.31.38.23:2335,172.31.13.180:2335,172.31.45.195:2335,172.31.2.41:2335

The example above defines parameters for 4 machines in a cluster. The server's IPs are   172.31.38.23, 172.31.13.180, 172.31.45.195 and 172.31.2.41

 Cluster overview page

BellaDati administrators have access to cluster overview page , which provides useful information regarding each of clustered servers.

Glassfish cluster on Linux

Installation process example & description 

The installation procedure is divided into several installation blocks:

Environment Setting up

Root runs the commands

• Adding ssh user  "cluster" on each of the servers;  home directory is /home/cluster, user is in group wheel
• Test ssh user "cluster" login to each of the servers ;

User password authentication is allowed.

 This example uses Firewall protected internal network. The connections from outbound are strictly restricted for limited set of IP addresses. Mentioned password authentication is sufficient from security point of view.

Initial table describing working environment 

• Setting hostname (each server )

Run command :  hostnamectl set-hostname     Fully Qualified Domain Name           

    e.g    hostnamectl set-hostname ip-172-31-2-41.eu-central-1.compute.internal      ( FQDM for server belladati-main in private network)

  Command  hostnamectl status yields to :

• Setting hosts table for every server

Run command:   vim  /etc/hosts          Modify the file

   Modified Hosts table   ( /etc/hosts file content on all servers)

• Modifying PostgreSQL setting on dbs server

Run editor command     vim

• Testing connectivity to database

Database engine is located/running on server dbs

Run next command on every server:       psql -h 172.31.38.23 -p 5432 -U belladati_dbuser  -d  belladati_db   

                                                            run additional psql commands (\l+, \connect, \conninfo etc... \q)

• Testing ssh across servers communication and java commands availability for account     cluster  (login name)

Run next command on  belladati-main server testing every server:                       ssh -l login name  server identification 

e.g. : ssh -l cluster dbs,  ssh -l cluster node1, ssh -l cluster node2  etc (or IPs or FQDNs)

Run java testing commands on belladati-main server (for each of the servers)       ssh -l login name server identification 'java command'

e.g. :  ssh -l cluster node1 'java -version' ;  ssh -l cluster dbs 'jar' etc.

 SSH acaount cluster is going to spread GlassFish AS environment from belladati-main server to every cluster server.This is reason for the communication.

• Setting Application Properties parameters on server belladati-main

BellaDati application is running on belladati-main server, installation directory is /usr/local/BellaDati. It yields to next location for the properties file.

Run command:  vim /usr/local/BellaDati/glassfish5/glassfish/domains/domain1/applications/belladati/WEB-INF/classes/conf/application.properties

The file application.properties file must  be modified to cover new cluster communication requirements.

Add  lines  :

application.actor.server.port=2335

tapestry.clustered-sessions=true

application.servers=belladati-main:2335,node1:2335,node2:2335,dbs:2335

  instead of application servers alias names,  IP adress or FQDN (see /etc/hosts) could be used.

Cluster Installation

The processes described here are carried out by root on belladati-main machine.

   GlassFish asadmin utility subcommands are prefered but GlassFish Admin Console GUI is used for illustration as well.

• Setting account(user) cluster password

The role of the user cluster is to spread up glassfish environment from belladat-main machine  to all cluster machines. To utilize the account in next commands, its password alias is created:

run command : asadmin create-password-alias clusterpassword    

insert cluster account password

run vim /home/cluster/clusterpassword     

add line:    AS_ADMIN_SSHPASSWORD=${ALIAS=clusterpassword}

  asadmin list-password-aliases   shows the name, asadmin delete-password-alias  XXXXX  delete the password alias

• Node creation

             Machine belladati-main directory /usr/local/BellaDati/glassfish5 will be copied to the cluster machine (node1,node2 and dbs), to directory home directory of the user cluster.

             The user cluster has ssh connection to the machines. Its password information is in /home/password/clusterpassword file.

             The name of GlassFish nodes are defined (node1,node2 and node3 for host "dbs").

        run command step by step for hostname node1,node2 and dbs  changing operand adequately (node1,node2 and node3):  

       asadmin create-node-ssh  --nodehost node1 --installdir /home/cluster/glassfish --install=true --sshuser cluster --passwordfile  /home/cluster/clusterpassword  node1 

Set of commands to nodes creation:

•  Node verification - testing connection to node (s)

      run command  asadmin ping-node-ssh node1(node2,node3)

                            asadmin list-nodes-ssh    ssh nodes created are listed

                            asadmin list-nodes            this command list all nodes, including original node localhost-domain1 on localhost i.e. on belladati-main

     Have a look at node1, node2 or dbs user cluster home directory. Now there are additional files there - directory glassfish and its files/directories. GlassFish environment is now on all cluster servers.

GlassFiish AS GUI illustration:

•  cluster creation

         The base environment/configuration for cluster  itself is created now. The cluster name is  "belladati-cluster". 

     run command  asadmin create-cluster belladati-cluster

       New configuration is added to GlassFish configuration. Its name is belladati-cluster-config. It is necessary to modify it . See next block of the installation process.

       helpfull additional commands:   asadmin list-clusters     ;     asadmin delete-cluster belladati-cluster 

• Creating belladati-cluster instances

       Cluster instance on belladati-main; its name is  "instance-local"

       run command asadmin create-local-instance  --cluster belladati-cluster instance-local

  see /usr/local/BellaDati/glassfish5/glassfish    additional files hierarchy appear  nodes/localhost-domain1/...

Cluster instance on ssh nodes

edit asenv.conf file on node1,node2 and dbs servers i.e. node1,node2,node3 ssh nodes:

vim /home/cluster/glassfish/config/asenv.conf      modify line where AS_JAVA is defined inserting # before ( will be #AS_JAVA)

reason: PATH for JAVA environment; on ssh nodes is not equal to PATH copying from belladati-main in asenv.conf file

run commands for node1(2 and 3); names for the created instances will be instance-node1, instance-node2 and instance-node3

asadmin create-instance --node node1   --cluster belladati-cluster instance-node1

helpfull commands:  for visibility asadmin list-instances, for deleting asadmin delete-instance  NAME of the instance to delete

Set of commands creating jvm instances:

GlassFish GUI illustration:               

Setting up Cluster  Configuration

    Configuration file for belladati-cluster is named belladati-cluster-config. Untill now, there is no information regarding resources for the cluster, no information about application to run in cluster. Parameters

   defining e.g. thread-pools, heap space for jvm etc. are not valid for running application BellaDati in cluster environment. Next steps will modify the configuration file belladati-cluster-config to change the status.

• Setting up resources (JDBC)

        For belladati-cluster ("target") reference to jdbc/belladati_db i.e. definition of database for application will be created.

         run command      asadmin create-resource-ref   --target  belladati-cluster    jdbc/belladati_db

           helpfull commands - to see resources  :  asadmin list-resource-refs  belladati-cluster  ;   to delete resource jdbc/belladati_db:    asadmin delete-resource-ref --target belladati-cluster jdbc/belladati_db  

GlassFish GUI - resource targeting

• Setting up application reference

       run command:     asadmin create-application-ref   --target belladati-cluster  belladati

         to see the application references    asadmin list-application-refs  belladati-cluster   ;

           to delete belladati reference from cluster                  asadmin delete-application-ref  --target belladati-cluster belladati    

           to delete belladati reference from default server        asadmin delete-application-ref  --target server  belladati

GlassFish GUI - application belladati targeting

   Cluster "belladati-cluster" is able to use database belladati_db and application belladati could be deployed on in the cluster environment;

   For BellaDati application some parameters are recommended ( see https://support.belladati.com/techdoc/Installing+BellaDati+on+Linux).  

      The parameters need to be set up for cluster too; The fact yields to the following modifications:

• Setting up thread pools

        Taking into account recommendation for belladati installation max-thread-pool-size will be changed.

        run command     asadmin list-threadpools belladati-cluster  to see list of threadpools;

        modifications will be done threadpools: http-thread-pool   and   thread-pool-1

       http-thread-pool:

       run command  asadmin get belladati-cluster-config.thread-pools.thread-pool.http-thread-pool*         parameters values  before setting up

            and run      asadmin set belladati-cluster-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=512

      thread-pool-1:

       run command  asadmin get belladati-cluster-config.thread-pools.thread-pool.thread-pool-1*         parameters values  before setting up

            and run      asadmin set belladati-cluster-config.thread-pools.thread-pool.thread-pool-1.max-thread-pool-size=512

GlassFish GUI - thread pools

• JVM options setting

        Dependently on machine HW specifications there are recommendation to modify so called "heap-space" for running jvm. The parameter regarding is -Xmxnnnnnm. nnnnn is value of RAM size for jvm.

       run next command to see all JVM options as defined before modification for belladati-cluster:

                           asadmin list-jvm-options    --target  belladati-cluster

       run command to delete -Xmx value not valid for cluster:

                           asadmin delete-jvm-options  --target belladati-cluster -Xmx512m                 the "-Xmx512m" is default value, i.e. just 512 MB for running jvm

      run command to setup new, valid value for -Xmx:

                           asadmin create-jvm-options  --target belladati-cluster  -Xmx2048m             the "-Xmx2048m" i.e. 2048 MB for jvm is valid for this cluster. The value is based on given machines HW specification;

GlassFish GUI  screen representing JVM options :

Run command:                                                      asadmin restart-domain

Starting Cluster

  helpfull commands to check the cluster environment before starting cluster:

     asadmin list-domains ;   asadmin list-nodes ;   asadmin list-instances ;  asadmin list-resource-refs belladati-cluster ; asadmin list-application-refs belladati-cluster

       asadmin list-instances  belladati-cluster command lists all instances available for cluster belladati-cluster, namely: instance-local, instance-node1, instance-node2 and instance-node3. 

Set of checking commands, the last one is the cluster starting command:

The process of the cluster starting can take minutes. 

 run command          asadmin start-cluster  --verbose belladati-cluster      one command to start all

or alternatively

 run commands          asadmin start-instance   NAME   (instance-local, instance-node1,instance-node2 and instance-node3)  

command for the first instance start cluster environment with one instance; the next commands will add instances to the cluster.

This command is very illustrative with BellaDati application Monitoring cluster screen.

Visibility:           asadmin list-clusters       ;         asadmin list-instances beladati-cluster

Nodes:

 Cluster -general information:

Cluster -application,instances,resources and properties:

Have a look at .../glassfish/nodes/"node"/instance-XXXX directory:   applications  config  docroot  generated  lib  logs  osgi-cache;  now, there is belladati directory in applications  and application.properties file mentioned is in

   /home/cluster/glassfish/glassfish/nodes/node1/instance-node1/applications/belladati/WEB-INF/classes/conf;       application.properties file  was taken during the process from machine belladati-main from /usr/local/BellaDati/glassfish5/glassfish/domains/domain1/applications/belladati/WEB-INF/classes/conf/.  This is nice to know. The knowledge is useful for "Updating BellaDati".

Note properties of cluster belladati-cluster (last screen):  instead of http port 8080 there is port 28080. The port is new login port i.e.  http://Public IP:28080/

Modifying BellaDati License

• log in to BellaDati application

Application BellaDati is operating in the cluster environment.

To log into BellaDati application

for machine   "belladati-main"  use port 28080, for  "node1" port 28080, for "node2" port 28080 and for machine "dbs" port 28080 as well.

   BellaDati license screen for node1,node2 and dbs machine indicate :  license is not valid 

Reason:  BellaDati application cluster settings require a dedicated licence key.

   Instead of starting position with BellaDati running on machine belladati-main, there are four machines where BellaDati is deployed now. 

• new license

Contact BellaDati support for obtaining a new license.

To modify license, copying of  Server IDs of the machines (belladati-main, node1,node2 and dbs) and sending the set to BellaDati support is required.

The new license must be implemented on the every BellaDati installation in cluster machine environment.

GlassFish AS GUI screen illustration:

:

BellaDati cluster screen :

Accessing the Cluster screen via the BellaDati Administration: Administration > Monitoring > Cluster

The application running in the cluster is BellaDati 2.9.12.2.

Updating BellaDati

root, belladati-main server

•  backup application.properties file  on belladati-main server   (/usr/local/BellaDati/glassfish5/glassfish/domains/domain1/applications/belladati/WEB-INF/classes/conf/application.properties)
•  download BellaDati-2_9_12_3.war.zip
•  unzip the war file and change to directory where belladati.war
•  for example  

  [root@ip-172-31-2-41 BellaDati-2_9_12_3.war]# ls

  belladati.war

 user root will now operate with the belladati.war. This is new version of BellaDati application and is going to be redeployed as "belladati" (name of application) to GlassFish AS target "domain"

run   asadmin redeploy --name belladati --target domain belladati.war

replace   /usr/local/BellaDati/glassfish5/glassfish/domains/domain1/applications/belladati/WEB-INF/classes/conf/application.properties with backed-up application.properties

run   asadmin stop-cluster belladati-cluster

run   asadmin restart-domain

run asadmin start-cluster belladati-cluster   

or alternatively run commands

asadmin start-instance instance-local          and     login to BellaDati application --->  see  "Servers in cluster" screen

asadmin start-instance instance-node1(2,3)  login to application on node1, node2 and dbs  and see "servers in cluster" screen

the second way of starting cluster offers illustrative view how hosts will appear in the list of active servers

The version BellaDati 2.9.12.3 is running in belladati-cluster environment on machines belladati-main, node1,node2 and dbs now.