( Payara 5.2021.4 https://docs.payara.fish/community/docs/5.2021.4/documentation/payara-server/README.html, java 11.0.2 https://openjdk.org/projects/jdk/11/ )
Vertical scaling is done adding more power to machine - more CPUs, Memory, storage, network speed. Horizontal scaling is realized by adding additional nodes to the pool of resources.
The application server makes possible to increase BellaDati complex power by horizontal scaling. BellaDati application is deployed to many nodes of multiple-node environment of Payara application server. More details see web pages of Payara. |
Multiple-node (Cluster) environment for several computers - horizontal scalability can be created to achieve HA or other desired operation levels. Payara server is derived from GlassFish Server Open Source Edition. To install multiple-node (cluster) solution on set of computers is similar (but not the same) as that, used for GlassFish application server clustering. ( detail description for BellaDati GlassFish is in https://support.belladati.com/techdoc/Running+BellaDati+in+Cluster+-+Linux ) |
There is not limitation to two nodes in BellaDati installation for horizontal scalability based on Payara application server. Tests with deploying BellaDati application have been done to 4+ nodes. |
Next manual illustrates and step by step describes how to realize multiple-node ( cluster) installation with two computers up to to operational status;
Private network for nodes communication is used.
User "cluster" is added to server, group admin,
password authentication allowed in private network, ssh allowed in firewall
/etc/hosts table(s):
server 172.31.40.20
127.0.0.1 localhost
127.0.0.1 ip-172-31-40-20
172.31.40.20 ip-172-31-40-20.eu-central-1.compute.internal belladati-main
172.31.38.252 ip-172-31-38-252.eu-central-1.compute.internal node1
server 172.31.38.252
127.0.0.1 localhost
127.0.0.1 ip-172-31-38-252
172.31.40.20 ip-172-31-40-20.eu-central-1.compute.internal belladati-main
172.31.38.252 ip-172-31-38-252.eu-central-1.compute.internal node1
i.e. two servers are included to multiple-node (cluster) environment ; the static (private) FQDN IP are ip-172-31-40-20.eu-central-1.compute.internal and p-172-31-38-252.eu-central-1.compute.internal
FQDN stands for Fully Qualified Domain Name
e.g hostnamectl set-hostname ip-172-31-40-20.eu-central-1.compute.internal ( FQDM for server belladati-main in private network)
Testing ssh across servers communication and java commands availability for account cluster (login name) ssh -l cluster node1, ( belladati-main or IPs or FQDNs)
As mentioned above, BellaDati version 2.10 comes with java11 and Payara application server.
The fact is utilized in the following installation instructions:
Run java testing commands on belladati-main server (for each of the servers) ssh -l login name server identification 'java command'
ssh -l cluster node1 'java -version' ; ssh l cluster node1 'jar -help ' etc.
SSH account cluster is going to spread Payara application server environment from belladati-main server to every multiple-node server.
|
The processes described here are carried out by root on belladati-main machine.
Payara asadmin utility subcommands are preferred ( GUI of Payara console is available too).
The role of the user cluster is to spread up Payara environment from belladat-main machine to all cluster machines. To utilize the account in next commands, its password alias is created:
run command : asadmin create-password-alias clusterpassword
insert cluster account password
run vim /home/cluster/clusterpassword
add line: AS_ADMIN_SSHPASSWORD=${ALIAS=clusterpassword}
asadmin list-password-aliases shows the name, asadmin delete-password-alias XXXXX delete the password alias
Machine belladati-main directory /usr/local/BellaDati/payara5 will be copied to the multiple-node (cluster) machine node1 to directory /home/cluster/payara5 directory of the user cluster.
The user cluster has ssh connection to the machines. Its password information is in /home/password/clusterpassword file.
/home/cluster/payara5 directory exists on node1 computer Before next step check/modify nadmin file permissions on belladati-main server, directory ...payara5/glassfish/lib ; ( chmod +x nadmin) |
asadmin create-node-ssh --nodehost node1 --installdir /home/cluster/payara5 --install=true --sshuser cluster --passwordfile /home/cluster/clusterpassword node1
Node verification - testing connection to node (s)
run command asadmin ping-node-ssh node1
asadmin list-nodes-ssh ssh nodes created are listed
asadmin list-nodes this command list all nodes, including original node localhost-domain1 on localhost i.e. on belladati-main
example:
root@ip-172-31-40-20:/usr/local/BellaDati/payara5/glassfish/lib# asadmin list-nodes
localhost-domain1 CONFIG localhost
node1 SSH node1
Command list-nodes executed successfully.
Modifying Path to Payara java variable:
change variable AS_JAVA in /home/cluster/payara5/glassfish/config/asenf.conf AS_JAVA=/home/cluster/jre/
reason: PATH for JAVA environment; on ssh nodes is not equal to PATH copying from belladati-main in asenv.conf file
The base environment/configuration for multiple-node (cluster) itself is establishing now. The multiple-node (cluster) name will be "belladati-cluster".
run command asadmin create-cluster belladati-cluster
New configuration is added to Payara configuration. Its name is belladati-cluster-config. Its configuration is the same as default-config.
helpful additional commands: asadmin list-clusters ; asadmin delete-cluster belladati-cluster
Cluster instance on belladati-main; its name is "instance-local"
run command asadmin create-local-instance --cluster belladati-cluster instance-local
output example:
Rendezvoused with DAS on localhost:4848.
Port Assignments for server instance instance-local:
OSGI_SHELL_TELNET_PORT=26666
JAVA_DEBUGGER_PORT=29009
HTTP_LISTENER_PORT=28080
IIOP_SSL_LISTENER_PORT=23820
ASADMIN_LISTENER_PORT=24848
IIOP_SSL_MUTUALAUTH_PORT=23920
JMX_SYSTEM_CONNECTOR_PORT=28686
HTTP_SSL_LISTENER_PORT=28181
IIOP_LISTENER_PORT=23700
Command create-local-instance executed successfully
see /usr/local/BellaDati/payara5/glassfish additional files hierarchy appears nodes/localhost-domain1/...
Cluster instance on ssh node
run commands for node1; names for the created instances will be instance-node1
asadmin create-instance --node node1 --cluster belladati-cluster instance-node1
output example
Command _create-instance-filesystem executed successfully.
Port Assignments for server instance instance-node1:
OSGI_SHELL_TELNET_PORT=26666
JAVA_DEBUGGER_PORT=29009
HTTP_LISTENER_PORT=28080
IIOP_SSL_LISTENER_PORT=23820
ASADMIN_LISTENER_PORT=24848
IIOP_SSL_MUTUALAUTH_PORT=23920
JMX_SYSTEM_CONNECTOR_PORT=28686
HTTP_SSL_LISTENER_PORT=28181
IIOP_LISTENER_PORT=23700
The instance, instance-node1, was created on host node1
Command create-instance executed successfully.
helpful commands; for visibility: asadmin list-instances, for deleting: asadmin delete-instance NAME of the instance to delete
Configuration file for belladati-cluster is named belladati-cluster-config. Until now, there is no information in its contents regarding resources for BellaDati in the cluster, no information about application to run in cluster. Parameters defining now (overtaking from default-config) e.g. thread-pools, heap space for jvm etc. are not valid for running application BellaDati in cluster environment. Next steps describe how to modify the configuration file belladati-cluster-config |
run commands: asadmin create-application-ref --target belladati-cluster belladati
asadmin delete-application-ref --target server belladati
BellaDati was installed on Payaraapplication serverand its target was server; for cluster arrangement target change to "belladati-cluster" is needed
to see the application references asadmin list-application-refs belladati-cluster ;
to delete belladati reference from cluster asadmin delete-application-ref --target belladati-cluster belladati
to delete belladati reference from default server asadmin delete-application-ref --target server belladati
Taking into account recommendation for belladati installation max-thread-pool-size will be changed.
run command asadmin list-threadpools belladati-cluster to see list of threadpools;
modifications will be done threadpools: http-thread-pool and thread-pool-1
http-thread-pool:
run command asadmin get belladati-cluster-config.thread-pools.thread-pool.http-thread-pool.* parameters values before setting up
and run asadmin set belladati-cluster-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=512
thread-pool-1:
run command asadmin get belladati-cluster-config.thread-pools.thread-pool.thread-pool-1* parameters values before setting up
and run asadmin set belladati-cluster-config.thread-pools.thread-pool.thread-pool-1.max-thread-pool-size=512
New BellaDati 2.10+ architecture transfered some parameters from application server (see https://support.belladati.com/techdoc/Running+BellaDati+in+Cluster+-+Linux) to jvm options. The parameters must be defined in belladati-cluster-config. |
to see all JVM options as defined before modification for belladati-cluster:
asadmin list-jvm-options --target belladati-cluster
Dependently on machine HW specifications there are recommendation to modify so called "heap-space" for running jvm. The parameter -Xmxnnnnnm. nnnnn is value of RAM size for jvm.
Heap space modification
run command to delete -Xmx value not valid for cluster:
asadmin delete-jvm-options --target belladati-cluster -Xmx512m the "-Xmx512m" is default value, i.e. just 512 MB for running jvm - not enough for BellaDati operation
run command to setup new, valid value for -Xmx:
asadmin create-jvm-options --target belladati-cluster -Xmx4096m the "-Xmx2048m" i.e. 4096 MB for jvm is valid for this cluster. (The value is based on given machines HW specification and usually is higher;)
Parameters for jdbc resources
asadmin create-jvm-options --target belladati-cluster -Djdbc.driverClassName=org.postgresql.Driver
asadmin create-jvm-options --target belladati-cluster -Dfile.encoding=UTF-8
asadmin create-jvm-options --target belladati-cluster -Dlogs.location=../../../../../logs
asadmin create-jvm-options --target belladati-cluster "-Djdbc.url=jdbc\:postgresql\://IPaddress where dbs engine runs\:port/dbs_name"
asadmin create-jvm-options --target belladati-cluster -Djdbc.username=dbs_user_name
asadmin create-jvm-options --target belladati-cluster -Djdbc.password=password
asadmin create-jvm-options --target belladati-cluster -Djdbc.pool.maxActive=200
asadmin create-jvm-options --target belladati-cluster -Djdbc.pool.minIdle=8
asadmin create-jvm-options --target belladati-cluster -Djdbc.pool.maxIdle=200
asadmin create-jvm-options --target belladati-cluster -Djdbc.pool.minIdleEvictionTime=300000
asadmin create-jvm-options --target belladati-cluster -Djdbc.pool.maxWaitTime=60000
Parameters specifying cluster environment:
asadmin create-jvm-options --target belladati-cluster -Dtapestry.clustered.session=true
asadmin create-jvm-options --target belladati-cluster -Dapplication.actor.server.port=2335
asadmin create-jvm-options --target belladati-cluster "-Dapplication.servers=belladati-main\:2335,node1\:2335"
New, modified configuration for cluster is available now. Many changes occurred during the process. Payara application server needs restarting. |
Run command: asadmin restart-domain
helpful commands to check the cluster environment before starting cluster:
asadmin list-domains ; asadmin list-nodes ; asadmin list-instances ; asadmin list-application-refs belladati-cluster
asadmin list-instances belladati-cluster command lists all instances available for cluster belladati-cluster, namely: instance-local, instance-node1
The process of the cluster starting can take minutes.
run command asadmin start-cluster --verbose belladati-cluster one command to start all cluster nodes (instances)
or alternatively
run commands asadmin start-instance NAME (instance-local, instance-node1 etc)
command for the first instance start cluster environment with one instance; the next commands will add instances to the cluster.
This command is very illustrative with BellaDati application Monitoring cluster screen.
Visibility: asadmin list-clusters ; asadmin list-instances belladati-cluster
Application BellaDati is operating in the cluster environment.
To log into BellaDati application
for machine "belladati-main" public IP and port 28080, for "node1" public IP and port 28080
BellaDati license screens indicate : license is not valid
Reason: BellaDati application cluster settings require a dedicated licence key.
Instead of starting position with BellaDati running on machine belladati-main, there are two machines where BellaDati is deployed now.
Contact BellaDati support for obtaining a new license.
To modify license, copy all cluster Server IDs of the machines (belladati-main, node1) and send both to BellaDati support .
The new license must be implemented on the every BellaDati installation in cluster machine environment.
Who: root, Location: belladati-main server
user root will now operate with the belladati.war. This is new version of BellaDati application and is going to be redeployed as "belladati" (name of application) to Payara application server target "domain"
run asadmin redeploy --name belladati --target domain belladati.war
run asadmin stop-cluster belladati-cluster
run asadmin restart-domain
run asadmin start-cluster belladati-cluster