Installing and Configuring Keycloak - Domain Clustered Deployment

Overview

Keycloak is an open source IAM developed as part of the RedHat community and ecosystem. It allows for securing applications and providing authorization and authentication services straight from within it, through federated services or external providers such as Active Directory, OpenLDAP, Social Login services (Facebook, Google, etc.) and programmatic Client Adapters via OpenID Connect, OAuth2 and SAML2.0.

The flexibility provided by Keycloak can be very advantageous when properly utilized within a existing production environment allowing securing applications and better controlling authentication and authorization with minimum overhead and code rewrite.

Source: keycloak.org

Source: keycloak.org

Source: keycloak.org

The subject of application security is quite wide and diverse and is specific to need, so in this article we will focus solely on the installation and configuration of Keycloak as a Domain Cluster deployment, which offers centralized configuration management as well as high availability fault tolerance through WildFly admin panel for deployment and load balancing services, as well as Keycloak’s admin panel for managing Realms, authentication and authorizations services, users and groups.

Pre-Requisites

Keycloak requires a Java runtime environment to be installed. On Ubuntu you can install the default OpenJDK for your distribution as long as its Java 8 or 11 for the latest release.

The simplest way to install the default JRE on Ubuntu 18.06 is by using the apt package manager and then checking java is installed by having it print out its current active version:

$ apt install default-jre
...
$ java --version

In a clustered deployment, Keycloak also requires a working installation RDBMS installation such as MariaDB or MySQL. Make sure one is installed and properly working before continuing.

Installing Keycloak

1. Download keycloak-12.0.1.zip from Keycloak downloads.
2. Place the file in a /opt.
3. Unpack the ZIP file using the appropriate unzip utility, such as unzip, tar, or Expand-Archive.
4. Linux/Unix
   $ unzip keycloak-12.0.1.zip

Quick Start

Keycloak can be quickly started in standalone mode with little to no modification to its default configuration.

Only an admin user is needed to login to Keycloak and is usually created via the Keycloak welcome screen if logging in via localhost or using the add-user-keycloak CLI script located under /bin directory under Keycloak home, as follows:

$ bin/add-user-keycloak.sh -r master -u <username> -p <password>

Once the user created you can launch the standalone deployment of Keycloak for a quick test through the following script:

$ cd bin /
$ ./standalone.sh -b [IP ADDRESS]

From your browser go to the selected IP address you bound the server to on port 8080 i.e. http://[IP ADDRES]:8080/auth

Use the created username and password for the master realm using the add-user-keycloak.sh script to login. Once certain that Keycloak starts in standalone mode, its time to start modifying the needed configuration files to allow for a Clustered Domain deployment.

The following section outlines the changes required to specific configuration files JDBC Datasource/Drivers required to connect to an RDBMS, the necessary changes to make to allow access to the WildFly application management server.

Keycloak Directory Structure & Key Configuration Files

Below are the most pertinent directories for the purpose of this article:

Directories

bin/
This contains various scripts to either boot the server or perform some other management action on the server.

domain/
This contains configuration files and working directory when running Keycloak in domain mode.

modules/
These are all the Java libraries used by the server.

standalone/
This contains configuration files and working directory when running Keycloak in standalone mode.

Configurations

standalone/configuration/standalone.xml
The default configuration file for launching Keycloak in Standalone mode.

domain/configuration/domain.xml
The default configuration file for launching Keycloak in Domain Clustered mode. The file is broken down into two main profiles, standalone (HA) and clustered. For the purposes of this article we will be focusing on the clustered profile section for our changes (though making changes to both profiles would be totally ok as well).

domain/configuration/host.xml
Contains the WildFly default configuration for Domain Clustered mode, including built-in load balancer settings and Wildfly administration binding addresses

domain/configuration/host-master.xml
The domain primary controller and authentication node configuration options

domain/configuration/host-slave.xml
The domain slave authentication node(s) configuration options.
(P.S. By default only one node is configured)

Topology of a Domain Clustered Deployment

As mentioned earlier when deploying Keycloak in a Domain Clustered mode, the underlying cluster is controlled centrally via a domain host controller through WildFly, the application runtime management platform.

The diagram below covers the general topology of Domain Clustered deployment generalized to any runtime application using WildFly and not specific just to Keycloak.

Source: wildfly.org

Host 1 is the main Domain controller but can also host a Server component as well.

The slave hosts 2 through 4 all have local Host Controllers that synchronize with the Domain controller.

Two additional elements not shown in the diagram above are:

1- The built-in load balancer which also usually runs on the main domain controller and can distribute traffic across the cluster. Keycloak recommends not using the WildFly’s built-in load balancer but rather rely on a third party hardware or software solution.

2- When deploying Keycloak in Domain Clustered mode a RDBMS systems is mandatory in order to provide the master and slave with a centralized managed data repository for settings and controls.

To manage all these various services (RDBMS excluded), a number of interfaces are predefined in WildFly in order to create logical parts for each of the components.

Three main interfaces are included in Keycloak:

Management Interface

Primarily for accessing the WildFly application runtime administration environment. By default this is bound to the loopback interface (127.0.0.1).

Private Interface

Used for the host controller, load balancing and multicast components of WildFly. Often the management and private interfaces share the same physical host interface. As with the management interface, it is bound by default to the loopback interface (127.0.0.1).

Public Interface

Is used to access the Keycloak application layer and is usually bound to the host’s public facing physical interface.

Note:
Before modifying the configuration files make sure that the server is not running. If it is, then changes to some of the parameters in these files may be overwritten or removed by the domain controller.

Making the Wildfly Console Interface Publicly Accessible

As mentioned earlier, when launching Keycloak in Clustered mode, the Wildfly application management server is the first to be bootstrapped.

When testing Clustered Keycloak on a local machine, Wildfly console interface is directly accessible through the loopback interface on port 9990. Note that even on the loopback interface a management user needs to be created in order to access the management admin interface; (you can skip the configuration file changes and go straight to creating a management user via the CLI tool).

If running in production, this might not be possible and therefore some configuration changes are in order to make the management interface accessible through one of the host’s interfaces. Usually in production, the management IP would be bound to a dedicated physical ethernet interface which in turn is on a private network only accessible to select administrators.

For the purpose of this example we do not make that distinction but only bind the management console to the host’s unique IP interface.

In order to do so the domain.xml, host.xml and host-master.xml management interface block needs to be updated as follows.

domain.xml

<!-- domain.xml -->
...
<interfaces>
    <interface name="management"/>
    <interface name="private">
        <inet-address value="${jboss.bind.address.private:[[Public IPv4 Address]]}"/>
    </interface>
    <interface name="public"/>
    </interfaces>
...

host.xml

<!-- host.xml -->
...
<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:[[Public IPv4 Address]]}"/>
    </interface>
...
</interfaces>
...

host-master.xml

<!-- host-master.xml -->
...
<interfaces>
    <interface name="management">
        <inet-address value="${jboss.bind.address.management:[[Public IPv4 Address]]}"/>
    </interface>
...
</interfaces>
...

One additional step is needed in order to login and that’s the creation of a management account for authentication via the CLI script bin/add-user.sh.

Note that when prompted to enter a group select Administrator if you’re going to be using RBAC in Wildfly otherwise the default none works as well; this can always be changed later via the Wildfly admin interface.

On another note, make sure to to answer yes when asked wether the user would be needed to connect from one AS server to another and take note of the secret hash generated which is required when configuring the slave node.

$ ./add-user.sh
What type of user do you wish to add?
a) Management User (mgmt-users.properties)
b) Application User (application-users.properties)
(a): a
Enter the details of the new user to add.
Using realm 'ManagementRealm' as discovered from the existing property files.
Username : wadmin
Password recommendations are listed below. To modify these restrictions edit the add-user.properties configuration file.
- The password should be different from the username
- The password should not be one of the following restricted values {root, admin, administrator}
- The password should contain at least 8 characters, 1 alphabetic character(s), 1 digit(s), 1 non-alphanumeric symbol(s)
Password :
Re-enter Password :
What groups do you want this user to belong to? (Please enter a comma separated list, or leave blank for none)[  ]: Administrator
About to add user 'wadmin' for realm 'ManagementRealm'
Is this correct yes/no? yes
Added user 'wadmin' to file '/opt/keycloak/standalone/configuration/mgmt-users.properties'
Added user 'wadmin' to file '/opt/keycloak/domain/configuration/mgmt-users.properties'\
Added user 'wadmin' with groups Administrator to file '/opt/keycloak/standalone/configuration/mgmt-groups.properties'
Added user 'wadmin' with groups Administrator to file '/opt/keycloak/domain/configuration/mgmt-groups.properties'
Is this new user going to be used for one AS process to connect to another AS process?
e.g. for a slave host controller connecting to the master or for a Remoting connection for server to server EJB calls.
yes/no? yes
To represent the user add the following to the server-identities definition <secret value="bWdtdDEyMyE=" />

Once this user added, use the provided credentials when prompted upon accessing the console admin interface.

With the above changes made and management user added, the Wildfly management admin interface will be available via the browser at: http://[[Public IPv4 Address]]:9990/

Wildfly Administration Console Homepage

The various deployed services controller, application, load balancer) can also be monitored and controlled via Topology under the Runtime tab.

Deployed Services Status Overview and Control

Configuring Keycloak to Use a Database Connection

As mentioned earlier, when deploying Keycloak in Domain Clustered mode, an RDBMS is required in order to centralize settings and controls for the master node and its slaves.

By default, Keycloak uses the H2 JDBC Driver which it comes preconfigured for, loading and using a memory loaded database directly when running out of the box.

It is beyond the scope of this article to delve into the details of such an RDBMS deployment which can take multiple forms depending on scalability, availability and performance needs.

Usually in production environments a database cluster deployment would be considered to cover three parameters listed earlier along with the necessary tweaks to maximize performance.

For the purposes of this example we will be using MariaDB and it’s JDBC Connector Driver as a backend. A database schema named keycloak should have already been created along with a user with full grants to that database.

Note:
When creating the database using the latest MariaDB (or MySQL). the InnoDB engine should be enabled and the database should be created using UTF8 character set rather than the default UTF8mb4 which is not compatible with Keycloak.

For starters download the corresponding JDBC driver for the database being used. For MariaDB the JDBC the latest driver can be download from the following link. Make sure the driver matches the Java version installed on the system being deployed to.

Recent versions of Keycloak will automagically load JDBC drivers properly installed under the modules/ folder. To do so the driver’s .jar file can be placed in a folder structure that follows that driver’s package name. In this case we opted for modules/com/mariadb/main/ or any other logical structure that works making the path unique to that module to avoid namespace clashes.

In addition to the .jar file a module.xml file should also be created within the same directory in order to provide WildFly with the necessary information to automagically create the Datasource when launching Keycloak. In our example the module.xml looks as follows:

<?xml version="1.0" ?>
<module xmlns="urn:jboss:module:1.3" name="com.mariadb">
    <resources>
        <resource-root path="mariadb-java-client-2.7.1.jar"/
    </resources>
    <dependencies>
        <module name="javax.api"/>
        <module name="javax.transaction.api"/>
    </dependencies>
</module>

Once the files copied/created to the above location, a number of changes are needed to domain.xml configuration in order to use the new JDBC driver as the data source. Update both the standalone and clustered sections.

Locate the drivers tag in both sections and append a new driver child tag:

...
<datasource>
    ...
</datasource>
<drivers>
    <driver name="mariadb" module="com.mariadb">
        <driver-class>org.mariadb.jdbc.Driver</driver-class
    </driver>
...
</drivers>
...

You will find a driver entry already exist for the H2 JDBC driver; do not remove it, just append the above before it.

This change will need to be done twice in the standalone and clustered sections of the domain.xml file.

Next we move on to update the <datasource> section to provide the connection string, credentials and connection options using the newly added MariaDB driver.

By default Keycloak will come with two preconfigured datasources: ExampleDS and KeycloakDS that use the H2 JDBC driver.

For simplicity, we will reuse the KeycloakDS JNDI datasource and link it to MariaDB rather than the H2.

...
<datasource jndi-name="java:jboss/datasources/KeycloakDS" pool-name="KeycloakDS" enabled="true" use-java-context="true" statistics-enabled="${wildfly.datasources.statistics-enabled:${wildfly.statistics-enabled:false}}">
    <connection-url>jdbc:mariadb://[DB IP Address]:3306/keycloak</connection-url>
    <driver>mariadb</driver>
    <security>
        <user-name>[DB Username]</user-name>
        <password>[DB Password]</password>
    </security>
</datasource>
...

To make sure all is in order, two more sections in domain.xml need to be checked to make sure that the keycloak datasource defined above is the default datasource.

The <default-binding/>

...
<default-bindings context-service="java:jboss/ee/concurrency/context/default" datasource="java:jboss/datasources/KeycloakDS" managed-executor-service="java:jboss/ee/concurrency/executor/default" managed-scheduled-executor-service="java:jboss/ee/concurrency/scheduler/default" managed-thread-factory="java:jboss/ee/concurrency/factory/default"/>
...

and the <SPI> section which is responsible to populating the database on initial launch:

...
<spi name=“connectionsJpa">
    <provider name="default" enabled="true">
        <properties>
            <property name="dataSource" value="java:jboss/datasources/KeycloakDS"/>
            <property name="initializeEmpty" value="true"/>
            <property name="migrationStrategy" value="update"/
            <property name="migrationExport" value="${jboss.home.dir}/keycloak-database-update.sql"/>
        </properties>
    </provider>
</spi>
...

At this stage the Keycloak master node is ready to be started. You can do so by issuing the following command from Keycloak’s bin/ directory:

$ sudo ./domain.sh --host-config=host-master.xml \
-Djboss.bind.address.private=[[Master Node Public IP]] \
-Djboss.bind.address=[[Master Node Public IP]]

If any errors are displayed check the possible error messages section later in this document.

Setting up the Slave Identity Server

The slave identity server can use the same deployment as the master, with the added difference of launching Keycloak using the host-slave.xml configuration.

Copy the configured keycloak directory to the target slave server and edit the host-slave.xml file as follows:

<management>
        ​<security-realms>
            ​<security-realm name="ManagementRealm">
                ​<server-identities>
                    ​<secret value="bWdtdDEyMyE="/>

Use the secret saved from earlier when creating the Management User from the ./add-user.sh script.

Next we need to provide the username associated with the above secret as well by editing the following section:

...
<remote username="wadmin" security-realm="ManagementRealm">
    <discovery-options>
        <static-discovery name="primary" protocol="${jboss.domain.master.protocol:remote}" host="${jboss.domain.master.address:[[Master Host IP]]}" port="${jboss.domain.master.port:9999}"/>
    </discovery-options>
</remote>
...

Save the file and launch the slave node using the following command:

$ ./domain.sh --host-config=host-slave.xml \
-Djboss.bind.address.private=[[Slave Node Public IP]] 
-Djboss.bind.address=[[Slave Node Public IP]]

By default Keycloak is configured to bootstrap one master and one slave node named respectively server1 and server2. Additional nodes can be added as needed by modifying the host-slave.xml and renaming the server name in the <server> section. All servers belong by default to a group named “auth-server-group”.

Possible Error Messages

As with all complex deployments, it is very probable errors will be encountered. Needless to say that the first thing to check would be any typos such as misspelled configuration names or options and unclosed tags.

Keycloak’s startup output provides a very detailed error logging which usually allows for efficient tracing.

Two of the more obscure errors encountered when testing Keycloak and resource strained system posed more fo a challenge to remedy. These are included below for reference:

WFLYCTL0348 Timeout:

This error usually happens when using a database backend and running Keycloak on a system with constrained resources. The Wildfly management subsystem will timeout before the resources are avaialble to Keycloak and throw the following error:

ERROR [org.jboss.as.controller.management-operation] (Controller Boot Thread) WFLYCTL0348: Timeout after [300] seconds waiting for service container stability. Operation will roll back. Step that first updated the service container was 'add' at address '[("core-service" => "management"),("management-interface" => "http-interface")]'

To fix this error we need to modify the management.blocking.timeout property by adding it to the domain.xml configuration file:

...
</extensions>
    <system-properties>
        <property name="java.net.preferIPv4Stack" value="true"/>
        <property name="jboss.as.management.blocking.timeout" value="1800"/>
    </system-properties>
<management>
...

ARJUNA012117: TransactionReaper::Check Timeout

Another database and timeout related error stems from the Arjuna TX Manager timing out before completing the initial table creation in the DB and not releasing the Tx lock in a timely manner.

The full error may look something like this:

ARJUNA012117: TransactionReaper::Check Timeout For TX 0:Ffff0a400e22:2251d537:519c37fa:35d4

To address the timeout, the coordinator environment timeout should be tweaked in the domain.xml configuration in order to give the Tx manager more time to finish its operations on the database:

<coordinator-environment statistics-enabled="${wildfly.transactions.statistics-enabled:${wildfly.statistics-enabled:false}}" default-timeout="1800"/>

Conclusion

Keycloak is an excellent IAM solution and offers excellent support out of the box to standard authentication and authorization mechanisms such as OAuth2.0, OpenID Connect and SAML2.0.

Integrating applications with Keycloak is also straightforward depending on the used development environment which can range from Python to .NET. Examples using .NET may be published at a later date to provide working proof using both .NET Core and .NET Framework 4.7 via Owin.

When deploying Keycloak in a production environment the standalone and standalone-ha modes may not provide, respectively, the necessary fault tolerance, centralized management and scalability needed for most medium to large businesses and enterprises.

This guide offers a brief overview of setting up Keycloak in a Domain Clustered mode and is intended to augment the official Keycloak documentation.

For further information you can visit the reference links listed below for a more in-depth documentation on deploying, managing and using Keycloak.

References

• Keycloak Official Documentation for version 12.0.1
  https://www.keycloak.org/docs/latest/server_installation/index.html
• Wildfly Official Documentation
  http://docs.wildfly.org/22/Admin_Guide.html
• Janua: How to install Keycloak with MariaDB
  https://www.janua.fr/how-to-install-keycloak-with-mariadb/
• Setting up Keycloak Standalone with MySQL Database
  https://medium.com/@pratik.dandavate/setting-up-keycloak-standalone-with-mysql-database-7ebb614cc229