PingDirectory Deployments Across Kubernetes Clusters ¶
The following example is an extension of the topic Orchestrating a replicated PingDirectory deployment in Kubernetes orchestration for general use. In this example, you'll deploy PingDirectory containers across multiple Kubernetes clusters.
Overview ¶
Having a replicated PingDirectory topology across multiple kubernetes clusters is desired for highly-available active/active deployments as well as active/partial-active scenarios where a hot backup is expected.
Ping Identity PingDirectory Docker images abstract away much of the complexity of replication initialization scripts, even across clusters. Instead, the focus is on providing accessible DNS hostnames across clusters and environment variables to build ordinal hostnames for each Directory instance.
About PingDirectory clusters ¶
You will need the following:
- PingDirectory Container Keys To Success.
- Draft Hostnames - Determine the variables needed to create your hostnames.
- Additional Variables - Use these variables for use-case flexibility.
- Cluster Startup Background - Walk through what happens when a cluster starts.
- Design Infrastructure to Match Hostnames - Based on your infrastructure constraints, you might need to alter your hostname plans. Use these reference examples to help.
11-replication-timing
- Use this folder to test replication speeds in the deployment.
Because details within each Kubernetes cluster are well-hidden from outside the cluster, external access to each pod within the cluster is required. The PingDirectory images set up access to each of the pods using load balancers from an external host to allow each pod to communicate over the LDAP and replication protocols.
PingDirectory Host Naming ¶
The most important aspect of a successful PingDirectory cross-cluster deployment assigning accessible and logical DNS hostnames according to the following rules:
- Each PingDirectory needs its own hostname available in DNS.
- The hostname has a space for the ordinal representing the instance in the
statefulset
. - All hostnames are accessible to all directory instances.
These rules still leave plenty of room for flexibility, especially when accounting for cluster-native DNS names Kubernetes creates.
Single-Cluster Multiple-Namespace ¶
If you were to simulate a "multi-cluster" environment in a single cluster, you could just set up two namespaces and create a separate ClusterIP service for each directory, as follows:
Primary Cluster ¶
Pod | Service Name | Namespace | Hostname |
---|---|---|---|
pindirectory-0 | pingdirectory-0 | primary | pingdirectory-0.primary |
pindirectory-1 | pingdirectory-1 | primary | pingdirectory-1.primary |
pindirectory-2 | pingdirectory-2 | primary | pingdirectory-2.primary |
Secondary Cluster ¶
Pod | Service Name | Namespace | Hostname |
---|---|---|---|
pindirectory-0 | pingdirectory-0 | secondary | pingdirectory-0.secondary |
pindirectory-1 | pingdirectory-1 | secondary | pingdirectory-1.secondary |
pindirectory-2 | pingdirectory-2 | secondary | pingdirectory-2.secondary |
External DNS Names ¶
In a Prod Environment with external hostnames, it might look more like:
us-west cluster ¶
Pod | Service Name | DNS / Hostname |
---|---|---|
pindirectory-0 | pingdirectory-0 | pingdirectory-0-us-west.ping-devops.com |
pindirectory-1 | pingdirectory-1 | pingdirectory-1-us-west.ping-devops.com |
pindirectory-2 | pingdirectory-2 | pingdirectory-2-us-west.ping-devops.com |
us-east cluster ¶
Pod | Service Name | DNS / Hostname |
---|---|---|
pindirectory-0 | pingdirectory-0 | pingdirectory-0-us-east.ping-devops.com |
pindirectory-1 | pingdirectory-1 | pingdirectory-1-us-east.ping-devops.com |
pindirectory-2 | pingdirectory-2 | pingdirectory-2-us-east.ping-devops.com |
Variables to Create Hostnames ¶
To provide flexibility on how PingDirectory finds other instances, a full DNS hostname is broken into multiple variables.
Variable | Description |
---|---|
K8S_POD_HOSTNAME_PREFIX |
The string used as the prefix for all host names. Defaults to name of StatefulSet . |
K8S_POD_HOSTNAME_SUFFIX |
The string used as the suffix for all pod host names. Defaults to K8S_CLUSTER . |
K8S_SEED_HOSTNAME_PREFIX |
The string used as the prefix for all seed host names. Defaults to K8S_POD_HOSTNAME_PREFIX . |
K8S_SEED_HOSTNAME_SUFFIX |
The string used as the suffix for all seed host names. Defaults to K8S_SEED_CLUSTER (discussed later). |
A full hostname is created using the following command:
${K8S_POD_HOSTNAME_PREFIX}<instance-ordinal>${K8S_SEED_HOSTNAME_SUFFIX}
Using Previous Hostname Examples ¶
| hostname | K8S_POD_HOSTNAME_PREFIX | K8S_SEED_HOSTNAME_PREFIX | K8S_POD_HOSTNAME_SUFFIX | K8S_SEED_HOSTNAME_SUFFIX |
|---|:---:|---|---|
| pingdirectory-0.primary|pingdirectory-
| pingdirectory-
| .primary
|.primary
|
| pingdirectory-2-us-west.ping-devops.com| pingdirectory-
| pingdirectory-
| -us-west.ping-devops.com
|-us-west.ping-devops.com
|
Environment Variables ¶
Variable | Required | Description |
---|---|---|
K8S_CLUSTERS |
*** | The total list of Kubernetes clusters that the StatefulSet replicates to. |
K8S_CLUSTER |
*** | The Kubernetes cluster the StatefulSet is deployed to. |
K8S_SEED_CLUSTER |
*** | The Kubernetes cluster that the seed server is deployed to. |
K8S_NUM_REPLICAS |
The number of replicas that make up the StatefulSet . |
|
K8S_POD_HOSTNAME_PREFIX |
The string used as the prefix for all host names. Defaults to StatefulSet . |
|
K8S_POD_HOSTNAME_SUFFIX |
The string used as the suffix for all pod host names. Defaults to K8S_CLUSTER . |
|
K8S_SEED_HOSTNAME_PREFIX |
The string used as the prefix for all seed host names. Defaults to K8S_POD_HOSTNAME_PREFIX . |
|
K8S_SEED_HOSTNAME_SUFFIX |
The string used as the suffix for all seed host names. Defaults to K8S_SEED_CLUSTER . |
|
K8S_INCREMENT_PORTS |
true or false . If true , each pod's port will be incremented by 1. |
An example of the YAML configuration for these environment variables:
K8S_STATEFUL_SET_NAME=pingdirectory
K8S_STATEFUL_SET_SERVICE_NAME=pingdirectory
K8S_CLUSTERS=us-east-2 eu-west-1
K8S_CLUSTER=us-east-2
K8S_SEED_CLUSTER=us-east-2
K8S_NUM_REPLICAS=3
K8S_POD_HOSTNAME_PREFIX=pd-
K8S_POD_HOSTNAME_SUFFIX=.us-cluster.ping-devops.com
K8S_SEED_HOSTNAME_PREFIX=pd-
K8S_SEED_HOSTNAME_SUFFIX=.us-cluster.ping-devops.com
K8S_INCREMENT_PORTS=true
LDAPS_PORT=8600
REPLICATION_PORT=8700
These environment variable settings would map out as follows:
Seed | Pod | Instance | Host name | LDAP | REPL |
---|---|---|---|---|---|
CLUSTER: us-east-2 | |||||
*** | *** | pingdirectory-0.us-east-2 | pd-0.us-cluster.ping-devops.com | 8600 | 8700 |
pingdirectory-1.us-east-2 | pd-1.us-cluster.ping-devops.com | 8601 | 8701 | ||
pingdirectory-2.us-east-2 | pd-2.us-cluster.ping-devops.com | 8602 | 8702 | ||
CLUSTER: eu-west-1 | |||||
pingdirectory-0.eu-west-1 | pd-0.eu-cluster.ping-devops.com | 8600 | 8700 | ||
pingdirectory-1.eu-west-1 | pd-1.eu-cluster.ping-devops.com | 8601 | 8701 | ||
pingdirectory-2.eu-west-1 | pd-2.eu-cluster.ping-devops.com | 8602 | 8702 |
Cluster Startup Walkthrough ¶
This in-depth variable conversation offers the flexibility to accommodate various infrastructure constraints. For example, in some environments, you can't use the same port for each instance, so we must accommodate incrementing ports.
To understand why the initial creation of a cluster requires a prescriptive approach, it's helpful to know what happens when a cluster starts.
- The first pod must start on its own and become healthy. This is critical to prevent replication islands. The first time the first pod starts, we call it "GENESIS". All other pods are dependent on this
SEED_POD
in theSEED_CLUSTER
starting correctly on its own. The entire purpose of definingSEED_POD
andSEED_CLUSTER
variables is to avoid multiple genesis scenarios.
Because we're deploying a
statefulset
, you can deploy the entire first cluster at once. Astatefulset
creates one pod in the number of replicas at a time.
- When the first pod is healthy, it begins DNS querying combinations of hostnames at their LDAPS port to find another Directory instance.
In our first cluster, this would be the hostname of pingdirectory-1
. but it could also be pingdirectory-0
of another cluster. When the query returns successful, creation of the replication topology automatically begins.
- From this point onward, the order in which instances start is less important.
Note on Replication Traffic ¶
It might not always be clear what truly happens during "replication". Though it is partially proprietary, you can think of it as:
-
A modify request arrives at one pod.
-
The corresponding Directory instance, say
pingdirectory-0
, makes the change locally and tracks the change in the changelog. -
The instance broadcasts the change out to all other instances with the time of change. This request is added to a message queue of sorts on the other instances and processed in order.
Because of this, horizontal scaling of directories isn't something to be taken lightly.
Reference Modes of Deployment ¶
There are multiple types of deployments that have been tested because of various infrastructure constraints discussed in this guide. The example files in 09-multi-k8s-pingdirectory
show that the biggest key is having a way to provide an accessible hostname to all the other pods. In most examples, you do this by creating a service name and using statefulset.kubernetes.io/pod-name
as the selector to isolate one pod.
Multi-cluster Simulations ¶
These are examples for demo environments to get a feel for what a multi-region deployment looks like.
Single Namespace ¶
20-kustomize/09-multi-k8s-pingdirectory/01-single-namespace
is the least constrained example. It's good to just see what logs on a cross-cluster topology look like:
- Relies only on DNS names that kubernetes provides.
- All traffic is in one namespace, so it should have no network constraints.
kubectl apply -f 01-west.yaml
- Wait for
pingdirectory-0
to be healthy.
kubectl apply -f 02-east.yaml
Note, logs with stern look better.
brew install stern
-
Watch the logs.
stern pingdirectory
or
sh
kubectl logs -f -l role=pingdirectory
When the all the instances are up and running, you should see something similar to:
Single Cluster Multiple Namespaces ¶
20-kustomize/09-multi-k8s-pingdirectory/02-single-cluster-two-namespaces
-
Use this example when you only have one cluster available for testing.
The files in this example are templates and expect namespaces to be added.
export NAMESPACE_1=<west-namespace> export NAMESPACE_2=<east-namespace> envsubst < 03-multi-cluster-dns/01-west.yaml | kubectl apply -f - envsubst < 03-multi-cluster-dns/02-east.yaml | kubectl apply -f -
-
Watch logs.
VPC Peered K8s Clusters ¶
These example should be possible in most Kubernetes providers as long as you can:
- Give external dns names to clusterIP services.
- Have replication and ldaps ports peered (open) between clusters.
Consider the EKS Peering Config example if you want to test this.
Using External DNS Names ¶
20-kustomize/09-multi-k8s-pingdirectory/03-multi-cluster-dns
This example uses Headless Services instead of regular clusterIp services. This is necessary in a VPC-peered environment because typically the route-tables and IP ranges you've peered correspond to container IP addresses, not service addresses.
If you were to use clusterIp addresses, the instances might, unexpectedly, not have network connectivity to each other.
The headless services use externalDNS to dynamically add records to the DNS provider (example, Route53)
Because this example requires more care, you should sift through the yamls to understand what is going on.
When the example is stood up, you'll see logs similar to:
Without VPC Peering ¶
Some organizations don't allow VPC peering or similar networking functions, or there might be no way to create external hostnames on clusterIp services. Here are some examples that may help.
Using NodePorts ¶
In a scenario where you don't have VPC peering or must create external DNS names manually, it might be beneficial to use NodePorts.
To use NodePorts:
- Use the
20-kustomize/09-multi-k8s-pingdirectory/03-multi-cluster-dns
example as reference - Make the pod-selector services NodePort services instead of clusterIp.
- Optionally, remove the external name and create a routable dns name.
If these names are being manually, then create the services and assign names before starting the statefulset.
Single Load Balancer ¶
Caution
The following examples are for extremely constrained environments where traffic must go out through an external load balancer. For many purposes, these can be considered deprecated. Avoid doing replication through load balancers when possible.
The following diagram shows how you can use a single load balancer.
Advantages:
- Decreased cost of a single load balancer
- Single IP required
- Easier DNS management
- Wildcard DNS domain
- Or separate hosts pointing to load balancer
Disadvantages:
- More port mapping requirements
- Many external ports to manage and track
Multiple Load Balancers ¶
The following diagram shows how you can use muliple load balancers.
Advantages:
- Use the same well-known port, such as 1636/8989
- Separate IP addresses per instance
Disadvantages:
- DNS management
- Separate hostname required per instance
StatefulSet
Pod Services ¶
The StatefulSet
service manages stateful objects for each pod.
The following is an example StatefulSet
service configuration for one pod:
kind: Service
apiVersion: v1
metadata:
name: pingdirectory-0-service
spec:
type: ClusterIP
selector:
statefulset.kubernetes.io/pod-name: pingdirectory-0
ports:
- protocol: TCP
port: 8600
targetPort: 8600
name: ldaps
- protocol: TCP
port: 8700
targetPort: 8700
name: repl
Additional Kubernetes Resources Required ¶
In addition to the StatefulSet
, other resources are required to properly map the load balancers to the
pods. The following diagram shows each of those resources.
DNS ¶
A DNS entry is required at the load balancer to direct a wildcard domain or individual hostname
to the load balancer created by the NGINX Ingress Service or Controller. For AWS, this can simply be an A record
alias for each host or a wildcard A record
for any host in that domain.
NGINX Ingress Service and Controller ¶
Several components map the ports from the external load balancer through the NGINX Service and Controller:
- External load balancer
Provides an external IP and obtains definitions from the NGINX Ingress Service.
- NGINX Ingress Service
Maps all port ranges (SEED_LDAPS_PORT, SEED_REPLICATION_PORT) to the same target port range.
- NGINX Ingress Controller
Maps all port ranges to stateful set pods.
Warning
Typically, the NGINX Service and TCP services require additional namespace access, ingress-nginx-public
. Any additional applications using this service or controller generally requires additional privileges to manage this resource. For more information, see the NGINX TCP services topic.
The following is an example NGINX Service configuration:
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/role: ingress-nginx-public
namespace: ingress-nginx-public
annotations:
service.beta.kubernetes.io/aws-load-balancer-type: nlb
spec:
selector:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/role: ingress-nginx-public
externalTrafficPolicy: Local
type: load-balancer
ports:
- name: http
port: 80
targetPort: http
- name: https
port: 1443
targetPort: https
- name: ldaps-pingdiretory-0
port: 8600
targetPort: 8600
- name: ldaps-pingdiretory-1
port: 8601
targetPort: 8601
- name: ldaps-pingdiretory-2
port: 8602
targetPort: 8602
- name: repl-pingdiretory-0
port: 8700
targetPort: 8700
- name: repl-pingdiretory-1
port: 8701
targetPort: 8701
- name: repl-pingdiretory-2
port: 8702
targetPort: 8702
NGINX TCP Services ¶
The ConfigMap for TCP services (tcp-services
) provides the mappings from the target ports on the NGINX Controller to the associated pod service.
You must replace the variable
${PING_IDENTITY_K8S_NAMESPACE}
with the namespace that yourStatefulSet
and Services are deployed into.
The following is an example of the ConfigMap for the NGINX TCP services configuration:
apiVersion: v1
kind: ConfigMap
metadata:
name: tcp-services
namespace: ingress-nginx-public
labels:
app.kubernetes.io/name: ingress-nginx
app.kubernetes.io/part-of: ingress-nginx
app.kubernetes.io/role: ingress-nginx-public
data:
8600: "${PING_IDENTITY_K8S_NAMESPACE}/pingdirectory-0-service:8600"
8601: "${PING_IDENTITY_K8S_NAMESPACE}/pingdirectory-1-service:8601"
8602: "${PING_IDENTITY_K8S_NAMESPACE}/pingdirectory-2-service:8602"
8700: "${PING_IDENTITY_K8S_NAMESPACE}/pingdirectory-0-service:8700"
8701: "${PING_IDENTITY_K8S_NAMESPACE}/pingdirectory-1-service:8701"
8702: "${PING_IDENTITY_K8S_NAMESPACE}/pingdirectory-2-service:8702"
Deployment Example ¶
The examples in 20-kustomize/05-multi-k8s-cluster-pingdirectory
create an
example deployment across two clusters in AWS EKS:
- us-east-2 (SEED Cluster)
- eu-west-1
Deploy the NGINX services and ConfigMap for the example.
This allows the services to be reached by the NGINX controller with an AWS Network Load Balancer (nlb). You must run these with an AWS/Kubernetes profile allowing for application into the ingress-nginx-public
namespace. Also, be aware that there might already be other definitions found. You might need to merge.
kubectl apply -f nginx-service.yaml
kubectl apply -f nginx-tcp-services.yaml
The cluster.sh
script creates the .yaml necessary to deploy a set of PingDirectory instances in each cluster and replication between.
Usage: cluster.sh OPERATION {options}
where OPERATION in:
create
apply
delete
where options in:
--cluster {cluster} - Cluster name used to identify different
env_vars.pingdirecory files
--context {context} - Name of Kubernetes context.
Defaults to current context:
jsmith.ping-dev-aws-us-east-2
-d,--dry-run - Provides the commands
Example:
cluster.sh create --cluster us-east-2
Creating .yaml for us-east-2 ¶
-
Replace
your-cluster-name
with the name you are using. Using the cluster nameus-east-2
the script generates a .yaml using kustomize, with the files:multi-cluster
kustomization.yaml
pingdirectory-service-clusterip.yaml
env_vars.pingdirectory
(built from env_vars.pingdirectory.multi-cluster and us-east-2)
base
kustomization.yaml
https://github.com/pingidentity/pingidentity-devops-getting-started/20-kustomize/03-replicated-pingdirectory
env_vars.pingdirectory
limits.yaml
./cluster.sh delete \ --cluster us-east-2 \ --context <your-cluster-name> \ --dry-run
This creates a .yaml called
ouptut-us-east-2.yaml
. -
Ensure that your
devops-secret
andtls-secret
are created.ping-devops generate devops-secret | kubectl apply -f - ping-devops generate tls-secret ping-devops.com | kubectl create -f -
-
Create the instances using the generated
output-us-east-2.yaml
.kubectl create -f output-us-east-2.yaml