Deploy a Kubernetes Cluster Metrics Stack ¶
This document covers deploying and using a sample open-source monitoring stack in a Kubernetes cluster. The resulting environment should not be considered production-ready. It is meant to show how quickly Ping DevOps software can produce metrics for consumption by a popular open-source monitoring system. This example metrics stack is not maintained or directly supported by Ping.
Kubernetes version notice
The Prometheus operator used for this example is incompatible with Kubernetes v1.25 or later due to the removal of the deprecated PodSecurityPolicy object with that release. As of September 2022, the operator only supports up through Kubernetes v1.24. Until this is updated, your Kubernetes cluster must be V1.24 or lower to follow the example on this page.
See the compatibility chart at the Prometheus operator repository for more information.
Stack Components ¶
Open Source Tools
- kube-prometheus-stack - includes:
- Prometheus - Metrics collection and storage.
- Grafana - Metrics visualization in Dashboards.
- telegraf-operator - Metrics exposure and formatting.
Grafana Dashboard - JSON file to import for dashboard definition.
ping-devops values.yaml - values relevant only to exposing metrics for Ping Identity software.
Prerequisites ¶
It is assumed you are familiar with the prerequisites for the base Helm examples. Beyond that, any knowledge of Prometheus, Grafana, and Telegraf is helpful.
Deploy the Stack ¶
In the pingidentity-devops-getting-started/30-helm/cluster-metrics directory
of this repository, edit the 01-prometheus-values.yaml
as needed. This file provides configurations beyond the default kube-prometheus-stack. In this sample deployment, the monitoring stack is granted read access to the entire cluster and is deployed into the metrics
namespace.
Not for production use
Changing these settings or making the deployment production-ready is beyond scope of this document. The full set of optional values can be found on the Github repository for the Prometheus chart.
There are numerous lines that have ##CHANGEME
. These lines should be considered for configuration options to meet your needs.
After updating the file, deploy the kube-prometheus-stack
. The path to the configuration file assumes you are in the root folder of the local repository copy:
kubectl create namespace metrics
helm upgrade --install metrics --repo https://prometheus-community.github.io/helm-charts kube-prometheus-stack -n metrics --version 30.0.1 -f 30-helm/cluster-metrics/01-prometheus-values.yaml
Deploy telegraf-operator
:
helm upgrade --install telegraf --repo https://helm.influxdata.com/ telegraf-operator -n metrics --version 1.3.3 -f 30-helm/cluster-metrics/02-telegraf-values.yaml
Telegraf operator makes it very easy to add monitoring sidecars to your deployments. All you need to do is add annotations, which are shown in 30-helm/cluster-metrics/03-ping-with-metrics-values.yaml
These values can be copied to your ping-devops values.yaml
manually, or the file can be referenced at the end of your helm install command. For example:
helm upgrade --install ping-metrics pingidentity/ping-devops -f my-values.yaml -f 30-helm/cluster-metrics/03-ping-with-metrics-values.yaml
After the Ping software is healthy and producing metrics, there should be sidecars on Ping pods.
NAME READY STATUS
ping-metrics-pingaccess-admin-0 1/1 Running
ping-metrics-pingaccess-engine-68464d8cc8-mhlsv 2/2 Running
ping-metrics-pingdataconsole-559786c98f-8wsrm 1/1 Running
ping-metrics-pingdirectory-0 2/2 Running
ping-metrics-pingfederate-admin-64fdb4b975-2xdjl 1/1 Running
ping-metrics-pingfederate-engine-64c5f896c7-fn99v 2/2 Running
Note the 2/2
indicator for pods with sidecars.
View Metrics ¶
Browse to Grafana using the Ingress URL or by running a kubectl port-forward
command. For example: kubectl port-forward svc/metrics-grafana --namespace metrics 9000:80
In your browser, navigate to http://localhost:9000
and log in with the user admin
and the password set in 01-prometheus-values.yaml
Finally, import the 04-ping-overview-dashboard.json
using the +
button on the left of Grafana's home screen.
The Ping Identity Overview
dashboard will have a dropdown for namespace at the top. Select the namespace running Ping products to see something similar to this example:
Any of the panels can be edited, or new ones created to fit your needs.
HorizontalPodAutoscaler ¶
Provided that you use the autoscaling/v2
API version, you can configure a HorizontalPodAutoscaler to scale based on a custom metric (that is not built in to Kubernetes or any Kubernetes component).
If you are using our Helm Charts, you can pass the custom metrics under global.cluster.autoscalingMetricsTemplate
. The example code here will scale on a requests-per-second threshold of 10,000:
- type: Pods
pods:
metric:
name: custom-metric
target:
type: AverageValue
averageValue: 10000m
- type: Resource
resource:
name: cpu
target:
type: Utilization
averageUtilization: 50
- type: Object
object:
metric:
name: requests-per-second
describedObject:
apiVersion: networking.k8s.io/v1
kind: Ingress
name: main-route
current:
value: 10k
As well as the behaviors for scaling up and down under global.cluster.autoscaling.behavior
.
scaleDown:
stabilizationWindowSeconds: 300
policies:
- type: Percent
value: 100
periodSeconds: 15
scaleUp:
stabilizationWindowSeconds: 0
policies:
- type: Percent
value: 100
periodSeconds: 15
- type: Pods
value: 4
periodSeconds: 15
selectPolicy: Max
For more information on custom HPA metrics please visit Kubernetes