Deploy a Kubernetes Cluster Metrics Stack ¶
This document covers deploying and using a sample open-source monitoring stack in a Kubernetes cluster. The resulting environment should not be considered production-ready. It is meant to show how quickly Ping DevOps software can produce metrics for consumption by a popular open-source monitoring system. This example metrics stack is not maintained or directly supported by Ping.
Kubernetes version notice
The Prometheus operator used for this example is incompatible with Kubernetes v1.25 or later due to the removal of the deprecated PodSecurityPolicy object with that release. As of September 2022, the operator only supports up through Kubernetes v1.24. Until this is updated, your Kubernetes cluster must be V1.24 or lower to follow the example on this page.
See the compatibility chart at the Prometheus operator repository for more information.
Stack Components ¶
Open Source Tools
- kube-prometheus-stack - includes:
- Prometheus - Metrics collection and storage.
- Grafana - Metrics visualization in Dashboards.
- telegraf-operator - Metrics exposure and formatting.
Grafana Dashboard - JSON file to import for dashboard definition.
ping-devops values.yaml - values relevant only to exposing metrics for Ping Identity software.
It is assumed you are familiar with the prerequisites for the base Helm examples. Beyond that, any knowledge of Prometheus, Grafana, and Telegraf is helpful.
Deploy the Stack ¶
pingidentity-devops-getting-started/30-helm/cluster-metrics directory of this repository, edit the
01-prometheus-values.yaml as needed. This file provides configurations beyond the default kube-prometheus-stack. In this sample deployment, the monitoring stack is granted read access to the entire cluster and is deployed into the
Not for production use
Changing these settings or making the deployment production-ready is beyond scope of this document. The full set of optional values can be found on the Github repository for the Prometheus chart.
There are numerous lines that have
##CHANGEME. These lines should be considered for configuration options to meet your needs.
After updating the file, deploy the
kube-prometheus-stack. The path to the configuration file assumes you are in the root folder of the local repository copy:
kubectl create namespace metrics helm upgrade --install metrics --repo https://prometheus-community.github.io/helm-charts kube-prometheus-stack -n metrics --version 30.0.1 -f 30-helm/cluster-metrics/01-prometheus-values.yaml
helm upgrade --install telegraf --repo https://helm.influxdata.com/ telegraf-operator -n metrics --version 1.3.3 -f 30-helm/cluster-metrics/02-telegraf-values.yaml
Telegraf operator makes it very easy to add monitoring sidecars to your deployments. All you need to do is add annotations, which are shown in
These values can be copied to your ping-devops
values.yaml manually, or the file can be referenced at the end of your helm install command. For example:
helm upgrade --install ping-metrics pingidentity/ping-devops -f my-values.yaml -f 30-helm/cluster-metrics/03-ping-with-metrics-values.yaml
After the Ping software is healthy and producing metrics, there should be sidecars on Ping pods.
NAME READY STATUS ping-metrics-pingaccess-admin-0 1/1 Running ping-metrics-pingaccess-engine-68464d8cc8-mhlsv 2/2 Running ping-metrics-pingdataconsole-559786c98f-8wsrm 1/1 Running ping-metrics-pingdirectory-0 2/2 Running ping-metrics-pingfederate-admin-64fdb4b975-2xdjl 1/1 Running ping-metrics-pingfederate-engine-64c5f896c7-fn99v 2/2 Running
2/2 indicator for pods with sidecars.
View Metrics ¶
Browse to Grafana using the Ingress URL or by running a
kubectl port-forward command. For example:
kubectl port-forward svc/metrics-grafana --namespace metrics 9000:80
In your browser, navigate to
http://localhost:9000 and log in with the user
admin and the password set in
Finally, import the
04-ping-overview-dashboard.json using the
+ button on the left of Grafana's home screen.
Ping Identity Overview dashboard will have a dropdown for namespace at the top. Select the namespace running Ping products to see something similar to this example:
Any of the panels can be edited, or new ones created to fit your needs.
Provided that you use the
autoscaling/v2 API version, you can configure a HorizontalPodAutoscaler to scale based on a custom metric (that is not built in to Kubernetes or any Kubernetes component).
If you are using our Helm Charts, you can pass the custom metrics under
global.cluster.autoscalingMetricsTemplate. The example code here will scale on a requests-per-second threshold of 10,000:
- type: Pods pods: metric: name: custom-metric target: type: AverageValue averageValue: 10000m - type: Resource resource: name: cpu target: type: Utilization averageUtilization: 50 - type: Object object: metric: name: requests-per-second describedObject: apiVersion: networking.k8s.io/v1 kind: Ingress name: main-route current: value: 10k
As well as the behaviors for scaling up and down under
scaleDown: stabilizationWindowSeconds: 300 policies: - type: Percent value: 100 periodSeconds: 15 scaleUp: stabilizationWindowSeconds: 0 policies: - type: Percent value: 100 periodSeconds: 15 - type: Pods value: 4 periodSeconds: 15 selectPolicy: Max
For more information on custom HPA metrics please visit Kubernetes