Runtimes Monitoring


Go back to index

Kiali can display custom dashboards to monitor application metrics. They are available for Applications and Workloads.

  In order to display custom dashboards, you must set the app and version labels on your pods. These labels are necessary for Kiali to identify which application and workload the metrics originate from.

1. Prometheus Configuration

Kiali runtimes monitoring feature works exclusively with Prometheus, so it must be configured correctly to pull your application metrics.

If you are using the default Istio installation, your Prometheus instance should already be configured as shown below and you can skip to the next section.

But if you want to use another Prometheus instance, please refer to Prometheus documentation to setup kubernetes_sd_config for pods. As a reference, here is how it is configured in Istio.

It is important to preserve label mapping, so that Kiali can filter on app and version. This is done with this config:

      - action: labelmap
        regex: __meta_kubernetes_pod_label_(.+)

Additionally, you have to tell Kiali the URL of that Prometheus server: either by setting the environment variable PROMETHEUS_CUSTOM_METRICS_URL or by modifying the Kiali ConfigMap:

# ...
external_services:
  prometheus_custom_metrics_url: URL_TO_SERVER
# ...

Make sure that Kiali pod can reach this URL.

2. Pods Annotations

Application pods must be annotated, both for the Prometheus scraper and to associate with Kiali dashboards. For instance, within a Deployment definition:

spec:
  template:
    metadata:
      annotations:
        prometheus.io/scrape: "true"
        prometheus.io/port: "8080"
        prometheus.io/scheme: http
        prometheus.io/path: "/metrics"
        kiali.io/runtimes: vertx-server
  • prometheus.io/scrape tells Prometheus to fetch these metrics or not

  • prometheus.io/port is the port under which metrics are exposed

  • prometheus.io/scheme must be set to http for non-secure or https for secure connection

  • prometheus.io/path is the endpoint path where metrics are exposed, default is /metrics

  • kiali.io/runtimes is a coma-separated list of runtimes / dashboards that Kiali will look for

3. Default dashboards

Since version 0.15, Kiali comes with a set of default dashboards for three different runtimes: Node.js, Thorntail and Vert.x. In Kiali 0.16, we’ve added Go and Spring Boot.

3.1. Go

Contains metrics such as the number of threads, goroutines, and heap usage. The expected metrics are provided by the Prometheus Go client.

Example to expose default Go metrics:

        http.Handle("/metrics", promhttp.Handler())
        http.ListenAndServe(":2112", nil)

As an example and for self-monitoring purpose Kiali itself exposes Go metrics.

The pod annotation for Kiali is: kiali.io/runtimes: go

3.2. Node.js

Contains metrics such as active handles, event loop lag, and heap usage. The expected metrics are provided by prom-client.

Example of Node.js metrics for Prometheus:

const client = require('prom-client');
client.collectDefaultMetrics();
// ...
app.get('/metrics', (request, response) => {
  response.set('Content-Type', client.register.contentType);
  response.send(client.register.metrics());
});

The pod annotation for Kiali is: kiali.io/runtimes: nodejs

3.3. Spring Boot

Two dashboards are provided: one for JVM metrics and another for Tomcat metrics. The expected metrics come from Spring Boot Actuator for Prometheus. Example with maven:

    <dependency>
      <groupId>org.springframework.boot</groupId>
      <artifactId>spring-boot-starter-actuator</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-core</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

The pod annotation for Kiali with the full list of dashboards is: kiali.io/runtimes: springboot-jvm,springboot-tomcat

By default, the metrics are exposed on path /actuator/prometheus, so it must be specified in the corresponding annotation: prometheus.io/path: "/actuator/prometheus"

3.4. Thorntail

Contains mostly JVM-related metrics such as loaded classes count, memory usage, etc. The expected metrics are provided by the MicroProfile Metrics module. Example with maven:

    <dependency>
      <groupId>io.thorntail</groupId>
      <artifactId>microprofile-metrics</artifactId>
    </dependency>

The pod annotation for Kiali is: kiali.io/runtimes: thorntail

3.5. Vert.x

Several dashboards are provided, related to different components in Vert.x: HTTP client/server metrics, Net client/server metrics, Pools usage, and Eventbus metrics. The expected metrics are provided by the vertx-micrometer-metrics module. Example with maven:

    <dependency>
      <groupId>io.vertx</groupId>
      <artifactId>vertx-micrometer-metrics</artifactId>
    </dependency>
    <dependency>
      <groupId>io.micrometer</groupId>
      <artifactId>micrometer-registry-prometheus</artifactId>
    </dependency>

Init example of Vert.x metrics, starting a dedicated server (other options are possible):

      VertxOptions opts = new VertxOptions().setMetricsOptions(new MicrometerMetricsOptions()
          .setPrometheusOptions(new VertxPrometheusOptions()
              .setStartEmbeddedServer(true)
              .setEmbeddedServerOptions(new HttpServerOptions().setPort(9090))
              .setPublishQuantiles(true)
              .setEnabled(true))
          .setEnabled(true));

The pod annotation for Kiali with the full list of dashboards is: kiali.io/runtimes: vertx-client,vertx-server,vertx-eventbus,vertx-pool

4. Create new dashboards

The default dashboards described above are just examples of what we can have. It’s pretty easy to create new dashboards.

When installing Kiali, a new CRD is installed in the system: monitoringdashboard.monitoring.kiali.io. It declares the resource kind MonitoringDashboard. Here’s what this resource looks like:

apiVersion: "monitoring.kiali.io/v1alpha1"
kind: MonitoringDashboard
metadata:
  name: vertx-server
spec:
  runtime: Vert.x
  title: Vert.x Server Metrics
  charts:
    - name: "Server response time"
      unit: "s"
      spans: 6
      metricName: "vertx_http_server_responseTime_seconds"
      dataType: "histogram"
      aggregations:
        - label: "path"
          displayName: "Path"
        - label: "method"
          displayName: "Method"
    - name: "Server active connections"
      unit: ""
      spans: 6
      metricName: "vertx_http_server_connections"
      dataType: "raw"

The name field (from metadata) corresponds to what you will have to set in pods annotation kiali.io/runtimes.

Spec fields definitions are:

  • runtime: name of the related runtime. It will be displayed on the corresponding Workload Details page. If omitted no name is displayed.

  • title: dashboard title, displayed as a tab in Application or Workloads Details

  • charts: list of charts in the dashboard

    • name: name of the chart

    • unit: unit for Y-axis. This unit may be converted using metric SI system, for instance "s" can be shown as "ms" or "┬Ás" as well.

    • spans: number of "spans" taken by the chart, from 1 to 12, using bootstrap convention

    • metricName: the metric name in Prometheus

    • dataType: type of data to be displayed in the chart. Can be one of raw, rate or histogram. Raw data will be queried without transformation. Rate data will be queried using promQL rate() function. And histogram with histogram_quantile() function.

    • aggregator: defines how the time-series are aggregated when several are returned for a given metric and label set. For example, if a Deployment creates a ReplicaSet of several Pods, you will have at least one time-series per Pod. Since Kiali shows the dashboards at the workload (ReplicaSet) level or at the application level, they will have to be aggregated. This field can be used to fix the aggregator, with values such as sum or avg (full list available in Prometheus documentation). However, if omitted the aggregator will default to sum and can be changed from the dashboard UI.

    • aggregations: list of labels eligible for aggregations / groupings (they will be displayed in Kiali through a dropdown list)

      • label: Prometheus label name

      • displayName: Name to display in Kiali

Dashboard resources are added in Kubernetes just like any other resource:

kubectl apply -f mydashboard.yml

Or for OpenShift:

oc apply -f mydashboard.yml

To make the dashboard resources available cluster-wide, just create them in Kiali namespace (usually istio-system). Else, they will be available only for applications or workloads of the same namespace. In the case where the same dashboard name exists in a specific namespace and in Kiali namespace, the former takes precedence.