Kube-Prometheus-Top [ kptop ]

A Python tool that provides Monitoring for Kubernetes Nodes, Pods, Containers, and PVCs resources on the terminal through Prometheus metircs

kptop2.mp4

📌 Try out the new beta features

kptop_0.0.7 (Introducing K8s Pod port-forward connection method))

Environment variables

Motivation

The resources metrics provided by the K8s APIs are very limited compared what what's scraped by Prometheus.

This tool is using Prometheus as a data source for metrics to display all the needed informations right on the terminal.

Project Status

Additional

Check the connection with Prometheus Server
Check the existence of the needed Prometheus metrics _{[ good for similar use cases ]}

Installation

Compatible with Python 3.6+

on PyPi

pip3 install kptop --upgrade

Environment Variables

ENV	Description	Default	Required
`KPTOP_CONNECTION_METHOD`	The way to connect to Prometheus server options: ['prometheus_endpoint', 'pod_portForward']		Yes

There are 2 options to connect KPtop to Prometheus:

With a Prometheus server endpoint
- Suitable if Prometheus is exposed (with an ingress for example)
With 'K8s pod port-forward' (Through K8s API-Server)
- Suitable if Prometheus is Not exposed (Only rechable within the K8s cluster)

`prometheus_endpoint` ENVs

ENV	Description	Default	Required
`KPTOP_PROMETHEUS_SERVER`	Prometheus server URL		Yes
`KPTOP_BASIC_AUTH_ENABLED`	Whether basic authentication is needed to connect to Prometheus	False	No
`KPTOP_PROMETHEUS_USERNAME`	Prometheus username		No
`KPTOP_PROMETHEUS_PASSWORD`	Prometheus password		No
`KPTOP_INSECURE`	Verify SSL certificate	False	No

`pod_portForward` ENVs

ENV	Description	Default	Required
`KPTOP_PROMETHEUS_POD_NAME`	Prometheus pod name		Yes
`KPTOP_PROMETHEUS_POD_PORT`	Prometheus port number	9090	No
`KPTOP_PROMETHEUS_POD_NAMESPACE`	The name space in which the Prometheus pod is deployed	default	No
`KUBECONFIG`	custom K8s kube config file	default path	No

General ENVs

ENV	Description	Default	Required
`KPTOP_NODE_EXPORTER_NODE_LABEL`	node exporter "node label"	"node"	NO
`KPTOP_START_GRAPHS_WITH_ZERO`	By default graphs begin with '0' to let the graph take its full hight	True	NO
`KPTOP_LOGGING_DIR`	Choose a different logging directory	/tmp/	NO
`KPTOP_GRAPH_WIDTH`	Choose a custom graphs width	45	NO

CLI Arguments

ENV	Description	Default
`--namespace`, `-n`	Specify a Kubernetes Namespace	default
`--all-namespaces`, -A
`--container`, `-c`	Specify a container
`--interval`, `-i`	Live monitoring update interval	8 _{[NOTE: the actuall update depends on the Prometheus scaping interval (15s by default)]}
`--debug`, `-d`	Enable debugging logging mode	False
`--verify-prometheus`, `-V`	Verify connectivity to Prometheus server & check the existence of the needed exporters
`--sort-by-mem-usage`, `-s`	Sort top result by memory usage	False
`--check-metrics` , `-C`	Check the existence of the needed metrics, needs to be done with `-V`	False

Usage

Add the ENVs and you're good to go.

Examples

export KPTOP_CONNECTION_METHOD="pod_portForward"
export KPTOP_PROMETHEUS_POD_NAME="my-prometheus-server-0"
export KPTOP_PROMETHEUS_POD_PORT="9090"
export KPTOP_PROMETHEUS_POD_NAMESPACE="monitoring"

Or

export KPTOP_CONNECTION_METHOD="prometheus_endpoint"
export KPTOP_PROMETHEUS_SERVER="http://prometheus.home-lab.com"

Top nodes

kptop nodes

NODE      MEM TOTAL    MEM USAGE    MEM FREE      CPU CORES  CPU USAGE%      RUNNING PODS
worker-1  19.6 gb      16.92 gb     2.69 gb               6  9%                        14
worker-2  19.6 gb      9.52 gb      10.08 gb              6  9%                        27

kptop nodes --option / -o 🎉

kptop nodes -o json --colorize-json

Under testing (currently tested with EKS)

kptop nodes -o cloud

Live monitoring for Nodes

kptop node <NODE>

Top pods

kptop pods -n <NAMESPACE>

kptop pods -n elk-stack

NAMESPACE    POD                                      MEM LIMIT    MEM USAGE    MEM USAGE %    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
elk-stack    elasticsearch-master-0                   2.0 gb       1.38 gb      68%            2.0 gb           635.25 mb   1000m        0.04m
elk-stack    elasticsearch-master-1                   2.0 gb       1.49 gb      74%            2.0 gb           522.05 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-f8ms7          200.0 mb     85.73 mb     42%            174.16 mb        114.27 mb   1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-pq2wf  200.0 mb     87.73 mb     43%            171.31 mb        112.27 mb   1000m        0.03m
elk-stack    strimzi-filebeat-filebeat-r7dht          200.0 mb     119.12 mb    59%            199.52 mb        80.88 mb    1000m        0.02m
elk-stack    haproxy-ingress-filebeat-filebeat-lzqdt  200.0 mb     98.66 mb     49%            199.57 mb        101.34 mb   1000m        0.02m
elk-stack    my-kibana-kibana-79448f7fb7-wf4t6        2.0 gb       342.87 mb    16%            618.07 mb        1.67 gb     1000m        0.02m
elk-stack    my-logstash-logstash-0                   1.5 gb       1008.22 mb   65%            1.21 gb          527.78 mb   1000m        0.02m

kptop pod -n kube-system

NAMESPACE    POD                                              MEM LIMIT    MEM USAGE    MEM USAGE%    MEM USAGE MAX    MEM FREE    CPU LIMIT    CPU USAGE
kube-system  coredns-558bd4d5db-nfcjq                         170.0 mb     26.0 mb      15%           42.77 mb         144.0 mb    ---          0.0m
kube-system  coredns-558bd4d5db-vcstr                         170.0 mb     17.45 mb     10%           24.04 mb         152.55 mb   ---          0.0m
kube-system  etcd-master                                      ---          85.02 mb     ---           391.7 mb         ---         ---          0.02m
kube-system  kube-apiserver-master                            ---          635.97 mb    ---           731.13 mb        ---         ---          0.09m
kube-system  kube-controller-manager-master                   ---          95.55 mb     ---           145.41 mb        ---         ---          0.03m
kube-system  kube-proxy-q6nr7                                 ---          27.46 mb     ---           58.91 mb         ---         ---          0.0m
kube-system  kube-proxy-q489q                                 ---          21.98 mb     ---           63.0 mb          ---         ---          0.0m
kube-system  kube-proxy-bghp6                                 ---          22.35 mb     ---           64.1 mb          ---         ---          0.0m
kube-system  kube-scheduler-master                            ---          37.04 mb     ---           61.68 mb         ---         ---          0.0m
kube-system  nfs-subdir-external-provisioner-b97f4d9f5-bjp2h  ---          9.43 mb      ---           37.48 mb         ---         ---          0.0m

Live monitoring for Pods

kptop pod <POD> -n <NAMESPACE>

Live monitoring for Containers

kptop pod <POD> -n <NAMESPACE> -c <CONTAINER>

Top PVCs

kptop pvcs <NAMESPACE>

NOTE: in this example, all VPCs have the same capacity because this is a testing environment (using nfs-provisioner)

kptop pvcs --all-namespaces

NAMESPACE    PVC                                          VOLUME                CAPACITY    USED      USED %    FREE      FREE %
elk-stack    elasticsearch-master-elasticsearch-master-0  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-1  elasticsearch-master  123.14 gb   21.42 gb  17%       95.43 gb  77%
elk-stack    elasticsearch-master-elasticsearch-master-2  elasticsearch-master  ?           ?         ?         ?         ?
kafka        data-0-kafka-cluster-region1-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region1-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-0         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-1         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-0-kafka-cluster-region2-kafka-2         data-0                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-0         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-1         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-1-kafka-cluster-region1-kafka-2         data-1                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-0         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-1         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-2-kafka-cluster-region1-kafka-2         data-2                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-0         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-1         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-3-kafka-cluster-region1-kafka-2         data-3                123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region1-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
kafka        data-kafka-cluster-region2-zookeeper-0       data                  123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-alertmanager                   storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%
prometheus   my-prometheus-server                         storage-volume        123.14 gb   21.42 gb  17%       95.43 gb  77%

Verify Prometheus connectivity

kptop --verify-prometheus

Sample output

Verifying Prometheus connection: Connected                     
{
  "connected": true,
  "status_code": 200,
  "reason": "",
  "fail_reason": ""
}

Verifying Prometheus Exporters:

* Node Exporter:  Found             
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_versions": {
      "1.3.1": "2"
    }
  }
}

* Kubernetes Exporter:  Found           
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_git_versions": {
      "v1.21.0": "3",
      "v1.21.14": "1"
    }
  }
}

Check Prometheus metrics

kptop --verify-prometheus --check-metrics

Sample output

Verifying Prometheus connection: Connected                     
{
  "connected": true,
  "status_code": 200,
  "reason": "",
  "fail_reason": ""
}

Verifying Prometheus Exporters:

* Node Exporter:  Found             
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_versions": {
      "1.3.1": "2"
    }
  }
}

* Kubernetes Exporter:  Found           
{
  "success": true,
  "fail_reason": "",
  "result": {
    "found_git_versions": {
      "v1.21.0": "3",
      "v1.21.14": "1"
    }
  }
}
 
  0:00:00 0:00:00 Checking Metrics  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 100%  [ container_fs_writes_bytes_total ]
+---------------------------------------------------+---------------+-----------+-----------+
| METRIC                                            | EXPORTER      | STATE     | COMMENT   |
+===================================================+===============+===========+===========+
| node_memory_MemFree_bytes                         | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemAvailable_bytes                    | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemTotal_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Cached_bytes                          | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Buffers_bytes                         | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapTotal_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapFree_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapCached_bytes                      | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_cpu_seconds_total                            | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load1                                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load5                                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load15                                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_physical_cores                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_sockets                               | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| up                                                | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_boot_time_seconds                            | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_size_bytes                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes                       | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_receive_bytes_total                  | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_transmit_bytes_total                 | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_written_bytes_total                     | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_read_bytes_total                        | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_cores                                 | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_running_pods                              | node_exporter | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_last_seen                               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_working_set_bytes                | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_max_usage_bytes                  | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_limit_bytes                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_cache                            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_swap_limit_bytes            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_load_average_10s                    | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_usage_seconds_total                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_system_seconds_total                | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_user_seconds_total                  | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_cpu_quota                          | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kube_pod_spec_volumes_persistentvolumeclaims_info | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_capacity_bytes               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_used_bytes                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_available_bytes              | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_receive_bytes_total             | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_transmit_bytes_total            | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_start_time_seconds                      | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_file_descriptors                        | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_threads                                 | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_processes                               | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_reads_bytes_total                    | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total                   | kubernetes    | available |           |
+---------------------------------------------------+---------------+-----------+-----------+

Logging

Default log file location is "/tmp/kptop.log"

Known Issues

[1] Node Exporter metrics don't return data

This is NOT an issue, the node exporter NODE label change from version to another, currently we encountered only "kubernetes_node" or "node"
"node" is the default, to fix the it you can change it with the "KPTOP_NODE_EXPORTER_NODE_LABEL" Environment variables

export KPTOP_NODE_EXPORTER_NODE_LABEL="node" # default
export KPTOP_NODE_EXPORTER_NODE_LABEL="kubernetes_node"

auto detection of exporters verstions can be implemented later (if needed).

Reach me anytime on Linkedin

kptop
Release 0.0.6

Release 0.0.6

0.0.7

0.0.6

0.0.11

0.0.10

0.0.9

0.0.5

0.0.4

0.0.3

0.0.2

0.0.0

Documentation

Kube-Prometheus-Top [ kptop ]

📌 Try out the new beta features

Motivation

Project Status

Installation

Environment Variables

`prometheus_endpoint` ENVs

`pod_portForward` ENVs

General ENVs

CLI Arguments

Usage

Top nodes

Live monitoring for Nodes

Top pods

Live monitoring for Pods

Live monitoring for Containers

Top PVCs

Verify Prometheus connectivity

Check Prometheus metrics

Logging

Known Issues

Stats

Development practices

Releases

Contributors

kptop Release 0.0.6

Release 0.0.6 Toggle Dropdown 0.0.7 0.0.6 0.0.11 0.0.10 0.0.9 0.0.5 0.0.4 0.0.3 0.0.2 0.0.0

Documentation

Kube-Prometheus-Top [ kptop ]

📌 Try out the new beta features

Motivation

Project Status

Installation

Environment Variables

prometheus_endpoint ENVs

pod_portForward ENVs

General ENVs

CLI Arguments

Usage

Top nodes

Live monitoring for Nodes

Top pods

Live monitoring for Pods

Live monitoring for Containers

Top PVCs

Verify Prometheus connectivity

Check Prometheus metrics

Logging

Known Issues

Stats

Development practices

Releases

Contributors

kptop
Release 0.0.6

Release 0.0.6

0.0.7

0.0.6

0.0.11

0.0.10

0.0.9

0.0.5

0.0.4

0.0.3

0.0.2

0.0.0

`prometheus_endpoint` ENVs

`pod_portForward` ENVs