A Python tool that provides Monitoring for Kubernetes Nodes, Pods, Containers, and PVCs resources on the terminal through Prometheus metircs
kptop2.mp4
The resources metrics provided by the K8s APIs are very limited compared what what's scraped by Prometheus.
This tool is using Prometheus as a data source for metrics to display all the needed informations right on the terminal.
Additional
- Check the connection with Prometheus Server
- Check the existence of the needed Prometheus metrics [ good for similar use cases ]
Compatible with Python 3.6+
pip3 install kptop --upgrade
ENV | Description | Default | Required |
---|---|---|---|
KPTOP_CONNECTION_METHOD |
The way to connect to Prometheus server options: ['prometheus_endpoint', 'pod_portForward'] |
Yes |
There are 2 options to connect KPtop to Prometheus:
- With a Prometheus server endpoint
- Suitable if Prometheus is exposed (with an ingress for example)
- With 'K8s pod port-forward' (Through K8s API-Server)
- Suitable if Prometheus is Not exposed (Only rechable within the K8s cluster)
ENV | Description | Default | Required |
---|---|---|---|
KPTOP_PROMETHEUS_SERVER |
Prometheus server URL | Yes | |
KPTOP_BASIC_AUTH_ENABLED |
Whether basic authentication is needed to connect to Prometheus | False | No |
KPTOP_PROMETHEUS_USERNAME |
Prometheus username | No | |
KPTOP_PROMETHEUS_PASSWORD |
Prometheus password | No | |
KPTOP_INSECURE |
Verify SSL certificate | False | No |
ENV | Description | Default | Required |
---|---|---|---|
KPTOP_PROMETHEUS_POD_NAME |
Prometheus pod name | Yes | |
KPTOP_PROMETHEUS_POD_PORT |
Prometheus port number | 9090 | No |
KPTOP_PROMETHEUS_POD_NAMESPACE |
The name space in which the Prometheus pod is deployed | default | No |
KUBECONFIG |
custom K8s kube config file | default path | No |
ENV | Description | Default | Required |
---|---|---|---|
KPTOP_NODE_EXPORTER_NODE_LABEL |
node exporter "node label" | "node" | NO |
KPTOP_START_GRAPHS_WITH_ZERO |
By default graphs begin with '0' to let the graph take its full hight | True | NO |
KPTOP_LOGGING_DIR |
Choose a different logging directory | /tmp/ | NO |
KPTOP_GRAPH_WIDTH |
Choose a custom graphs width | 45 | NO |
ENV | Description | Default |
---|---|---|
--namespace , -n
|
Specify a Kubernetes Namespace | default |
--all-namespaces , -A |
||
--container , -c
|
Specify a container | |
--interval , -i
|
Live monitoring update interval | 8 [NOTE: the actuall update depends on the Prometheus scaping interval (15s by default)] |
--debug , -d
|
Enable debugging logging mode | False |
--verify-prometheus , -V
|
Verify connectivity to Prometheus server & check the existence of the needed exporters | |
--sort-by-mem-usage , -s
|
Sort top result by memory usage | False |
--check-metrics , -C
|
Check the existence of the needed metrics, needs to be done with -V
|
False |
Add the ENVs and you're good to go.
Examples
export KPTOP_CONNECTION_METHOD="pod_portForward"
export KPTOP_PROMETHEUS_POD_NAME="my-prometheus-server-0"
export KPTOP_PROMETHEUS_POD_PORT="9090"
export KPTOP_PROMETHEUS_POD_NAMESPACE="monitoring"
Or
export KPTOP_CONNECTION_METHOD="prometheus_endpoint"
export KPTOP_PROMETHEUS_SERVER="http://prometheus.home-lab.com"
kptop nodes
NODE MEM TOTAL MEM USAGE MEM FREE CPU CORES CPU USAGE% RUNNING PODS
worker-1 19.6 gb 16.92 gb 2.69 gb 6 9% 14
worker-2 19.6 gb 9.52 gb 10.08 gb 6 9% 27
kptop nodes --option / -o π
kptop nodes -o json --colorize-json
Under testing (currently tested with EKS)
kptop nodes -o cloud
kptop node <NODE>
kptop pods -n <NAMESPACE>
kptop pods -n elk-stack
NAMESPACE POD MEM LIMIT MEM USAGE MEM USAGE % MEM USAGE MAX MEM FREE CPU LIMIT CPU USAGE
elk-stack elasticsearch-master-0 2.0 gb 1.38 gb 68% 2.0 gb 635.25 mb 1000m 0.04m
elk-stack elasticsearch-master-1 2.0 gb 1.49 gb 74% 2.0 gb 522.05 mb 1000m 0.03m
elk-stack strimzi-filebeat-filebeat-f8ms7 200.0 mb 85.73 mb 42% 174.16 mb 114.27 mb 1000m 0.02m
elk-stack haproxy-ingress-filebeat-filebeat-pq2wf 200.0 mb 87.73 mb 43% 171.31 mb 112.27 mb 1000m 0.03m
elk-stack strimzi-filebeat-filebeat-r7dht 200.0 mb 119.12 mb 59% 199.52 mb 80.88 mb 1000m 0.02m
elk-stack haproxy-ingress-filebeat-filebeat-lzqdt 200.0 mb 98.66 mb 49% 199.57 mb 101.34 mb 1000m 0.02m
elk-stack my-kibana-kibana-79448f7fb7-wf4t6 2.0 gb 342.87 mb 16% 618.07 mb 1.67 gb 1000m 0.02m
elk-stack my-logstash-logstash-0 1.5 gb 1008.22 mb 65% 1.21 gb 527.78 mb 1000m 0.02m
kptop pod -n kube-system
NAMESPACE POD MEM LIMIT MEM USAGE MEM USAGE% MEM USAGE MAX MEM FREE CPU LIMIT CPU USAGE
kube-system coredns-558bd4d5db-nfcjq 170.0 mb 26.0 mb 15% 42.77 mb 144.0 mb --- 0.0m
kube-system coredns-558bd4d5db-vcstr 170.0 mb 17.45 mb 10% 24.04 mb 152.55 mb --- 0.0m
kube-system etcd-master --- 85.02 mb --- 391.7 mb --- --- 0.02m
kube-system kube-apiserver-master --- 635.97 mb --- 731.13 mb --- --- 0.09m
kube-system kube-controller-manager-master --- 95.55 mb --- 145.41 mb --- --- 0.03m
kube-system kube-proxy-q6nr7 --- 27.46 mb --- 58.91 mb --- --- 0.0m
kube-system kube-proxy-q489q --- 21.98 mb --- 63.0 mb --- --- 0.0m
kube-system kube-proxy-bghp6 --- 22.35 mb --- 64.1 mb --- --- 0.0m
kube-system kube-scheduler-master --- 37.04 mb --- 61.68 mb --- --- 0.0m
kube-system nfs-subdir-external-provisioner-b97f4d9f5-bjp2h --- 9.43 mb --- 37.48 mb --- --- 0.0m
kptop pod <POD> -n <NAMESPACE>
kptop pod <POD> -n <NAMESPACE> -c <CONTAINER>
kptop pvcs <NAMESPACE>
NOTE: in this example, all VPCs have the same capacity because this is a testing environment (using nfs-provisioner)
kptop pvcs --all-namespaces
NAMESPACE PVC VOLUME CAPACITY USED USED % FREE FREE %
elk-stack elasticsearch-master-elasticsearch-master-0 elasticsearch-master 123.14 gb 21.42 gb 17% 95.43 gb 77%
elk-stack elasticsearch-master-elasticsearch-master-1 elasticsearch-master 123.14 gb 21.42 gb 17% 95.43 gb 77%
elk-stack elasticsearch-master-elasticsearch-master-2 elasticsearch-master ? ? ? ? ?
kafka data-0-kafka-cluster-region1-kafka-0 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region1-kafka-1 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region1-kafka-2 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region2-kafka-0 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region2-kafka-1 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-0-kafka-cluster-region2-kafka-2 data-0 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-1-kafka-cluster-region1-kafka-0 data-1 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-1-kafka-cluster-region1-kafka-1 data-1 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-1-kafka-cluster-region1-kafka-2 data-1 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-2-kafka-cluster-region1-kafka-0 data-2 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-2-kafka-cluster-region1-kafka-1 data-2 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-2-kafka-cluster-region1-kafka-2 data-2 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-3-kafka-cluster-region1-kafka-0 data-3 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-3-kafka-cluster-region1-kafka-1 data-3 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-3-kafka-cluster-region1-kafka-2 data-3 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-kafka-cluster-region1-zookeeper-0 data 123.14 gb 21.42 gb 17% 95.43 gb 77%
kafka data-kafka-cluster-region2-zookeeper-0 data 123.14 gb 21.42 gb 17% 95.43 gb 77%
prometheus my-prometheus-alertmanager storage-volume 123.14 gb 21.42 gb 17% 95.43 gb 77%
prometheus my-prometheus-server storage-volume 123.14 gb 21.42 gb 17% 95.43 gb 77%
kptop --verify-prometheus
Sample output
Verifying Prometheus connection: Connected
{
"connected": true,
"status_code": 200,
"reason": "",
"fail_reason": ""
}
Verifying Prometheus Exporters:
* Node Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_versions": {
"1.3.1": "2"
}
}
}
* Kubernetes Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_git_versions": {
"v1.21.0": "3",
"v1.21.14": "1"
}
}
}
kptop --verify-prometheus --check-metrics
Sample output
Verifying Prometheus connection: Connected
{
"connected": true,
"status_code": 200,
"reason": "",
"fail_reason": ""
}
Verifying Prometheus Exporters:
* Node Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_versions": {
"1.3.1": "2"
}
}
}
* Kubernetes Exporter: Found
{
"success": true,
"fail_reason": "",
"result": {
"found_git_versions": {
"v1.21.0": "3",
"v1.21.14": "1"
}
}
}
0:00:00 0:00:00 Checking Metrics ββββββββββββββββββββββββββββββ 100% [ container_fs_writes_bytes_total ]
+---------------------------------------------------+---------------+-----------+-----------+
| METRIC | EXPORTER | STATE | COMMENT |
+===================================================+===============+===========+===========+
| node_memory_MemFree_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemAvailable_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_MemTotal_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Cached_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_Buffers_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapTotal_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapFree_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_memory_SwapCached_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_cpu_seconds_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load1 | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load5 | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_load15 | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_physical_cores | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_sockets | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| up | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_boot_time_seconds | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_size_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_filesystem_avail_bytes | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_receive_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_network_transmit_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_written_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| node_disk_read_bytes_total | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| machine_cpu_cores | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_running_pods | node_exporter | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_last_seen | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_working_set_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_max_usage_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_limit_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_memory_cache | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_memory_swap_limit_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_load_average_10s | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_usage_seconds_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_system_seconds_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_cpu_user_seconds_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_spec_cpu_quota | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kube_pod_spec_volumes_persistentvolumeclaims_info | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_capacity_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_used_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| kubelet_volume_stats_available_bytes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_receive_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_network_transmit_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_start_time_seconds | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_file_descriptors | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_threads | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_processes | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_reads_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
| container_fs_writes_bytes_total | kubernetes | available | |
+---------------------------------------------------+---------------+-----------+-----------+
Default log file location is "/tmp/kptop.log
"
- This is NOT an issue, the node exporter NODE label change from version to another, currently we encountered only "kubernetes_node" or "node"
- "node" is the default, to fix the it you can change it with the "KPTOP_NODE_EXPORTER_NODE_LABEL" Environment variables
export KPTOP_NODE_EXPORTER_NODE_LABEL="node" # default
export KPTOP_NODE_EXPORTER_NODE_LABEL="kubernetes_node"
auto detection of exporters verstions can be implemented later (if needed).
Reach me anytime on Linkedin