node-problem-detector aims to make various node problems visible to the upstream layers in cluster management stack. It is a DaemonSet detecting node problems and reporting them to apiserver. Now it is running as a Kubernetes Addon enabled by default in the GCE cluster.
There are tons of node problems could possibly affect the pods running on the node such as:
- Hardware issues: Bad cpu, memory or disk;
- Kernel issues: Kernel deadlock, corrupted file system;
- Container runtime issues: Unresponsive runtime daemon;
Currently these problems are invisible to the upstream layers in cluster management stack, so Kubernetes will continue scheduling pods to the bad nodes.
To solve this problem, we introduced this new daemon node-problem-detector to collect node problems from various daemons and make them visible to the upstream layers. Once upstream layers have the visibility to those problems, we can discuss the remedy system.
NodeCondition to report problems to
NodeCondition: Permanent problem that makes the node unavailable for pods should be reported as
Event: Temporary problem that has limited impact on pod but is informative should be reported as
A problem daemon is a sub-daemon of node-problem-detector. It monitors a specific kind of node problems and reports them to node-problem-detector.
A problem daemon could be:
- A tiny daemon designed for dedicated usecase of Kubernetes.
- An existing node health monitoring daemon integrated with node-problem-detector.
Currently, a problem daemon is running as a goroutine in the node-problem-detector binary. In the future, we'll separate node-problem-detector and problem daemons into different containers, and compose them with pod specification.
List of supported problem daemons:
|KernelMonitor||KernelDeadlock||A problem daemon monitors kernel log and reports problem according to predefined rules.|
make in the top directory. It will:
- Build the binary.
- Build the docker image. The binary and
config/are copied into the docker image.
- Upload the docker image to registry. By default, the image will be uploaded to
gcr.io/google_containers. It's easy to modify the
Makefileto push the image to another registry
- Create a file node-problem-daemon.yaml with the following yaml.
apiVersion: extensions/v1beta1 kind: DaemonSet metadata: name: node-problem-detector spec: template: spec: hostNetwork: true containers: - name: node-problem-detector image: gcr.io/google_containers/node-problem-detector:v0.1 imagePullPolicy: Always securityContext: privileged: true volumeMounts: - name: log mountPath: /log readOnly: true volumes: - name: log # Config `log` to your system log directory hostPath: path: /var/log/
- Edit node-problem-detector.yaml to fit your environment: Set
logvolueme to your system log diretory. (Used by KernelMonitor)
- Create the DaemonSet with
kubectl create -f node-problem-detector.yaml
- If needed, you can use ConfigMap
to overwrite the
node-problem-detector has several flags:
--kernel-monitor: The path to the kernel monitor config file. Default:
--hostname-override: If non-empty, node problem detector will use the specified hostname as node identification. Default: ""
--insecure-connection: If true, node problem detector will skip TLS verification while connecting with apiserver. Default: false.