WEBVTT

00:02.230 --> 00:03.230
In this lecture,

00:03.230 --> 00:04.690
we will see different ways of

00:04.700 --> 00:06.900
troubleshooting worker node failures.

00:07.430 --> 00:08.190
Again,

00:08.200 --> 00:09.760
we start by checking the status

00:09.760 --> 00:11.560
of the nodes in the cluster,

00:11.560 --> 00:14.530
are they reported as ready or not ready?

00:14.530 --> 00:16.730
If they are reported as not ready,

00:16.730 --> 00:18.360
check details about the nodes

00:18.360 --> 00:21.330
using the kubectl describe node command.

00:22.060 --> 00:24.060
Each node has a set of conditions

00:24.060 --> 00:25.730
that can point us in a direction

00:25.730 --> 00:28.090
as to why a node might have failed.

00:28.100 --> 00:29.430
Depending on the status,

00:29.430 --> 00:33.090
they're either set to true or false or unknown.

00:33.660 --> 00:35.690
When the node is out of disk space,

00:35.700 --> 00:38.300
the OutOfDisk flag is set to true.

00:38.300 --> 00:39.930
When a node is out of memory,

00:39.930 --> 00:42.490
the MemoryPressure flag is set to true.

00:42.500 --> 00:44.600
When the disk capacity is low,

00:44.600 --> 00:47.430
the DiskPressure flag is set to true.

00:47.430 --> 00:50.130
Similarly, when there are too many processes,

00:50.130 --> 00:52.760
the PIDPressure flag is set to true.

00:52.760 --> 00:55.490
Finally, if the node as a whole is healthy,

00:55.500 --> 00:57.430
the Ready flag is set to true.

00:57.430 --> 01:00.760
When a worker node stops
communicating with the master,

01:00.760 --> 01:02.690
maybe due to a crash,

01:02.700 --> 01:05.130
these statuses are set to unknown.

01:05.130 --> 01:08.430
This can indicate a possible loss of a node.

01:08.430 --> 01:10.560
Check the LastHeartbeatTime field

01:10.560 --> 01:11.730
to find out

01:11.730 --> 01:14.090
the time when the node might have crashed.

01:14.960 --> 01:16.160
In such cases,

01:16.160 --> 01:18.960
proceed to checking the status
of the node itself.

01:18.960 --> 01:21.890
If the node is online at all or is crashed.

01:21.900 --> 01:24.430
If it's crashed, bring it back up.

01:24.430 --> 01:28.430
Check for possible CPU memory
and disk space on the nodes.

01:29.430 --> 01:31.160
Check the status of the kubelet.

01:33.560 --> 01:36.390
Check the kubelet logs for possible issues.

01:39.330 --> 01:41.460
Check the kubelet certificates,

01:41.460 --> 01:42.960
ensure they're not expired

01:42.960 --> 01:45.360
and they are part of the right group

01:45.360 --> 01:47.090
and that the certificates are issued

01:47.100 --> 01:48.630
by the right CA.

01:48.630 --> 01:50.560
That's it for this lecture,

01:50.560 --> 01:51.990
head over to the practice test,

01:52.000 --> 01:55.200
and practice fixing broken clusters.