WEBVTT

00:01.420 --> 00:05.470
In this lab, we're going to practice

00:05.470 --> 00:07.900
troubleshooting worker node failures.

00:09.610 --> 00:12.930
We'll wait for
the lab environment to be ready

00:13.300 --> 00:15.420
and we'll get started.

00:17.220 --> 00:19.970
While we wait, I'm going to set

00:23.250 --> 00:25.140
the alias for kubectl.

00:27.790 --> 00:30.410
The first question
is to fix the broken cluster.

00:30.410 --> 00:32.980
Let's run a kubectl get nodes,

00:33.180 --> 00:36.400
and we see that indeed
there is something broken.

00:36.400 --> 00:39.000
Node01 is in a not ready state.

00:39.340 --> 00:40.680
Let's see why that is.

00:40.680 --> 00:44.050
Let's do a describe node01.

00:47.630 --> 00:53.770
Here we see a set of events.

00:53.770 --> 00:56.220
All of these seem to be normal

00:56.530 --> 00:58.330
so they're starting the kubelet.

00:59.210 --> 01:01.210
All of these are good statuses

01:01.620 --> 01:04.900
and the node itself is ready.

01:08.580 --> 01:10.360
There's something missing.

01:11.560 --> 01:15.460
Let's go to that node, ssh node01

01:15.620 --> 01:17.280
and we'll check the status.

01:18.340 --> 01:23.290
The first thing that we want to check
on a worker node is the kubelet,

01:23.290 --> 01:28.330
because as we know,
the kubelet is the controller of a node,

01:28.330 --> 01:31.660
and the kubelet
is what helps in communication

01:31.660 --> 01:35.330
between a worker node
and the master or the control plane node.

01:35.330 --> 01:38.460
Let's do the service of the kubelet.

01:38.600 --> 01:40.610
Service kublet status,

01:41.370 --> 01:44.850
and we see that it's loaded,
but it's in an inactive state.

01:45.370 --> 01:47.980
We don't know why it's an inactive state.

01:48.270 --> 01:52.200
We're going to just try and start it.

01:53.100 --> 01:57.100
Let's do a service kubelet start

01:58.140 --> 02:00.180
and let's check the status again.

02:00.960 --> 02:04.140
We see that it's now
in an active and running state.

02:04.350 --> 02:07.170
Let's go back to the control plane

02:07.440 --> 02:11.170
and let's do a kubectl get nodes.

02:11.870 --> 02:15.260
We see that both are in a ready state.

02:16.300 --> 02:18.450
I think that should fix that question.

02:20.210 --> 02:21.650
Let's go to the next one.

02:37.470 --> 02:40.960
The cluster is broken again,
investigate and fix the issue.

02:41.390 --> 02:43.630
Let's check the status again.

02:46.610 --> 02:48.980
We see that is again in a not ready state.

02:49.180 --> 02:52.240
Let's check the status of the node,

02:53.400 --> 02:54.770
node01

02:55.800 --> 02:58.810
and put a full screen.

02:59.040 --> 03:00.670
We see that

03:02.130 --> 03:06.340
all of these events are normal,
so that's okay.

03:07.060 --> 03:09.170
There's nothing here either.

03:11.770 --> 03:13.680
Let's go to the node

03:15.170 --> 03:18.640
and check the status of the kubelet again,

03:18.640 --> 03:20.730
service kubelet status,

03:21.210 --> 03:24.660
and we see that
it's in an activating state

03:25.120 --> 03:28.680
and it looks like there is an exit code

03:29.840 --> 03:31.900
and it's exited.

03:32.220 --> 03:34.110
The status is 255.

03:34.110 --> 03:37.330
That means there is something wrong
in the process,

03:37.330 --> 03:38.780
it's not able to start.

03:38.780 --> 03:43.050
Even if we do a service kubelet start,

03:45.440 --> 03:47.870
it's not actually
going to start the service

03:47.870 --> 03:50.300
because there seems to be something wrong.

03:50.300 --> 03:52.490
Let's check the status
of the service again,

03:52.580 --> 03:55.090
and it's in an activating state.

03:55.220 --> 03:57.760
To look at the logs of a service,

03:58.460 --> 04:04.650
do a journalctl -u kubelet

04:06.540 --> 04:09.710
and check the status.

04:10.560 --> 04:16.760
These are the logs
and let's look for the most recent logs.

04:18.980 --> 04:20.570
We are here.

04:23.440 --> 04:24.580
Let's go up.

04:26.560 --> 04:31.250
All of these seem to be an error
from a previous error.

04:31.500 --> 04:33.250
All of these seem to be an error message

04:33.250 --> 04:35.840
and here you can see
that there's an error that says

04:35.840 --> 04:41.760
unable to load client CA file
/etc/kubernetes/pki/WRONG-CA-FILE.

04:41.870 --> 04:43.470
There seems to be a wrong file.

04:43.470 --> 04:46.430
Let's try and find that out.

04:50.530 --> 04:52.750
The kubelet configuration files

04:52.750 --> 04:57.980
are one in the /etc/kubernetes/

05:03.210 --> 05:04.330
kubelet.conf.

05:04.330 --> 05:05.690
That's one file,

05:05.690 --> 05:09.080
but this is basically the kube config file

05:09.080 --> 05:10.560
used by the kubelet.

05:10.560 --> 05:12.880
Here you have the certificate,

05:13.040 --> 05:14.970
the context,
and everything that the kubelet uses

05:14.970 --> 05:17.320
to connect to the kube-apiserver.

05:18.920 --> 05:21.640
This is not the place
where there's an issue.

05:21.640 --> 05:23.810
It's basically the kubelet service itself.

05:24.380 --> 05:29.260
The kubelet service picks the options

05:29.480 --> 05:34.040
from a file located at var/lib/kubelet.

05:34.040 --> 05:35.680
This is something to remember.

05:35.800 --> 05:38.860
Here you have a file called config.yaml,

05:39.330 --> 05:42.130
so var/lib/kubelet/config.yaml.

05:42.220 --> 05:44.400
Within this file,
there are some properties

05:44.400 --> 05:47.200
that are passed through to the kubelet.

05:47.200 --> 05:51.170
Here you can see
there is this WRONG-CA-FILE right here,

05:51.170 --> 05:53.170
and that's the clientCAfile.

05:53.170 --> 05:55.220
It should be the correct file.

05:55.220 --> 05:57.970
Let's find what the correct file is.

05:59.680 --> 06:01.420
There's this ca.crt file.

06:01.420 --> 06:03.460
That's what it should be,

06:04.370 --> 06:07.630
ca.crt instead of the WRONG-CA-FILE.

06:07.630 --> 06:09.640
Let's edit that.

06:21.360 --> 06:23.320
Let me put the ca.crt.

06:24.090 --> 06:25.370
Save that.

06:26.920 --> 06:32.060
Now we're going to restart
the kubelet service.

06:33.950 --> 06:36.810
Let's just check the status
of the service now.

06:37.880 --> 06:40.720
We see that
it's in an active running state.

06:42.030 --> 06:43.470
Let's go back.

06:45.550 --> 06:47.630
Let's go back to the control plane node

06:49.540 --> 06:52.670
and let's check the status of the nodes.

06:52.890 --> 06:55.930
We see that they're in a ready state.

06:58.380 --> 07:00.420
Let's go to the next one.

07:13.100 --> 07:16.880
Now the cluster is broken again,
so let's check.

07:17.250 --> 07:22.030
We see that, yes, it is indeed broken
and node01 is not ready.

07:22.030 --> 07:24.660
Let's go to node01

07:25.210 --> 07:28.220
and let's check the status of the kubelet.

07:32.900 --> 07:35.300
We see that it's active
and it's in a running state.

07:35.740 --> 07:39.610
Let's check the logs,

07:41.080 --> 07:43.520
journalctl -u kubelet.

07:47.020 --> 07:48.670
Scroll all the way to the end

07:49.820 --> 07:52.490
and let's work our way backwards.

07:59.400 --> 08:01.450
Let's see
if there's something useful here.

08:01.570 --> 08:04.340
Here you can see, unable to register node.

08:05.720 --> 08:07.600
You can see the control plane

08:08.410 --> 08:14.080
and dial TCP 10:54:130:2:6553,
connection refused.

08:14.080 --> 08:17.790
This seems to be the error
that's causing the remaining errors.

08:17.950 --> 08:22.000
If you look at it,
it says connection refused to this.

08:22.210 --> 08:26.240
This is basically
for the control plane that we have.

08:27.860 --> 08:30.030
It looks like
this port number is incorrect

08:30.030 --> 08:34.930
because we know that the control
plane port number is 6443 and not 6553.

08:35.120 --> 08:36.430
That could be the error.

08:36.430 --> 08:41.290
Let's take a look at the kube config file
used by the kubelet.

08:41.290 --> 08:45.560
We saw that the cube config file
used by the kubelet

08:45.810 --> 08:52.170
is etc/kubernetes/kubelet.conf.

08:52.770 --> 08:55.330
Let's take a look
at the contents of the file.

08:55.550 --> 08:59.250
Here if you look at this,
you see that right here,

08:59.250 --> 09:02.030
the control plane
port number is incorrect.

09:03.680 --> 09:05.360
Let's get that corrected.

09:09.070 --> 09:11.500
Let's set it to 6443.

09:14.140 --> 09:18.690
We're going to do
a service kubelet restart

09:22.160 --> 09:25.570
and service kubelet status.

09:25.950 --> 09:29.080
You see that it's now in an active state.

09:29.490 --> 09:31.420
Let's check the logs as well.

09:33.710 --> 09:35.410
We no longer see those errors,

09:35.410 --> 09:39.760
and it seems to be good.

09:40.130 --> 09:41.390
Let's go back

09:43.390 --> 09:47.240
and let's check the status of the nodes.

09:47.490 --> 09:49.490
We see that it's now ready.

09:50.260 --> 09:51.580
Let's check our work.

09:52.320 --> 09:54.570
That's the end of this lab.

09:55.380 --> 09:58.020
When it comes
to troubleshooting worker nodes,

09:58.700 --> 10:02.940
always first check the status from here,
from the control plane

10:02.940 --> 10:05.060
and check whatever you can find from here,

10:05.060 --> 10:09.080
and then go to each
of the worker node that has the issue

10:09.080 --> 10:11.900
and start with
checking the kubelet service.

10:11.900 --> 10:15.480
If the kubelet service is good,
check the logs of the kubelet service

10:15.480 --> 10:18.220
and continue your way from there.

10:18.960 --> 10:20.460
That's all for this lab.

10:20.620 --> 10:21.640
Thank you.