WEBVTT

00:00.300 --> 00:02.187
Okay, so in this video,

00:02.267 --> 00:05.867
we're going to go through the lab
on cluster upgrade process.

00:06.113 --> 00:08.763
This lab tests your skills
on upgrading a Kubernetes cluster,

00:08.843 --> 00:12.333
and we have a production cluster
with applications running on it.

00:12.413 --> 00:13.923
Let's explore the setup first.

00:14.003 --> 00:17.933
The first question is, what is the current
version of the cluster?

00:18.503 --> 00:20.753
We're going to do kubectl get nodes.

00:21.953 --> 00:28.583
It's not set up, so let me do that,
kubectl get nodes.

00:29.663 --> 00:31.853
We see that there are two nodes.

00:33.593 --> 00:39.053
The version that we're currently
at is 1.19.0, so 1.19.0.

00:42.713 --> 00:44.763
How many nodes are part of this cluster?

00:44.843 --> 00:48.413
There are two nodes,
including the master and worker nodes.

00:51.983 --> 00:55.163
How many nodes can host
workloads in this cluster?

00:55.613 --> 00:58.413
Inspect the applications
and taints set on the nodes.

00:58.493 --> 01:02.633
Let's do a kubectl describe node command

01:03.133 --> 01:05.843
and look for taints.

01:06.353 --> 01:08.163
Let's see if there are any taints
set up on them.

01:08.243 --> 01:09.453
There are no taints.

01:09.533 --> 01:12.953
That means both the nodes
can host applications.

01:16.673 --> 01:17.673
That's two.

01:18.053 --> 01:21.713
We can have other nodes running,
hosting applications.

01:22.103 --> 01:24.273
How many applications
are hosted on the cluster?

01:24.353 --> 01:26.963
Let's do a get deploy.

01:27.623 --> 01:31.943
We see that there's just one application
and that's the blue application.

01:34.373 --> 01:36.053
Let's go to the next one.

01:38.543 --> 01:41.103
Now, what nodes are the pods hosted on?

01:41.183 --> 01:45.173
Let's do a get pods with the wide option.

01:45.593 --> 01:49.913
We see that the pods are hosted
on control plane as well as node01.

01:51.173 --> 01:54.083
Both control plane and node01.

01:58.403 --> 02:01.043
You are tasked to upgrade the cluster.

02:01.403 --> 02:04.643
Users accessing the applications
must not be impacted

02:05.003 --> 02:07.103
and you cannot provision new VMs.

02:07.403 --> 02:09.003
Currently, there are two VMs,

02:09.083 --> 02:10.896
the control plane and node01.

02:11.303 --> 02:14.883
You cannot provision new VMs
and the users accessing

02:14.963 --> 02:17.843
the applications must not be impacted,

02:18.053 --> 02:20.433
so we cannot fully take
the applications down.

02:20.513 --> 02:23.193
What strategy would you use
to upgrade the cluster?

02:23.273 --> 02:27.363
Upgrade one node at a time
while moving the workloads to the other,

02:27.443 --> 02:29.631
so that looks like the best option.

02:29.753 --> 02:31.113
We'll also see why the others are not.

02:31.193 --> 02:34.613
Users will be impacted
since there's only one worker node.

02:35.543 --> 02:38.193
That's incorrect
because we have seen that the workload

02:38.273 --> 02:40.593
can be scheduled on both the control
plane and the worker node,

02:40.673 --> 02:43.613
even though there's only one worker node.

02:44.333 --> 02:48.543
In this case, even if we take
down the worker node, the deployments,

02:48.623 --> 02:51.423
the applications
are going to be recreated on the control

02:51.503 --> 02:55.563
plane node so users
will not be impacted just in this case,

02:55.643 --> 02:57.753
because we've seen
that that's the way it is set up,

02:57.833 --> 02:59.513
this particular cluster.

02:59.663 --> 03:01.743
Upgrade all nodes at once.

03:01.823 --> 03:07.013
If we do that, of course the applications
will be down for the time,

03:07.253 --> 03:10.973
during which the nodes are being upgraded.

03:11.633 --> 03:14.703
The users are going to be impacted
so that's not the correct option.

03:14.783 --> 03:19.673
Add new nodes with newer versions
while taking down existing nodes.

03:20.153 --> 03:22.023
That's incorrect because it is said here

03:22.103 --> 03:24.233
that you cannot provision new VMs.

03:24.773 --> 03:26.373
We cannot add new nodes.

03:26.453 --> 03:29.253
The option here is to upgrade
one node at a time

03:29.333 --> 03:31.833
while moving the workloads to the other.

03:36.293 --> 03:39.393
What is the latest stable version available

03:39.473 --> 03:41.703
for upgrade using the kubeadm tool?

03:41.783 --> 03:44.973
Let's do the kubeadm tool

03:45.053 --> 03:47.953
and we know
that we have the upgrade plan option.

03:49.193 --> 03:53.013
This question is not to find out
the latest stable version of Kubernetes

03:53.093 --> 03:54.633
but what the latest stable version

03:54.713 --> 03:58.553
that this particular kubeadm
tool knows as of now.

04:01.913 --> 04:04.053
Running the upgrade plan command let's -.

04:04.133 --> 04:04.953
The options that we have.

04:05.033 --> 04:09.963
Currently, the kubeadm
tool itself is on version 1.19.

04:10.043 --> 04:12.573
As you can see here,
the kubeadm version is 1.19.

04:12.653 --> 04:14.653
This particular kubeadm version,

04:15.023 --> 04:21.503
this tool can upgrade
this cluster to 1.19.16.

04:21.953 --> 04:27.093
If we need to upgrade to a higher version,
if you look at this,

04:27.173 --> 04:33.113
the remote version
as of this recording is 1.23.5,

04:34.253 --> 04:39.653
so if you want to upgrade to any version
other than 1.19, so like 1.20, 1.21

04:39.893 --> 04:43.103
then we will need to upgrade
the kubeadm tool itself.

04:44.873 --> 04:50.453
To answer this question,
the available stable version

04:50.533 --> 04:54.173
as of this recording is 1.19.16.

05:01.073 --> 05:03.603
Now we will be upgrading
the master node first.

05:03.683 --> 05:06.887
Let's drain the master node of workloads

05:06.967 --> 05:09.867
and mark it unschedulable, so let's do that.

05:10.333 --> 05:12.803
Let's do a kubectl drain

05:13.223 --> 05:19.883
and controlplane with
the ignore daemon set option.

05:30.083 --> 05:34.763
The auto-complete didn't configure
for this lab so I just have to--

05:41.423 --> 05:45.053
We're just waiting
for the pods to be evicted.

05:47.813 --> 05:49.793
Let's check the status.

05:50.243 --> 05:53.283
We see that controlplane
shielding is disabled

05:53.363 --> 05:55.253
and if you look at the pods,

05:56.753 --> 06:01.073
just to be sure we see
that they're all on node01 now.

06:04.726 --> 06:07.286
Now next,
we upgrade the controlplane components

06:07.366 --> 06:11.720
to the exact version 1.20.0.

06:11.800 --> 06:14.050
That's what we need to upgrade into.

06:14.536 --> 06:16.753
For that,
we have to upgrade the kubeadm tool first

06:16.833 --> 06:20.093
and then the master components.

06:20.363 --> 06:22.323
First, we have to upgrade the kubeadm tool

06:22.403 --> 06:25.263
and then the master control components,
and then the kubelet.

06:25.343 --> 06:28.463
That's kind of the high-level steps.

06:29.362 --> 06:33.202
We will refer to the Kubernetes
documentation pages for this.

06:34.567 --> 06:40.523
Let's open up the Kubernetes
documentation page and it's for upgrade,

06:41.423 --> 06:43.683
and here we have upgrading kubeadm clusters.

06:43.763 --> 06:45.443
That's what we want to do.

06:45.713 --> 06:49.023
The goal here is that we know
that we're currently on 1.19

06:49.103 --> 06:51.443
and we want to upgrade it to 1.20.

06:51.983 --> 06:53.920
As of this recording,

06:54.000 --> 06:58.673
this lab is asking
to upgrade from 1.19 to 1.20,

06:59.333 --> 07:02.163
but depending on when you're working
on this lab,

07:02.243 --> 07:04.953
depending on what the question is here,

07:05.033 --> 07:07.613
you must select the right one from here.

07:08.033 --> 07:11.873
I'm going to select this one right here,
which is 1.19 to 20,

07:12.893 --> 07:16.193
and here are the steps.

07:17.993 --> 07:18.723
When going through this,

07:18.803 --> 07:22.643
remember that first we determine
which version to upgrade to,

07:22.913 --> 07:25.967
then here we have upgrading
control plane nodes,

07:26.267 --> 07:29.933
and then below we have upgrading
worker nodes.

07:30.953 --> 07:33.443
Remember to follow the right one.

07:34.403 --> 07:37.613
The first step,
and also there are two versions.

07:38.003 --> 07:40.387
This is for Ubuntu and Debian
and this is for CentOS

07:40.467 --> 07:43.473
and we're not sure which one you're on.

07:43.553 --> 07:46.163
You could do a cat /etc/-release,

07:48.773 --> 07:52.673
and this will tell you the current
operating system version.

07:52.913 --> 07:54.601
You're on Ubuntu right now.

07:55.523 --> 07:57.773
That's what we are going to go with.

08:01.767 --> 08:04.653
Let's run this first to identify the exact

08:04.733 --> 08:06.921
version that we want to upgrade to,

08:14.633 --> 08:20.693
but we already have it here but let's just
do this to find out for sure.

08:23.513 --> 08:26.783
The version that we want
to upgrade to is 1.20.0,

08:27.113 --> 08:28.801
but then there's also this.

08:30.083 --> 08:33.467
The last digit,

08:34.033 --> 08:37.763
this is what we're going to upgrade to,
1.20.0-00.

08:41.273 --> 08:44.163
Now, the first step is to upgrade
the control plane nodes.

08:44.243 --> 08:46.773
That's what we're going to do,
and this is the command.

08:46.853 --> 08:51.203
The first step is to upgrade kubeadm
itself, so the kubeadm tool itself.

08:51.683 --> 08:52.803
There are two ways to do it.

08:52.883 --> 08:54.663
You can either run
these three commands or you can just run

08:54.743 --> 08:57.413
these two commands that are together,

08:57.713 --> 09:01.523
but for this the apt-get version
should be greater than 1.1.

09:02.723 --> 09:05.933
You can actually verify that

09:07.267 --> 09:11.723
by running apt-get --version.

09:12.893 --> 09:16.073
We're on 1.6.
That means we can just run this command.

09:16.403 --> 09:23.223
We're going to copy
this and that should be 00.

09:23.303 --> 09:29.393
We're going to change this to 0,
that's what we want to upgrade to.

09:29.693 --> 09:30.818
Let's run that.

09:39.893 --> 09:41.573
Kubeadm is now upgraded.

09:43.493 --> 09:49.253
Next,
let's verify using the kubeadm command,

09:49.343 --> 09:53.633
and we see that it's now at 1.20.0.

09:54.533 --> 10:00.473
Next, we are on the upgrade plan
again and we can get the new plan.

10:10.823 --> 10:17.283
Here we see the details.

10:17.363 --> 10:21.093
You have the current version
of the API server and everything here

10:21.173 --> 10:24.353
and you have the available version here.

10:25.343 --> 10:29.033
What we want to do is to upgrade
this to this specifically.

10:29.663 --> 10:31.538
That's what we're going to do.

10:32.993 --> 10:37.523
We're going to do the kubeadm
upgrade apply command

10:38.663 --> 10:42.803
and specify the version as zero.

10:48.653 --> 10:50.943
Are you sure you want
to proceed with the upgrade?

10:51.023 --> 10:52.023
Yes.

11:07.613 --> 11:08.733
Okay, that's done.

11:08.813 --> 11:14.243
It says, "Your cluster
was upgraded to version 1.20.0."

11:16.823 --> 11:18.903
The next step is to upgrade
your CNI providers.

11:18.983 --> 11:20.013
We're going to skip that.

11:20.093 --> 11:22.218
For now, we don't want to do that.

11:22.733 --> 11:23.823
For other control panels,

11:23.903 --> 11:27.143
we do not have any other control
plane nodes as of now.

11:28.943 --> 11:31.973
Finally, the node is already drained.

11:33.443 --> 11:35.403
We don't have to do that either.

11:35.483 --> 11:39.303
The final step is to upgrade
kubelet and kubectl utility, right?

11:39.383 --> 11:43.313
Here again,
we're going to follow this second command

11:46.883 --> 11:53.213
and we'll change it to version 00.

11:59.753 --> 12:00.753
Okay.

12:03.803 --> 12:07.583
Then the final step is to restart kubelet.

12:08.393 --> 12:09.563
Let's do that.

12:12.053 --> 12:13.053
Okay.

12:14.633 --> 12:15.873
Then it's to uncordon the node.

12:15.953 --> 12:18.893
Let's just check status now.

12:20.123 --> 12:21.753
We see that here on get nodes,

12:21.833 --> 12:25.067
you see that the control
plane is now at 1.20.

12:25.667 --> 12:27.242
Let's check the work.

12:30.083 --> 12:31.083
Okay.

12:34.133 --> 12:38.213
Now we've got to mark the control
plane node as schedulable again.

12:38.723 --> 12:40.383
That's basically this step.

12:40.463 --> 12:41.888
We can do that now.

12:42.569 --> 12:47.933
That's uncordon controlplane.

12:51.443 --> 12:53.273
Let's verify it's done.

12:53.543 --> 12:54.543
Yes.

12:56.663 --> 12:58.263
Okay, next is the worker node.

12:58.343 --> 13:02.183
Drain the worker node of the workloads
and mark it unschedulable.

13:03.683 --> 13:10.433
We see that it's in a ready state,
and we're going to drain node 01.

13:12.173 --> 13:16.103
Then we pass the ignore-daemonsets option.

13:18.353 --> 13:22.463
We'll just wait for the pods to be evicted.

13:25.973 --> 13:30.863
Okay,
now let's check the status of the node.

13:32.063 --> 13:35.363
Also, let's check the status of pod.

13:35.903 --> 13:38.603
Now you can see that node01 is uncordoned

13:39.053 --> 13:41.866
and then all the pods
are now on controlplane.

13:41.963 --> 13:44.493
The workload has now moved to controlplane.

13:44.573 --> 13:47.813
Now we can safely upgrade node01.

13:51.953 --> 13:57.200
Now we're going to upgrade the worker
node to the exact version, 1.20.0.

13:57.800 --> 13:59.573
Let's go here,

14:00.173 --> 14:03.183
the upgrade procedure on worker nodes
should be executed one node at a time

14:03.263 --> 14:04.353
or few nodes at a time

14:04.433 --> 14:06.933
without compromising
the minimum required capacity.

14:07.013 --> 14:08.523
We have taken care of that.

14:08.603 --> 14:11.033
The first step is to upgrade kubeadm.

14:11.783 --> 14:15.203
This needs to be on the worker node.

14:15.413 --> 14:16.463
Remember that.

14:17.603 --> 14:19.893
You could ssh to the worker
node by just typing

14:19.973 --> 14:23.513
the name of the node
if that's set up properly.

14:23.783 --> 14:28.203
Or another way
to do that is to find the IP address

14:28.283 --> 14:31.343
of the nodes by running
a get nodes wide command

14:31.763 --> 14:35.573
and then ssh to this internal IP of node01.

14:39.473 --> 14:41.363
Yes, okay.

14:42.203 --> 14:43.703
We're now on node01.

14:45.443 --> 14:49.313
The first step is to upgrade
the kubeadm tool.

14:50.063 --> 14:55.283
Let's copy this and set it to 0.

14:58.913 --> 15:01.103
Okay, that's complete.

15:02.303 --> 15:06.173
The next step is to call
the kubeadm upgrade node command.

15:07.313 --> 15:08.193
What this is going to do

15:08.273 --> 15:11.393
is it's going to upgrade
the kubelet configuration.

15:12.173 --> 15:13.923
This should be pretty quick.

15:15.900 --> 15:16.900
That's done.

15:18.173 --> 15:20.313
The next step is to drain the node.

15:20.393 --> 15:23.753
The node is already drained,
so we don't have to do that.

15:24.143 --> 15:27.363
The next step is to upgrade
the kubelet and kubectl utility.

15:27.443 --> 15:32.233
Lets copy these and

15:36.033 --> 15:38.879
lets change this to 0.

15:46.073 --> 15:48.636
Then you have the restart kubelet option.

15:53.723 --> 15:54.923
Restart kubelet.

15:55.973 --> 15:57.323
Finally, uncordon.

15:58.767 --> 15:59.817
We'll do that.

16:00.000 --> 16:02.287
Lets first check the status.

16:02.367 --> 16:04.053
Lets go back to the controlplane node

16:04.133 --> 16:05.883
and lets check the status of the node.

16:05.963 --> 16:08.003
We see that it is now on 1.20.

16:10.493 --> 16:11.920
Lets check the status.

16:12.000 --> 16:13.350
That's successful.

16:13.463 --> 16:14.823
Now we have to remove the restriction

16:14.903 --> 16:16.953
and mark the worker
node as schedulable again.

16:17.033 --> 16:18.273
That's the uncordon.

16:18.353 --> 16:21.653
We're going to uncordon node01.

16:24.100 --> 16:31.013
Lets check the nodes and we see
that both of them are in a ready stage.

16:34.703 --> 16:38.993
We have successfully moved
the workload between the nodes

16:39.203 --> 16:42.323
and managed to upgrade
both the nodes in the cluster.

16:43.103 --> 16:45.166
Well, that's the end of this lab.