WEBVTT

00:00.600 --> 00:02.400
-Hello, and welcome to this lecture.

00:02.670 --> 00:08.250
In this lecture, we discuss about
Cluster Upgrade Process in Kubernetes.

00:09.390 --> 00:14.010
In the previous lecture, we saw how
Kubernetes manages its software releases

00:14.190 --> 00:16.710
and how different components
have their versions.

00:17.460 --> 00:19.930
We will keep dependency
on external components

00:20.100 --> 00:23.100
like ETCD and CoreDNS aside for now,

00:23.520 --> 00:27.060
and focus on the core
control plane components.

00:28.350 --> 00:32.010
Is it mandatory for all of these
to have the same version?

00:32.970 --> 00:36.300
No. The components can be
at different release versions.

00:36.720 --> 00:40.510
Since the kube-apiserver is
the primary component in the control

00:40.680 --> 00:44.460
plane and that is the component
that all other components talk to,

00:44.790 --> 00:47.440
none of the other
components should ever be

00:47.610 --> 00:50.700
at a version higher
than the kube-apiserver.

00:51.960 --> 00:55.920
The controller-manager and scheduler
can be at one version lower.

00:56.250 --> 00:58.770
If kube-apiserver was at X,

00:59.190 --> 01:04.810
controller-managers and kube-schedulers
can be at X-1 and the kubelet

01:04.980 --> 01:09.930
and kube-proxy components can
be at two versions lower, X-2.

01:10.860 --> 01:13.950
If kube-apiserver was at 1.10,

01:14.280 --> 01:18.130
the controller-manager
and scheduler could be at 1.10

01:18.300 --> 01:23.580
or 1.9 and the kubelet
and kube-proxy could be at 1.8.

01:24.810 --> 01:30.030
None of them could be at a version higher
than the kube-apiserver, like 1.11.

01:30.990 --> 01:33.390
Now, this is not the case with kubectl.

01:33.990 --> 01:39.750
The kubectl utility could be at 1.11,
a version higher than the apiserver,

01:40.140 --> 01:43.260
1.10,
the same version as the apiserver,

01:43.560 --> 01:47.790
or at 1.9 a version
lower than the apiserver.

01:48.390 --> 01:54.420
Now, this permissible skew in versions
allows us to carry out live upgrades.

01:54.930 --> 01:58.140
We can upgrade component
by component if required.

01:58.800 --> 02:00.150
When should you upgrade?

02:01.680 --> 02:08.220
Say you were at 1.10 and Kubernetes
releases versions 1.11 and 1.12.

02:08.850 --> 02:15.210
At any time, Kubernetes supports only
up to the recent three minor versions.

02:15.780 --> 02:17.960
With 1.12 being the latest release,

02:18.130 --> 02:23.380
Kubernetes supports versions 1.12,
1.11, and 1.10.

02:23.880 --> 02:30.310
When 1.13 is released
only versions 1.13, 1.12,

02:30.490 --> 02:32.530
and 1.11 are supported.

02:33.390 --> 02:36.790
Before the release of 1.13
would be a good time

02:36.960 --> 02:41.580
to upgrade your cluster to the next
release. How do we upgrade?

02:41.880 --> 02:46.360
Do we upgrade directly
from 1.10 to 1.13? No.

02:46.530 --> 02:50.850
The recommended approach is to upgrade
one minor version at a time;

02:51.120 --> 02:57.930
version 1.10 to 1.11, then 1.11 to 1.12,
and then 1.12 to 1.13.

02:58.950 --> 03:02.340
The upgrade process depends
on how your cluster is set up.

03:02.760 --> 03:06.400
For example, if your cluster
is a managed Kubernetes cluster

03:06.570 --> 03:10.080
deployed on cloud service providers
like Google, for instance,

03:10.440 --> 03:12.760
Google Kubernetes
Engine lets you upgrade

03:12.930 --> 03:15.510
your cluster easily
with just a few clicks.

03:16.200 --> 03:19.350
If you deployed your cluster
using tools like kubeadm,

03:19.680 --> 03:22.470
then the tool can help you plan
and upgrade the cluster.

03:23.130 --> 03:25.260
If you deployed
your cluster from scratch,

03:25.500 --> 03:29.940
then you manually upgrade the different
components of the cluster yourself.

03:30.300 --> 03:34.230
In this lecture,
we will look at the options by kubeadm.

03:35.130 --> 03:39.070
You have a cluster with a master
and worker nodes running in production,

03:39.240 --> 03:41.490
hosting pods, serving users.

03:42.000 --> 03:44.790
The nodes and components
are at version 1.10.

03:45.450 --> 03:48.660
Upgrading a cluster
involves two major steps.

03:48.930 --> 03:53.490
First, you upgrade your master nodes
and then upgrade the worker nodes.

03:54.090 --> 03:56.010
While the master is being upgraded,

03:56.310 --> 04:00.990
the control plane components
such as the apiserver, scheduler,

04:01.350 --> 04:04.110
and controller-managers
go down briefly.

04:04.710 --> 04:07.300
The master going down
does not mean your worker

04:07.470 --> 04:10.080
nodes and applications
on the cluster are impacted.

04:10.680 --> 04:15.210
All workloads hosted on the worker
nodes continue to serve users as normal.

04:15.570 --> 04:19.410
Since the master is done,
all management functions are down.

04:19.710 --> 04:23.790
You cannot access the cluster using
kubectl or other Kubernetes API.

04:24.210 --> 04:28.240
You cannot deploy new applications
or delete or modify existing ones.

04:28.410 --> 04:30.930
The controller-managers
don’t function either.

04:31.260 --> 04:35.340
If a pod was to fail and a new pod
won’t be automatically created.

04:35.910 --> 04:38.580
As long as the nodes
and the pods are up,

04:39.000 --> 04:43.560
your applications should be up
and users will not be impacted.

04:44.040 --> 04:47.160
Once the upgrade is complete
and the cluster is back up,

04:47.610 --> 04:49.260
it should function normally.

04:49.620 --> 04:52.060
We now have the master
and the master of components

04:52.230 --> 04:56.610
at version 1.11 and the worker
nodes add version 1.10.

04:57.360 --> 05:00.840
As we saw earlier,
this is a supported configuration.

05:01.080 --> 05:04.140
It is now time to upgrade
the worker nodes.

05:04.590 --> 05:07.990
There are different strategies
available to upgrade the worker nodes.

05:08.280 --> 05:11.400
One is to upgrade all of them at once,

05:11.670 --> 05:14.350
but then your pods
are down and users are

05:14.520 --> 05:17.220
no longer able to access
the applications.

05:18.090 --> 05:22.720
Once the upgrade is complete, the nodes
are back up, new pods are scheduled,

05:22.890 --> 05:24.660
and users can resume access.

05:25.230 --> 05:27.660
That’s one strategy
that requires downtime.

05:28.350 --> 05:31.860
The second strategy is
to upgrade one node at a time.

05:32.220 --> 05:35.100
Going back to the state
where we have our master

05:35.370 --> 05:38.400
upgraded and nodes
waiting to be upgraded,

05:39.030 --> 05:42.550
we first upgrade the first
node where the workloads move

05:42.720 --> 05:46.140
to the second and third node
and users are also from there.

05:46.800 --> 05:49.260
Once the first node is
upgraded and back up,

05:49.770 --> 05:51.490
we then update the second node where

05:51.660 --> 05:54.420
the workloads move
to the first and third nodes.

05:56.250 --> 06:00.090
Finally, the third node where the workloads
are shared between the first two.

06:00.660 --> 06:03.760
Until we have all nodes
upgraded to a newer version,

06:03.930 --> 06:09.520
we then follow the same procedure
to upgrade the nodes from 1.11 to 1.12,

06:09.690 --> 06:10.980
and then 1.13.

06:11.880 --> 06:15.330
The third strategy would be
to add new nodes to the cluster.

06:16.410 --> 06:19.320
Nodes with newer software version.

06:21.030 --> 06:24.820
This is especially convenient
if you’re on a cloud environment

06:24.990 --> 06:28.560
where you can easily provision new
nodes and decommission old ones.

06:29.010 --> 06:32.580
Nodes with the newer software
version can be added to the cluster.

06:33.510 --> 06:35.280
Move the workload over to the new

06:37.620 --> 06:40.810
and remove the old node until

06:40.980 --> 06:44.850
you finally have all new nodes
with the new software version.

06:46.770 --> 06:48.390
Let us now see how it is done?

06:49.080 --> 06:53.130
Say we were to upgrade this
cluster from 1.11 to 1.13,

06:53.610 --> 06:57.660
kubeadm has an upgrade command
that helps in upgrading clusters.

06:58.110 --> 07:01.720
With kubeadm,
run the kubeadm upgrade plan command

07:01.890 --> 07:04.500
and it will give you a lot
of good information.

07:04.710 --> 07:08.080
The current cluster version,
the kubeadm tool version,

07:08.250 --> 07:10.710
the latest stable
version of Kubernetes.

07:11.340 --> 07:13.510
Then it lists all
the control plane components

07:13.680 --> 07:18.150
and their versions and what
version these can be upgraded to.

07:18.600 --> 07:22.510
It also tells you that after we upgrade
the control plane components,

07:22.680 --> 07:25.860
you must manually upgrade
the kubelet versions on each node.

07:26.910 --> 07:31.440
Remember, kubeadm does not
install or upgrade kubelets.

07:31.890 --> 07:35.250
Finally, it gives you the command
to upgrade the cluster.

07:35.520 --> 07:38.170
Also, note that you must
upgrade the kubeadm tool

07:38.340 --> 07:40.680
itself before you can
upgrade the cluster.

07:41.370 --> 07:46.080
The kubeadm tool also follows the same
software version as Kubernetes.

07:46.830 --> 07:53.500
We are at 1.11 and we want to go to 1.13,
but remember we can only go one minor

07:53.670 --> 07:56.460
version at a time so
we first go to 1.12.

07:57.000 --> 08:01.020
First, upgrade the kubeadm
tool itself to version 1.12.

08:01.710 --> 08:04.790
Then upgrade the cluster
using the command from

08:04.960 --> 08:08.640
the upgrade plan output,
kubeadm upgrade apply.

08:09.660 --> 08:13.590
It pulls the necessary images
and upgrades the cluster components.

08:13.770 --> 08:18.060
Once complete, your control
plane components are now at 1.12.

08:18.960 --> 08:21.400
If you run the kubectl
get nodes command,

08:21.570 --> 08:24.510
you will still see
the master node at 1.11.

08:24.960 --> 08:27.750
This is because,
in the output of this command,

08:27.960 --> 08:32.470
it is showing the versions of kubelets
on each of these nodes registered

08:32.640 --> 08:36.810
with the apiserver and not
the version of the apiserver itself.

08:37.620 --> 08:39.930
The next step is
to upgrade the kubelets.

08:40.260 --> 08:43.180
Remember, depending
on your setup you may or may not

08:43.350 --> 08:45.480
have kubelets running
on your master node.

08:45.780 --> 08:50.740
In this case, the cluster deployed with
kubeadm has kubelets on the master node,

08:50.910 --> 08:55.410
which are used to run the control plane
components as part on the master nodes.

08:56.130 --> 09:00.450
When we set up a Kubernetes cluster
from scratch, later during this course,

09:00.900 --> 09:03.220
we did not install kubelet
on the master node.

09:03.390 --> 09:07.260
You will not see the master node
in the output of this command in that case.

09:08.430 --> 09:10.180
The next step is to upgrade the kubelet

09:10.350 --> 09:12.900
on the master node
if you have kubelets on them.

09:13.440 --> 09:17.100
Run the apt-get upgrade
kubelet command for this.

09:17.640 --> 09:21.180
Once the package is upgraded,
restart the kubelet service.

09:22.980 --> 09:25.810
Running the kubectl get
nodes command now shows

09:25.980 --> 09:28.470
that the master has
been upgraded to 1.12.

09:29.190 --> 09:34.920
The worker nodes are still at 1.11.
Next, the worker nodes.

09:35.370 --> 09:39.730
Let us start one at a time.
We need to first move the workloads from

09:39.900 --> 09:42.330
the first worker node
to the other nodes.

09:42.660 --> 09:46.030
The kubectl drain command
lets you safely terminate all

09:46.200 --> 09:49.620
the pods from a node and reschedules
them on the other nodes.

09:50.430 --> 09:54.070
It also cordons the node
and marks it unschedulable.

09:54.240 --> 09:57.180
That way no new ports
are scheduled on it.

09:57.840 --> 10:00.340
Then upgrade the kubeadm
and kubelets packages

10:00.510 --> 10:02.940
on the worker nodes as
we did on the master node.

10:03.480 --> 10:06.150
Then using the kubeadm
tool upgrade command,

10:07.170 --> 10:10.560
update the node configuration
for the new kubelet version,

10:11.310 --> 10:13.170
and restart the kubelet service.

10:13.830 --> 10:16.770
The node should now be up
with the new software version.

10:17.400 --> 10:21.690
However, when we drain the node,
we actually marked it unschedulable.

10:22.020 --> 10:27.300
We need two unmark it by running
the command kubectl uncordon node-1.

10:28.260 --> 10:31.930
The node is now schedulable
but remember that it is not

10:32.100 --> 10:34.920
necessary that the pods come
right back to this node.

10:35.460 --> 10:37.620
It is only marked as schedulable.

10:38.070 --> 10:41.350
Only when the pods are deleted
from the other nodes or when new

10:41.520 --> 10:44.730
pods are scheduled do they really
come back to this first node.

10:45.990 --> 10:47.980
It will soon come when
we take down the second

10:48.150 --> 10:50.760
node to perform the same
steps to upgrade it.

10:51.300 --> 10:56.700
Finally, the third node.
We now have all nodes upgraded.

10:58.230 --> 10:59.500
That’s it for this lecture.

10:59.670 --> 11:03.400
Head over to the practice test where
you will practice upgrading a live

11:03.570 --> 11:08.700
cluster with applications running
on it without taking the applications down.

11:09.540 --> 11:10.540
Good luck.