WEBVTT

00:00.730 --> 00:03.690
Let us now look at
the backup and restore methods.

00:04.160 --> 00:07.520
We have a working kubernetes cluster
with a set of applications running.

00:07.600 --> 00:09.100
Let's first explore the setup.

00:09.960 --> 00:12.759
Let's see how many deployments exist.

00:13.590 --> 00:15.520
Currently,
we see that there are two applications.

00:15.550 --> 00:17.830
There is a red
and there is a blue application running.

00:18.310 --> 00:21.744
There are two deployments.

00:23.380 --> 00:26.270
Now, what is the version
of ETCD running on the cluster?

00:26.630 --> 00:28.880
Check the ETCD pod or process.

00:29.610 --> 00:34.509
Let's first see get pods so we know

00:34.510 --> 00:37.424
that ETCD runs
in the kube-system namespace.

00:39.140 --> 00:44.208
We see that we have the etcd-controlplane
and this is the pod.

00:44.580 --> 00:51.010
From this, we can also guess
that it's a static pod.

00:52.270 --> 00:55.510
That's what we learn
in the static pod section.

00:56.660 --> 00:58.710
Let's explore this in more detail.

00:58.860 --> 01:06.729
Let's do a describe pod kube-system.

01:08.990 --> 01:17.420
We see that we have
the single container which is etcd,

01:18.110 --> 01:22.950
and it has the image which is etcd: 3.5.1,

01:22.960 --> 01:25.100
so that's basically
the answer to this question.

01:25.584 --> 01:28.960
The version is 3.5.1.

01:29.930 --> 01:30.930
Let's select that.

01:34.110 --> 01:35.400
At what address can you reach

01:35.410 --> 01:37.720
the ETCD cluster
from the controlplane node?

01:38.880 --> 01:42.550
While most of those details
are given here, if you look at this,

01:42.590 --> 01:45.970
this container uses the ETCD command
and all of these options.

01:47.880 --> 01:55.010
From this, we see that the URL

01:55.020 --> 02:01.872
or the address that ETCD listens to
is 127.0.0.1 and 2379.

02:02.944 --> 02:05.520
It's 127.0.0.1 and 2379.

02:06.080 --> 02:08.970
2379 is the pod that ETCD listens on.

02:09.320 --> 02:11.930
There are many other options as well.

02:11.940 --> 02:14.030
Let's just select this for now.

02:16.350 --> 02:18.850
Where is the ETCD server
certificate file located?

02:19.500 --> 02:24.210
If you look at this,
we have the ETCD cert-file,

02:24.240 --> 02:26.510
so this is the ETCD server
certificate file.

02:26.880 --> 02:33.550
It's etc/kubernetes/pki/etcd/server.crt,
so that's this option right here.

02:36.790 --> 02:39.520
Where is the ETCD CA
certificate file located?

02:39.620 --> 02:43.210
The CA certificate is the option
that has the CA in it,

02:43.220 --> 02:44.450
so the trusted CA file.

02:45.120 --> 02:49.744
That's the etc/kubernetes/pki/etcd/ca.crt.

02:52.309 --> 02:53.720
That's this one right here.

03:02.280 --> 03:04.704
The next one is a task.

03:05.136 --> 03:10.500
The master node in our cluster is planned
for a regular maintenance reboot tonight.

03:10.920 --> 03:13.250
Now, while we do not anticipate
anything to go wrong,

03:13.260 --> 03:15.310
we are required
to take the necessary backups.

03:15.790 --> 03:19.740
Take a snapshot of the ETCD database
using the built-in snapshot functionality.

03:20.800 --> 03:22.680
We're going to have to take a snapshot.

03:23.160 --> 03:25.510
We've learned
that we used the etcdctl utility.

03:30.830 --> 03:32.500
Let's understand this a little bit more.

03:32.580 --> 03:36.439
Here, we know that we have the cert file.

03:37.730 --> 03:40.530
We have the data directory
which is set to var/lib/etcd.

03:41.860 --> 03:45.350
We have the key file,
so this is the private key file.

03:45.990 --> 03:47.300
We have the CA file.

03:47.830 --> 03:51.060
We also have the URL
that it's listening to.

03:51.248 --> 03:55.040
We'll need all of that information
for us to take a backup.

03:57.330 --> 03:58.960
Let me first take a look at

03:59.340 --> 04:02.576
the definition file
that's used to create this pod.

04:02.690 --> 04:04.180
We know that it's a static pod.

04:04.660 --> 04:08.760
We've learned
that the static pods are created

04:08.980 --> 04:12.816
from the etc/kubernetes/manifests
directory.

04:13.030 --> 04:18.400
Here are the files
that are used for creating this pod.

04:18.690 --> 04:24.410
Here, the ETCD is the one that's used,
so let's look into that file.

04:30.110 --> 04:31.580
If you look at this, this is a pod.

04:32.064 --> 04:34.570
Here are all the configurations for it.

04:35.190 --> 04:37.440
This is just the stuff that we just saw.

04:38.368 --> 04:44.790
I also want to take your attention
to some of the relevant information here.

04:45.200 --> 04:46.200
If you look at this,

04:46.832 --> 04:50.256
we have volumes configured
which is set to hostPath.

04:50.790 --> 04:56.180
This means that
this particular pod has these volumes,

04:56.190 --> 05:00.560
and these could be external volumes,
but in this case, it says hostPath.

05:00.660 --> 05:05.120
hostPath just refers to a path on the node
that this particular pod is running on.

05:05.670 --> 05:11.170
In this case, it's just a directory
on the host that this pod is running on,

05:11.180 --> 05:13.980
which happens
to be the control plane node in this case.

05:15.300 --> 05:18.816
There's a volume called etcd-certs

05:19.190 --> 05:21.720
which is created
from this path on the host.

05:22.610 --> 05:24.656
That's /etc/kubernetes/pki/etcd,

05:25.130 --> 05:27.920
and this basically is used
as a volumeMount here.

05:29.390 --> 05:32.510
What this says
is that this particular path on the host

05:32.520 --> 05:34.950
or on the controlplane is mounted

05:35.080 --> 05:37.910
to this particular path
within the container.

05:38.040 --> 05:40.610
This is /etc/kubernetes/pki/etcd.

05:40.620 --> 05:44.350
It's the same kind of path,
but this is on the controlplane host,

05:44.390 --> 05:47.850
and this is within the etcd container.

05:49.248 --> 05:53.090
Similarly,
we have the data path to etcd-data.

05:54.150 --> 05:56.070
This is where it actually
stores all the data.

05:56.559 --> 05:58.669
On the host, it's at /var/lib/etcd

05:59.160 --> 06:04.610
and the same is mount
to /var/lib/etcd within the container.

06:05.010 --> 06:06.830
If you look at the options
that are passed here,

06:06.840 --> 06:09.230
you have the data directory
and the data directory is

06:09.312 --> 06:13.260
/var/lib/etcd within the container.

06:13.710 --> 06:15.344
That's how it maps.

06:15.570 --> 06:21.856
Basically, what I'm trying to say is
if you look at these characters,

06:22.250 --> 06:29.230
if you look at this directory on the host
which is /var/lib/etcd,

06:29.240 --> 06:30.690
and that's what's given here.

06:31.550 --> 06:33.800
If you look at this directory on the host,

06:33.880 --> 06:38.190
you see these files or this directory
and anything that are under it.

06:38.200 --> 06:43.460
These are the files that are created
by etcd to store information.

06:44.380 --> 06:48.900
Similarly, if you look at
this particular path on the local host,

06:49.550 --> 06:54.150
these are the certificate files
that are used by ETCD,

06:55.340 --> 07:00.480
and these are automatically mapped
into the respective folders.

07:00.940 --> 07:04.860
In this case,
they happen to be the same kind of path.

07:05.260 --> 07:07.120
They have the same name set.

07:08.560 --> 07:11.110
They have the same path set
between this and this,

07:11.190 --> 07:13.990
so they are identical,
but they don't really have to be.

07:16.300 --> 07:17.500
Knowing that information,

07:17.510 --> 07:20.970
let's try and perform
the task of taking a backup.

07:21.390 --> 07:25.248
We have the etcdctl snapshot command.

07:25.760 --> 07:28.160
If you just run it directly as like this,

07:28.170 --> 07:30.680
it's going to tell
that there's no help topic for snapshot.

07:30.760 --> 07:33.960
Basically,
etcdctl can't recognize this command.

07:35.270 --> 07:36.570
You might come across this

07:37.870 --> 07:39.250
when you're working on
a Kubernetes cluster,

07:39.260 --> 07:42.110
or when you're working with etcdctl,
or maybe in the exam.

07:42.192 --> 07:46.240
You should know that this happens
not because your command is wrong.

07:46.280 --> 07:57.810
It happens because you've not set
the ETCDCTL_API version to three.

08:02.000 --> 08:05.230
This environment variable needs
to be set before you run this command.

08:05.712 --> 08:08.840
If you don't want
to specify this on each command,

08:09.120 --> 08:11.180
one thing you could do is just export

08:12.050 --> 08:13.900
in that environment
that you're working on.

08:13.980 --> 08:21.110
You could do an export
ETCDCTL and API was three.

08:22.859 --> 08:26.840
Now, you could just do etcdctrl snapshot

08:28.760 --> 08:32.230
either you pass it in at the beginning
of each command or you export it

08:32.710 --> 08:34.060
as an environment variable.

08:36.208 --> 08:38.150
We have the etcdctrl snapshot command,

08:38.360 --> 08:41.090
and we need to pass
some information to it,

08:41.109 --> 08:43.289
so we're going to do a save.

08:43.510 --> 08:44.780
We have to take a snapshot.

08:44.860 --> 08:48.880
Let's do a save and then we need
some of these parameters,

08:48.890 --> 08:49.550
so you have to pass

08:49.560 --> 08:56.448
in the CA certificate files for it
to communicate with the endpoint.

08:56.540 --> 08:59.860
We definitely need the endpoint,
so we're going to have that first.

09:00.340 --> 09:05.664
The endpoint as we have seen here is this.

09:08.660 --> 09:09.660
Copy that.

09:13.920 --> 09:15.370
You don't really need this.

09:17.430 --> 09:18.580
That's the first thing.

09:19.936 --> 09:22.416
Next, we need the cacert.

09:26.190 --> 09:28.030
If you go to the output of the file,

09:29.450 --> 09:30.450
we have this

09:36.064 --> 09:38.928
and then we need the certificate

09:42.160 --> 09:45.504
which happens to be this.

09:49.460 --> 09:52.576
Then, we need the key,

09:57.920 --> 09:59.960
which happens to be this.

10:01.630 --> 10:04.810
Just remember
that there are three certificate files

10:04.820 --> 10:07.450
that you should pass through.

10:08.016 --> 10:12.390
We've done the etcdctl snapshot save,

10:12.520 --> 10:14.410
and then we've specified the endpoint,

10:14.490 --> 10:16.690
and the certificate files
that are required.

10:17.500 --> 10:21.140
The one thing there you should specify
is the location of the backup.

10:21.220 --> 10:24.928
Store the backup file at location this,
so we're going to pass that.

10:27.480 --> 10:28.480
Let's give it a shot.

10:29.008 --> 10:31.440
Snapshot saved at this locations.

10:33.280 --> 10:37.010
Make sure that file is there.

10:39.136 --> 10:40.680
The snapshot has been taken.

10:42.720 --> 10:43.020
Great.

10:43.030 --> 10:45.390
Let us now wait for
the maintenance window to finish.

10:45.400 --> 10:49.930
Go get some sleep
and let's click OK to continue.

10:51.296 --> 10:53.900
It's about 2:00 AM at midnight
and you get a call.

10:55.590 --> 10:55.910
Wake up.

10:55.940 --> 10:57.240
We have a conference call.

10:57.260 --> 10:59.910
After the reboot,
the master nodes came back online,

11:00.280 --> 11:02.300
but none of our applications
are accessible.

11:02.380 --> 11:04.700
Check the status
of the applications on the cluster.

11:04.780 --> 11:05.430
What's wrong?

11:05.510 --> 11:08.260
Let's do a kubectl get deploy.

11:09.460 --> 11:11.360
See that the applications are not there.

11:11.440 --> 11:12.280
Check the pods.

11:12.360 --> 11:13.360
Pods are not there.

11:14.208 --> 11:15.910
Let's check services.

11:16.440 --> 11:20.120
We only have
the built-in kubernetes service,

11:20.520 --> 11:22.690
but the service
for the applications are not there.

11:22.779 --> 11:23.869
The deployments are not present.

11:23.870 --> 11:24.910
The services are not present.

11:24.920 --> 11:25.800
The pods are not present.

11:25.888 --> 11:27.580
Nothing seems to be there.

11:28.620 --> 11:31.900
That's all of the above,
not above actually,

11:31.910 --> 11:33.700
all of the options basically.

11:35.010 --> 11:36.560
Luckily, we took a backup.

11:36.700 --> 11:40.340
Restore the original state
of the cluster using the backup file.

11:41.330 --> 11:42.384
Let's do that next.

11:44.830 --> 11:47.930
The restore option,
what we're going to do is,

11:47.950 --> 11:50.010
so we have a file with the backup.

11:51.290 --> 11:55.570
Now, we could use
the etcdctl restore command to restore

11:55.580 --> 12:00.470
a given file and restore the data
to a directory on the local host.

12:00.912 --> 12:03.510
What we're going to do
is we're going to restore the data

12:03.550 --> 12:05.310
to another directory on the local host.

12:05.792 --> 12:10.820
Then, we're going to go
into this etcd manifests file,

12:11.219 --> 12:13.400
and we're going to change.

12:15.800 --> 12:21.820
We're going to reconfigure this
to use the data not from /var/lib/etcd,

12:21.830 --> 12:25.710
but from that new path
that we have restored the data to.

12:26.960 --> 12:30.010
That's the strategy
that we're going to follow.

12:30.500 --> 12:34.179
To restore etcd, again, remember

12:34.180 --> 12:37.130
if you've not set
the export etcdctl API version,

12:37.540 --> 12:39.700
remember to pass that in,
but in this case, since I've said it,

12:39.730 --> 12:42.630
I'm not going to put that
in the beginning of this command,

12:42.640 --> 12:46.130
so etcdctl snapshot restore.

12:47.610 --> 12:49.830
Now, remember that
when you're doing a snapshot,

12:49.849 --> 12:51.119
say when you're taking a snapshot,

12:51.120 --> 12:56.580
you're actually taking a snapshot
by contacting the etcd server directly.

12:57.210 --> 12:59.160
For that, you need all the certificates.

12:59.640 --> 13:02.970
Now, our goal
is that we have the file already,

13:02.990 --> 13:05.370
and all we're going to do
is we're going to restore

13:05.859 --> 13:09.399
this backup file to a folder.

13:09.550 --> 13:10.740
That's what we're going to do.

13:11.220 --> 13:13.670
We're going to specify a data directly

13:13.680 --> 13:17.010
where we want the backup
to be restored to.

13:18.890 --> 13:26.032
We'll create a new directory called
/var/lib/etcd-from-backup, for instance.

13:26.240 --> 13:28.740
This is where the data
is going to be restored to.

13:29.620 --> 13:33.560
We just specify
the location of the backup.

13:34.320 --> 13:37.520
Remember that we're not going to specify
the endpoints or the certificate details

13:37.530 --> 13:42.700
because this particular operation
of restoring data from an etcd backup

13:43.260 --> 13:46.510
is not done by communicating
with the etcd server.

13:46.620 --> 13:50.360
It's just a local activity
that we're running to restore it

13:50.370 --> 13:57.070
from this path and extracting the data
into a new directory that will be created.

13:59.190 --> 14:01.664
The restore is complete.

14:01.790 --> 14:09.008
Let's check the data is
in /var/lib/etcd-from-backup.

14:10.660 --> 14:11.660
That's the data.

14:11.740 --> 14:13.420
Now, what all we need to do is go back

14:13.500 --> 14:20.688
into kubernetes/manifests file
into the etcd.yaml.

14:23.010 --> 14:27.900
All we need to do here is go here

14:27.910 --> 14:35.100
and change the hostPath
and add etcd-from-backup here.

14:35.830 --> 14:37.610
Basically what's going to happen is,

14:38.096 --> 14:41.950
as we know that any change on this file
is going to recreate this pod,

14:41.980 --> 14:44.120
and when this pod recreate is recreated,

14:44.200 --> 14:46.770
it's going to consider this as the data

14:46.920 --> 14:51.510
on the local host that has
the data for the etcd server,

14:51.940 --> 14:59.440
and then is going to mount this
to the /var/lib/etcd within the pod.

14:59.770 --> 15:01.710
It doesn't matter if these are different.

15:02.190 --> 15:05.790
If this has changed, that's fine.

15:05.840 --> 15:09.870
This volume called etcd-data
is then going to be monitored

15:09.910 --> 15:12.304
to /var/lib/etcd within this.

15:12.960 --> 15:14.850
Either you can just leave it like this

15:14.870 --> 15:17.330
or you can change this
and also change this.

15:17.480 --> 15:18.650
If you change this,

15:18.730 --> 15:25.280
then you have to make sure
that you go up to the data directory here

15:25.310 --> 15:26.310
and change this.

15:27.300 --> 15:34.910
This line and this line should match,
not all the time,

15:35.430 --> 15:42.480
but this and this may not match only

15:42.960 --> 15:46.990
because they are connected
using this name here called etcd-data.

15:47.960 --> 15:49.140
This is what connects them.

15:49.620 --> 15:52.260
If these values are different,
that should be fine.

15:53.180 --> 15:54.330
I'm going to save this.

15:55.049 --> 15:56.470
When I save this what's going to happen

15:56.490 --> 16:02.820
is I might lose access
for a bit because now,

16:07.960 --> 16:11.840
the etcd server is going to
go down for a short period of time.

16:11.880 --> 16:15.190
Because I've changed
the etcd manifest file,

16:17.056 --> 16:20.290
the kubernetes
is now going to restart etcd.

16:20.880 --> 16:23.600
It's going to shut down
the running etcd pod.

16:24.020 --> 16:28.528
Then, it's going to create
a new pod-based on the new information.

16:29.340 --> 16:31.632
Let's just give it a few minutes.

16:31.730 --> 16:34.256
This can take a few minutes time.

16:35.660 --> 16:39.010
Let's just wait for it
to restore and then come back.

16:44.599 --> 16:45.430
I just got an output,

16:45.450 --> 16:48.079
but this is an old output

16:48.100 --> 16:55.480
because as you can see still says etcd
is 37 minutes old which is not true.

16:55.680 --> 16:57.552
I expect it to be just a few seconds.

16:57.990 --> 16:59.510
Right now, as you can see,

16:59.520 --> 17:04.120
it's just up
and it's just been up for 14 seconds.

17:04.130 --> 17:07.088
This is the new etcd-controlplane.

17:08.170 --> 17:12.730
Let's do a watch and wait for

17:12.740 --> 17:18.000
this etcd-controlplane to go
from a pending state to a running state.

17:18.450 --> 17:19.470
As you can see here,

17:19.480 --> 17:23.950
when the etcd-controlplane
at etcd pod is restarted,

17:23.969 --> 17:24.849
it also restarts

17:24.850 --> 17:27.488
the kube-controller-manager
and the scheduler.

17:29.360 --> 17:32.110
Let's take a look at
the logs of the etcd-controlplane.

17:34.784 --> 17:36.864
logs -n kube-system etcd-controlplane.

17:45.800 --> 17:46.800
It's waiting.

17:47.470 --> 17:49.170
It's in a ContainerCreating state.

17:49.920 --> 17:52.144
Let's take a look at it.

18:08.560 --> 18:15.620
As you can see, there are Liveness
and Startup probes that are failing.

18:16.320 --> 18:17.552
Let's do one thing.

18:18.240 --> 18:19.640
Let's take the status again.

18:23.240 --> 18:26.352
If it does not automatically
go to a running state,

18:26.656 --> 18:28.080
it might need a restart.

18:28.240 --> 18:30.656
Let's just delete it.

18:38.300 --> 18:39.300
It gets restarted.

18:41.340 --> 18:42.340
Check it again.

18:42.592 --> 18:45.344
This time, it's back,
and it's now in a running state.

18:46.976 --> 18:50.976
Let's see status of deployments now.

18:51.600 --> 18:53.650
We can see that the deployments are back.

18:55.328 --> 19:01.980
Pods are back
and the services as well are back.

19:04.870 --> 19:06.864
That's the end of this lab.