WEBVTT

00:00.467 --> 00:03.723
In this video,
we're going to look at the solution

00:03.803 --> 00:07.303
for the lab on viewing certificate details.

00:08.153 --> 00:10.143
The first question
is to identify the certificate

00:10.223 --> 00:13.056
file used for the kube-api server.

00:14.003 --> 00:18.063
Let's look at the manifest
files for the kube-api server.

00:18.143 --> 00:20.226
We know that they are under

00:20.306 --> 00:23.393
/etc/kubernetes/manifests,

00:24.023 --> 00:27.143
and we have the kube-api server.

00:28.793 --> 00:30.877
If you look at this file,

00:31.033 --> 00:33.553
we know that we need to look

00:33.633 --> 00:37.503
for the certificate file used
for the kube-api server.

00:37.583 --> 00:39.963
These are the files
you connect to the etcd server.

00:40.043 --> 00:43.043
These are the kubelet and the proxy.

00:45.563 --> 00:49.473
The certificate for the kube-api
server itself are these.

00:49.553 --> 00:52.893
We have the certificate
file here and the key here,

00:52.973 --> 00:57.700
so /etc/kubernetes/pki/apiserver.crt.

01:00.600 --> 01:03.053
Pki/apiserver.crt.

01:03.133 --> 01:04.733
That's this one.

01:08.153 --> 01:11.013
Identify the certificate
file used to authenticate

01:11.093 --> 01:14.463
kube-api server as a client to etcd server.

01:14.543 --> 01:17.043
This is the kube-api server configuration,

01:17.123 --> 01:22.343
and kube-api server
connecting to etcd server

01:22.553 --> 01:25.970
is connecting to etcd server as a client.

01:26.423 --> 01:28.840
These are the relevant files.

01:29.273 --> 01:32.073
This is the address of the etcd
server and then you have the cafile,

01:32.153 --> 01:34.323
the certificate file, and the key.

01:34.403 --> 01:36.453
Here, we're asked
to identify the certificate file

01:36.533 --> 01:39.903
used to authenticate
the kube-api server to etcd.

01:39.983 --> 01:40.953
The cert file is this.

01:41.033 --> 01:45.600
That's
/etc/kuberntes/pki/apiserver-etcd-client.crt.

01:46.200 --> 01:49.643
Etcd-client.crt.

01:52.553 --> 01:54.637
Looks like it's this one.

01:57.567 --> 02:00.483
Identify the key used to authenticate

02:00.563 --> 02:04.063
the kube-api server to the kubelet server.

02:04.763 --> 02:06.963
These are the files
that are relevant to the kubelet

02:07.043 --> 02:10.203
and you have the certificate
file here and then the key.

02:10.283 --> 02:14.533
That's
/etc/kubernetes/pki/apiserver-kubelet-client.

02:14.700 --> 02:16.783
Apiserver-kubelet-client.

02:17.433 --> 02:19.900
/etc/kubernetes/pki.

02:22.733 --> 02:25.943
We're asked to identify the key.

02:26.753 --> 02:28.420
It's not the crt.

02:28.500 --> 02:31.584
It's the one that has key at the end.

02:36.033 --> 02:37.951
Just to summarize here,

02:38.033 --> 02:43.023
these are the files used by the kube-api
server itself to host itself.

02:43.103 --> 02:44.943
That's under /etc/kubernetes/pki,

02:45.023 --> 02:49.440
and then you have the apiserver.crt
and apiserver.key.

02:50.423 --> 02:54.090
Everything for connecting
to the etcd server,

02:55.823 --> 02:57.906
the cafile is under etcd,

02:58.913 --> 03:05.333
but then the apiserver-client
files are still under the pki directory.

03:06.503 --> 03:11.887
Then here to the kubelet-client files,

03:11.967 --> 03:15.134
they're still under the pki directory.

03:15.983 --> 03:18.900
That's what we are concerned about.

03:19.463 --> 03:23.883
If you look at it, anything
to do with the kubernetes-apiserver,

03:23.963 --> 03:28.103
they're under the pki directory.

03:28.523 --> 03:34.083
Either the apiserver certificate
and key itself or the client files,

03:34.163 --> 03:37.413
which are used to connect
to either etcd or kubelet,

03:37.493 --> 03:41.703
they're all under /etc/kubernetes/pki
under the pki directory.

03:41.783 --> 03:43.593
Think about it that way.

03:43.673 --> 03:45.633
Now, we're asked to identify the etcd server

03:45.713 --> 03:49.083
certificate used to host
the etcd server itself.

03:49.163 --> 03:54.330
For this, we're going to look
at the etcd server manifest file.

03:55.253 --> 04:00.713
Here, we have the certificate file.

04:01.373 --> 04:03.956
It is under the etcd directory.

04:05.573 --> 04:08.990
The key is also under the etcd directory.

04:09.653 --> 04:11.373
Then you have the cafile.

04:11.453 --> 04:13.113
It's also under the etcd directory.

04:13.193 --> 04:16.113
Anything to do with the etcd
server itself for its own

04:16.193 --> 04:17.153
hosting is under

04:17.233 --> 04:21.083
the /etc/kubernetes/pki/etcd directory.

04:21.623 --> 04:25.293
In this case, we are asked
to identify the etcd server certificate.

04:25.373 --> 04:27.063
That's this certificate.

04:27.143 --> 04:31.200
That's /etc/kubernetes/pki/etcd/server.crt.

04:31.600 --> 04:33.900
That's /etc/kubernetes/pki.

04:34.700 --> 04:36.450
There's no etcd here.

04:40.043 --> 04:42.953
/etc/kubernetes/pki/etcd/server.crt.

04:43.033 --> 04:44.633
That's this one.

04:47.633 --> 04:50.181
Now, identify the etcd
server CA Root certificate

04:50.261 --> 04:52.761
used to serve the etcd server.

04:53.363 --> 04:58.943
Here, we have the CA file here,
which is a trusted CA file.

04:59.453 --> 05:03.453
This is under the kubernetes/pki/etcd/
directory.

05:05.663 --> 05:11.853
As we discussed, in this case,
the etcd server has its own CA file,

05:11.933 --> 05:16.433
trusted CA Root file
which is under the etcd directory.

05:16.523 --> 05:21.553
The kube-api server has its own
which is in a different directory.

05:21.633 --> 05:23.373
Let's keep that in mind.

05:23.453 --> 05:26.853
This is under
/etc/kubernetes/pki/etcd/ca.crt.

05:26.933 --> 05:29.267
It's etcd/ca.crt.

05:30.300 --> 05:31.700
It's this one.

05:34.373 --> 05:39.363
What is the common name configured
on the kube-api server certificate?

05:39.443 --> 05:44.123
We know that kube-api
server cerificate is this one.

05:47.573 --> 05:52.733
We run that through the openssl x509,

05:55.193 --> 05:59.063
filename and text, noout.

06:00.803 --> 06:03.183
That command is going to give
us this information.

06:03.263 --> 06:04.653
If you look inside here,

06:04.733 --> 06:06.843
you have information about the certificate.

06:06.923 --> 06:09.003
The issuer is Kubernetes.

06:09.083 --> 06:13.520
The subject which is what this certificate
is issued to

06:13.600 --> 06:16.083
is the command name
for that is kube-api server.

06:16.163 --> 06:18.080
That's kube-api server.

06:18.443 --> 06:20.493
That's the answer to this question.

06:20.573 --> 06:22.490
That's kube-api server.

06:26.303 --> 06:28.020
Now, what is the name of this CA

06:28.100 --> 06:31.023
who issued the kube-api server certificate?

06:31.103 --> 06:35.073
If you look at here, you have the issuer
and the name is Kubernetes.

06:35.153 --> 06:38.627
Kubernetes is the CA

06:39.333 --> 06:42.833
who issued the kube-api server certificate.

06:44.483 --> 06:46.503
Now, which of the below alternate names

06:46.583 --> 06:49.233
is not configured on the kube-api server?

06:49.313 --> 06:53.897
As we discussed,
the kube-api server has multiple names.

06:54.023 --> 06:57.363
It could be controlled name kubernetes,
kubernetes.default,

06:57.443 --> 07:03.353
kubernetes.default.svc,
kubernetes.default.svc.cluster.local,

07:03.953 --> 07:06.483
and this IP address is as well.

07:06.563 --> 07:08.163
Kubernetes, yes.

07:08.483 --> 07:10.163
Controlplane, yes.

07:10.253 --> 07:12.863
Kubernetes.default.svcs, yes.

07:13.373 --> 07:14.463
Kubemaser, no.

07:14.543 --> 07:15.920
There is no kubemaser.

07:16.000 --> 07:20.000
In this case,
that's the answer to that question.

07:21.503 --> 07:26.793
Now, what is the common name
configured on the etcd server certificate?

07:26.873 --> 07:32.123
Let's look at the etcd server certificate,
and that is this one.

07:36.833 --> 07:40.000
Let's run it through the same command.

07:40.223 --> 07:45.473
We're just going to replace
it with the etcd server certificate.

07:47.633 --> 07:50.800
Here we can see the issuer is etcd-ca.

07:50.993 --> 07:52.623
Etcd has its own CA.

07:52.703 --> 07:54.963
That's what we need to understand from here.

07:55.043 --> 07:59.127
If you remember,
kubernetes had the CA Kubernetes,

07:59.873 --> 08:02.207
but here you have anther CA.

08:03.353 --> 08:06.483
The common name configured
on the etcd server certificate,

08:06.563 --> 08:11.763
if you look at this common name
for the etcd server is controlplane.

08:11.843 --> 08:16.743
This is not the controlplane as this.

08:16.823 --> 08:18.423
This is not the kubernetes controlplane.

08:18.503 --> 08:22.753
This is referring to the etcd
controlplane, in fact.

08:23.633 --> 08:25.133
It's like that.

08:27.203 --> 08:28.893
Now, how long from the issued date

08:28.973 --> 08:31.743
is the kube-api server
certificate valid for?

08:31.823 --> 08:35.323
We have the output for the apiserver here.

08:35.483 --> 08:39.146
If you look at the validity,
it's not valid before.

08:39.226 --> 08:40.563
This is basically the starting.

08:40.643 --> 08:43.193
This is April 17, 2022.

08:43.823 --> 08:49.373
It's valid until April 17, 2023.

08:49.600 --> 08:51.200
That's one year.

08:55.613 --> 08:59.643
How long from the issued date
is the Root CA certificate valid for?

08:59.723 --> 09:03.390
The Root CA certificate
what we know is here.

09:05.066 --> 09:08.499
Let's run the same command.

09:11.843 --> 09:13.523
This will fnd this.

09:14.003 --> 09:19.763
If you look at the validity,
it's April 17, 2022

09:20.183 --> 09:23.287
to April 14, 2032.

09:23.367 --> 09:26.600
That's approximately just 10 years.

09:27.433 --> 09:29.113
Let's start that.

09:32.543 --> 09:37.377
Going to a change window,
there are some changes happening.

09:43.133 --> 09:46.520
Kubectl suddenly stops responding
to your command after a change window.

09:46.600 --> 09:47.220
Check it out.

09:47.300 --> 09:49.773
Someone recently
modified this pretty little file.

09:49.853 --> 09:51.693
You're asked to investigate
and fix the issue.

09:51.773 --> 09:57.190
Once you fix the issue,
wait for some time for kubectl to respond.

09:58.067 --> 09:59.353
Let's see that.

09:59.433 --> 10:05.167
Let's do a ctl get pods,

10:05.633 --> 10:08.217
and let's see what's happening.

10:09.863 --> 10:12.031
This might take some time.

10:18.053 --> 10:21.123
Okay. It says,
"The connection to the server was refused.

10:21.203 --> 10:24.213
Did you specify the right host or port?"

10:24.293 --> 10:29.303
Apparently none of the Kubernetes
commands are working.

10:31.467 --> 10:34.853
If this happens,
this means that the kubectl utility

10:34.933 --> 10:37.833
is not able to communicate
with the kube-api server.

10:37.913 --> 10:40.623
We need to look at the status
of the kube-api server,

10:40.703 --> 10:42.423
but you can't do that with
the kubectl command,

10:42.503 --> 10:44.887
because the kubectl
command cannot communicate

10:44.967 --> 10:46.887
with the kube-api server.

10:46.967 --> 10:48.820
We have to then use Docker,

10:48.900 --> 10:53.103
because we know that these pods are in fact

10:53.183 --> 10:56.673
running as containers
on the host Docker container.

10:56.753 --> 11:02.993
Let's do a docker ps
and then grep for kube-api server

11:04.403 --> 11:07.803
and we see that we have the kube-api server

11:07.883 --> 11:11.463
and we have a kube-api server
that exited at 27 seconds ago.

11:11.543 --> 11:13.203
Then we have a new kube-api server,

11:13.283 --> 11:17.933
so let's quickly check
the logs of that container.

11:18.353 --> 11:22.103
We see that it has this error.

11:22.913 --> 11:26.423
Error while dialing tcp 127.0.0.1:2379.

11:28.913 --> 11:34.073
Now we know that 2379 is a port of etcd.

11:34.493 --> 11:38.033
This is the etcd server port.

11:40.133 --> 11:45.693
Basically, the kube-api server
is not able to connect to the etcd server.

11:45.773 --> 11:47.620
Let's look at the etcd server then.

11:47.700 --> 11:51.143
Let's do a docker ps -a

11:54.293 --> 11:56.363
and let's look for etcd.

11:57.413 --> 11:59.693
We have the etcd

12:02.243 --> 12:03.343
about here.

12:03.683 --> 12:08.093
If you look at this,
let's see if we can find--

12:14.183 --> 12:16.266
Okay. Right here it says,

12:16.643 --> 12:20.387
"etcdmain cannot find

12:20.467 --> 12:26.120
/etc/kubernetes/pki/etcd/server-certificate.crt,

12:26.200 --> 12:26.953
it's not found."

12:27.033 --> 12:28.720
Let's see where that it, that's

12:28.800 --> 12:34.223
/etc/kubernetes/pki/etcd.

12:34.643 --> 12:38.703
If you look at this directory,
there is server.crt,

12:38.783 --> 12:43.623
but there is no server-certificate.crt,
so that's the error.

12:43.703 --> 12:45.903
Something has changed in the etcd server,

12:45.983 --> 12:48.066
so let's look at the etcd

12:50.603 --> 12:51.573
manifest files.

12:51.653 --> 12:57.600
That's /etc/kubernetes/manifest/etcd.yaml

12:59.467 --> 13:00.923
and let's see

13:01.223 --> 13:07.453
where we have this configured,
which is server-certificate.crt.

13:07.533 --> 13:11.703
Someone's put in the wrong path
to the certificate here.

13:11.783 --> 13:16.443
That's the crt file and that's the issue,
so we have to fix this.

13:16.523 --> 13:18.603
The crt file for etcd as we know

13:18.683 --> 13:23.003
and we have seen
earlier is the server.crt file.

13:23.663 --> 13:26.763
It's not server-certificate,
it's just server.crt.

13:26.843 --> 13:29.753
We're going to modify this.

13:31.667 --> 13:35.453
Let's go in, edit this file,

13:37.733 --> 13:39.923
and let's remove this.

13:42.863 --> 13:45.427
Okay, let's save that

13:46.400 --> 13:51.053
and let's now give it some time to start up.

13:51.133 --> 13:53.153
Let's look at

13:56.093 --> 13:57.300
etcd again.

13:58.313 --> 14:04.453
See that it's powered up now,
a new container.

14:04.533 --> 14:07.700
Let's check the logs of that container

14:09.323 --> 14:15.467
and see that the error is no longer there.

14:16.167 --> 14:17.567
Let's see if--

14:22.013 --> 14:24.656
And it's still not working
so we might have to give

14:24.736 --> 14:30.687
it some time for the kube-api
server to pick this up.

14:30.767 --> 14:34.313
Let's check the kube-api server.

14:42.633 --> 14:44.383
We'll check the logs.

14:44.933 --> 14:49.267
This is the previous instance
of the kube-api server,

14:49.493 --> 14:54.383
so we just have to wait
for a new instance to come up.

15:06.413 --> 15:07.413
Okay.

15:08.213 --> 15:11.713
Looks like there's there's a new container.

15:12.233 --> 15:14.763
Let's check the logs of that container.

15:14.843 --> 15:17.633
Let's do a local logs,

15:18.993 --> 15:24.736
and we don't see arrows anymore.

15:25.667 --> 15:26.867
Let's check.

15:28.883 --> 15:29.920
Yes. Okay.

15:30.000 --> 15:34.967
Now we have the kubectl utility back

15:35.733 --> 15:38.093
and the node is now ready?

15:38.393 --> 15:40.811
Okay, let's give that a shot.

15:43.403 --> 15:45.323
Okay. That's success.

15:46.283 --> 15:47.963
After a few hours.

15:52.973 --> 15:56.313
I believe someone's going to break
something again.

15:56.393 --> 16:01.643
We have the kube-api server,
and it stopped again, check it out.

16:01.900 --> 16:03.650
Let's check that out.

16:04.883 --> 16:08.199
Yes, it looks like that's stopped again.

16:08.279 --> 16:10.387
Now, inspect the kube-api server logs

16:10.467 --> 16:13.953
and identify the root
cause and fix the issue.

16:14.033 --> 16:17.273
Let's look at the logs again.

16:18.743 --> 16:19.773
Okay. This time, it says,

16:19.853 --> 16:24.437
"Unable to connect to the server
TLS handshake timeout."

16:24.653 --> 16:28.953
It looks like there's some issue
with the TLS certificates.

16:29.033 --> 16:30.513
Let's check that again.

16:30.593 --> 16:33.153
Let's check the kube-api server again.

16:33.233 --> 16:38.873
Docker, ps-a, grep kube-api server.

16:41.273 --> 16:43.700
You see that succeeded.

16:44.067 --> 16:45.817
Let's do docker logs.

16:47.900 --> 16:52.293
Here it says,
authentication handshake failed,

16:52.373 --> 16:55.263
certificate signed by unknown authority.

16:55.343 --> 16:58.403
Let's see if there's something else.

17:06.683 --> 17:07.953
Okay, that's the error.

17:08.033 --> 17:11.020
Also, here, you have some information

17:11.100 --> 17:16.523
that it's failed to connect
to 127.0.0.1:2379.

17:16.883 --> 17:20.723
We know that 2379 is etcd server,

17:20.933 --> 17:22.743
so it's got something
to do with the etcd server.

17:22.823 --> 17:29.823
Basically, the kube-api server
is unable to connect to the etcd server,

17:30.293 --> 17:33.153
because of a handshake issue.

17:33.233 --> 17:36.393
It's because the certificate
is signed by unknown authorities.

17:36.473 --> 17:38.223
Let's check that out.

17:38.933 --> 17:42.833
Let's first look at the--

17:45.113 --> 17:50.753
Let's delete pod and see
if that has any issues.

17:50.843 --> 17:56.193
If you look at this, the etcd container
has been running for four minutes

17:56.273 --> 17:59.773
so let's check the logs of that container.

18:02.873 --> 18:07.733
We see that it started--

18:07.973 --> 18:09.003
Well, there were no errors,

18:09.083 --> 18:13.667
but then started getting
these rejected connection from,

18:14.363 --> 18:18.280
and this is a remote and
it's a bad certificate.

18:18.443 --> 18:21.183
It's rejecting connections from somewhere.

18:21.263 --> 18:22.773
We assume that that's the kube-api server

18:22.853 --> 18:24.633
trying to connect to that etcd server

18:24.713 --> 18:27.993
and that's because it is sending
in a bad certificate.

18:28.073 --> 18:30.093
Okay, let's try to look at the certificates

18:30.173 --> 18:34.643
configured on the api
server under manifests,

18:37.193 --> 18:38.793
kube-api server.

18:44.400 --> 18:46.443
We're basically focusing on the etcd side

18:46.523 --> 18:50.543
of things so let's just look at those alone.

18:59.453 --> 19:00.453
Okay.

19:01.583 --> 19:04.250
This is what we need to look at.

19:05.543 --> 19:11.123
Here we have the etcd-ca
file and that's set to--

19:11.453 --> 19:16.503
It is in kubernetes/pki/ca.crt
and then you have the etcd cert file.

19:16.583 --> 19:19.773
This is the client
certificate for api server

19:19.853 --> 19:23.703
so that apiserver-etcd-client.crt
set correctly.

19:23.783 --> 19:25.020
This is a key file, again,

19:25.100 --> 19:29.850
this is the api server-etcd-client.key
file which is right

19:29.933 --> 19:35.663
and the etcd servers is set
to the correct URL as well.

19:37.223 --> 19:41.613
As we discussed earlier, the Kubernetes
has its own CA file which is this

19:41.693 --> 19:44.973
and etcd has its own
CA file which is not this.

19:45.053 --> 19:48.820
Right, so etcd has a CA file, which is under

19:48.900 --> 19:53.233
kubernetes/pki/etcd,

19:53.767 --> 19:56.753
and then the CA file that's here.

19:58.193 --> 20:01.193
Right. This is the CA file for etcd.

20:01.403 --> 20:05.493
Here you should be specifying
that and not the Kubernetes CA file.

20:05.573 --> 20:06.993
That seems to be the problem.

20:07.073 --> 20:10.313
Lets go in and change.

20:15.323 --> 20:18.563
You can just add this here like this.

20:23.663 --> 20:28.073
Lets check the status of kube-api server.

20:33.143 --> 20:36.083
It started eight seconds ago.

20:36.683 --> 20:38.363
Lets check the logs.

20:43.800 --> 20:46.717
We don't see those errors any more.

20:48.803 --> 20:50.920
These are all informational.

20:51.000 --> 20:57.383
Lets check if kube-api server is back up.

20:57.473 --> 20:59.043
Yes, it seems to be back up.

20:59.123 --> 21:00.663
That was the problem.

21:00.743 --> 21:03.063
We looked at the logs
on the kube-api server.

21:03.143 --> 21:08.139
We looked at the logs on the etcd
server and the continues for both.

21:08.219 --> 21:10.720
We realize that it was a certificate
issue somewhere,

21:10.800 --> 21:13.787
so we went through and looked
at the certificate files

21:13.867 --> 21:18.063
configured on each of them
and finally figured out it was this.

21:18.143 --> 21:18.873
That was a bit tricky,

21:18.953 --> 21:23.120
but that's just something
you have to keep in mind.

21:25.733 --> 21:28.233
Well, that's all for this lab.