WEBVTT 00:00.260 --> 00:06.140 In this video, we're going to work on troubleshooting controlplane failure. 00:06.220 --> 00:08.600 The first question is, the cluster is broken. 00:08.680 --> 00:11.290 We tried to play an application, but it's not working. 00:11.370 --> 00:14.390 Troubleshoot and fix the issue. 00:14.470 --> 00:16.620 Before we begin, a quick tip. 00:16.700 --> 00:21.590 If you don't have the alias set already on your prompt. 00:21.670 --> 00:25.460 You could do a alias k=kubectl. 00:25.540 --> 00:31.560 Also, if you don't have auto-completion, then one thing you could do 00:31.640 --> 00:34.459 is go to the Kubernetes documentation pages. 00:34.539 --> 00:41.220 Search for kubectl cheat sheet. 00:41.300 --> 00:45.970 The first thing that comes here is the kubectl completion bash. 00:46.050 --> 00:48.829 Let's copy and let's run that. 00:48.909 --> 00:51.040 This basically gives you auto-completion. 00:51.120 --> 00:57.059 You do a kubectl get and then it automatically gives you 00:57.139 --> 00:58.489 the options. 00:58.569 --> 01:02.069 The other thing is, if you're setting an alias, 01:02.149 --> 01:03.819 you should do this as well to get the alias 01:03.899 --> 01:08.530 for the short form for kubectl. 01:08.610 --> 01:10.690 I'm going to set this as well. 01:11.778 --> 01:14.338 That way I can just do kubectl get 01:14.908 --> 01:19.180 and then autocompletion 01:19.260 --> 01:20.560 for the short form. 01:20.640 --> 01:24.240 We're going to look at the status of the cluster. 01:24.320 --> 01:26.600 Let's do a kubectl get nodes. 01:26.680 --> 01:30.320 We see that the node seems to be in a ready state. 01:30.400 --> 01:33.120 That's good. 01:33.200 --> 01:36.810 It says there are deployments, so let's take a look at the deployments. 01:36.890 --> 01:41.810 We see that there is a app deployed, but it's not ready. 01:41.890 --> 01:46.100 There is one container or one pod in the deployment, 01:46.180 --> 01:47.250 but that pod is not ready. 01:47.330 --> 01:48.250 Let's take a look at it. 01:48.330 --> 01:49.548 Let's do 01:52.973 --> 01:55.219 a describe deployment app. 01:56.730 --> 02:03.220 We see that there's one desired, zero available. 02:03.300 --> 02:06.120 There's one unavailable. 02:06.200 --> 02:09.169 There isn't anything here that's really helpful. 02:09.249 --> 02:12.180 Here, it says scaled up replica set to one. 02:12.260 --> 02:14.709 Let's take a look at the replica set. 02:16.555 --> 02:18.365 get rs. 02:18.446 --> 02:19.840 We have one replica set. 02:19.920 --> 02:22.010 Let's take a look at that. 02:22.090 --> 02:26.010 describe replica set app. 02:26.090 --> 02:30.028 We see that the desired is one, 02:31.616 --> 02:33.570 but it's still waiting. 02:33.650 --> 02:38.549 Here, you have the events, and it says creating pod. 02:38.629 --> 02:40.859 Let's take a look at the pod. 02:40.939 --> 02:46.920 Now, we see that there is a pod created, but it's in a pending state. 02:47.000 --> 02:51.549 Let's take a closer look at that pod. 02:51.629 --> 02:58.309 We see the events have not started yet, but everything else seems okay, 02:58.389 --> 02:59.939 but it's in a pending state. 03:00.019 --> 03:05.790 When a pod is in a pending it's most likely that that indicates 03:05.870 --> 03:08.089 that it has not been assigned a node. 03:08.169 --> 03:11.469 If you look at the node here, it's set to none. 03:11.549 --> 03:14.460 That means the pod has not been assigned to a node. 03:14.540 --> 03:19.019 The job of assigning a pod to a node is a scheduler's job. 03:19.099 --> 03:21.019 We have to look at the status of the scheduler. 03:21.099 --> 03:28.010 We know that a scheduler runs as a pod in the kube-system namespace. 03:28.090 --> 03:33.329 Here, you have the scheduler and it has control plane at the end. 03:33.409 --> 03:35.760 That means it's a static pod. 03:35.840 --> 03:38.269 Its status is CrashLoopBackOff. 03:38.349 --> 03:42.239 Let's take a look at the pod in more detail 03:42.319 --> 03:45.850 in the kube-system namespace. 03:45.930 --> 03:51.799 We're looking at the kube scheduler. 03:51.879 --> 03:56.030 We see at the bottom here that seems to be restarting quite often, 03:56.110 --> 03:59.339 and then it says failed to start container kube-scheduler, 03:59.419 --> 04:03.100 error response from daemon: OCI runtime create failed. 04:03.180 --> 04:09.379 Starting container process, kube-scheduler: executable file not found. 04:09.459 --> 04:12.230 It says this executable is not found. 04:12.310 --> 04:14.620 Obviously, it looks like it's not right. 04:14.700 --> 04:17.730 Let's take a look at the options or the command that are used 04:17.810 --> 04:19.170 to run this pod. 04:19.250 --> 04:24.980 Here, we can see that there is an incorrect command that's run, 04:25.060 --> 04:26.760 and that's the reason. 04:26.840 --> 04:29.870 We know that the kube scheduler is a static pod, 04:30.746 --> 04:37.476 so the file is in /etc/kubernetes/manifests, 04:37.556 --> 04:41.760 and we have a kube-scheduler.yaml file here. 04:41.840 --> 04:45.050 This is the manifest file for kube scheduler. 04:45.130 --> 04:46.550 Within that, there is a run command. 04:46.630 --> 04:49.757 Let's go and edit that file 04:50.875 --> 04:57.440 and then we remove the extra characters, 04:57.690 --> 05:02.880 and we're going to save that. 05:02.960 --> 05:05.977 We'll see the status of the pod, 05:07.252 --> 05:08.829 and it's in 05:09.101 --> 05:12.170 a CreatingContainerConfigError state. 05:12.710 --> 05:14.535 It's now in a running state, 05:14.615 --> 05:21.183 and we'll just watch until it's ready. 05:22.530 --> 05:28.620 Let's give it a few seconds. 05:28.700 --> 05:32.761 Still not ready. 05:37.120 --> 05:38.709 Let's take a look at the logs. 05:51.710 --> 05:53.540 It's finally in a ready state. 05:53.620 --> 05:57.360 Let's take a look at the pods, and we have the pod is now 05:57.440 --> 05:59.140 in a ready state. 05:59.220 --> 06:06.480 Let's take a look at the deployment, and we have one of one ready. 06:06.560 --> 06:07.480 That's good. 06:07.560 --> 06:09.140 Let's go to the next one. 06:09.220 --> 06:13.410 Now, it says scale the deployment app to two pods. 06:13.490 --> 06:14.970 Let's do a kubectl. 06:16.522 --> 06:18.275 Let's get deploy, 06:20.535 --> 06:22.360 and we have app with one pod. 06:22.440 --> 06:24.380 Let's scale it up. 06:24.460 --> 06:28.174 kubectl scale deployment app 06:28.254 --> 06:31.607 and replicas is two. 06:33.180 --> 06:34.135 There's two pods. 06:40.310 --> 06:42.030 Even though the deployment was scaled to two, 06:42.110 --> 06:44.280 the number of pods does not seem to increase. 06:44.360 --> 06:46.660 Investigate and fix the issue. 06:46.740 --> 06:49.380 Let's check the status of pods. 06:49.460 --> 06:53.870 Now, we see that it's still just one. 06:53.950 --> 06:58.880 We have the deployment here, it says one of two ready. 06:58.960 --> 07:01.100 Let's take a look at the deployment. 07:01.180 --> 07:04.217 kubectl describe deploy app, 07:05.326 --> 07:10.398 and we see the image set. 07:11.520 --> 07:16.020 We see scaled up replica set to one, but we don't see it, 07:16.100 --> 07:19.900 the deployment scaling up the replica set to two. 07:19.980 --> 07:23.240 However, here we see the number of replica to be two. 07:23.320 --> 07:24.910 Two is the desired state, 07:25.612 --> 07:29.674 and only one is available. 07:33.120 --> 07:38.730 The job of updating a deployment, the job of updating a replica set 07:38.810 --> 07:44.750 or scaling up a replica set is the controller managers. 07:44.830 --> 07:49.090 It's the deployment controller and all the controllers as we know, 07:49.170 --> 07:52.650 the deployment controller, the replica set controller, 07:52.730 --> 07:55.530 they're all controlled by the controller manager. 07:55.610 --> 07:58.000 Let's take a look at the status of the controller manager. 07:58.080 --> 08:03.790 We're going to do a kube-system, and we see that the kube-controller-manager 08:03.870 --> 08:06.020 is in a CrashLoopBackOff state. 08:06.100 --> 08:12.100 Let's take a look at the kube-controller-manager. 08:12.180 --> 08:14.789 Within the new kube-system, 08:15.778 --> 08:21.760 we're going to describe this particular pod. 08:21.840 --> 08:28.320 We see that it says Back-Off restarting failed container. 08:28.400 --> 08:32.899 We don't see anything useful here in the events. 08:32.979 --> 08:33.899 Let's take a look at it. 08:33.979 --> 08:36.439 Here it says, CrashLoopBackOff, Terminated, Error. 08:36.519 --> 08:38.003 The exit code is one, 08:39.316 --> 08:44.228 and the command seems to be correct. 08:44.897 --> 08:48.120 There isn't anything added there. 08:48.200 --> 08:52.620 Let's take a look at the logs. 08:52.700 --> 08:54.949 Let's do kubectl logs. 08:56.298 --> 09:00.554 This one in the kube-system namespace. 09:02.790 --> 09:07.590 Here it says, this particular file, there's no such file or directory. 09:07.670 --> 09:09.790 This file is not found. 09:09.870 --> 09:11.160 Let's check if that file exists. 09:11.240 --> 09:13.210 Of course, that file does not exist because there seems 09:13.290 --> 09:17.500 to be something wrong with it. 09:17.580 --> 09:22.010 Let's check where this file is specified in the kube-controller-manager. 09:22.090 --> 09:26.340 We know that the kube-controller-manager is also a static pod. 09:26.420 --> 09:27.700 That means it must be 09:28.282 --> 09:33.980 in the cat /etc/kubernetes/manifests directory, 09:34.060 --> 09:40.360 and then we have controller-manager there. 09:40.440 --> 09:41.360 Let's see. 09:41.440 --> 09:46.170 It's just a grep for XXX because that's what we saw. 09:46.250 --> 09:47.250 Here we have it specified. 09:47.330 --> 09:51.446 The kubeconfig file is configured to this location, 09:53.604 --> 09:54.470 but let's see 09:54.550 --> 10:00.240 where the actual kube-config file is. 10:00.320 --> 10:03.480 The actual kubeconfig file for the controller manager 10:03.560 --> 10:08.120 is at /etc/kubernetes/controller-manager.conf. 10:08.200 --> 10:11.720 Let's just verify the contents of that file. 10:11.800 --> 10:16.150 Let's do a controller-manager. 10:16.230 --> 10:20.030 We see that this is indeed the kubeconfig file. 10:20.110 --> 10:21.740 That's the problem. 10:21.820 --> 10:25.041 We're going to edit 10:25.121 --> 10:31.669 the controller-manager manifest file 10:32.800 --> 10:35.590 to fix that issue. 10:35.670 --> 10:38.934 We're going to remove this XXXX 10:39.014 --> 10:42.770 that is not required and save that, 10:42.850 --> 10:49.520 and we'll check the pod system. 10:49.600 --> 10:50.910 It's in a pending state. 10:50.990 --> 10:55.220 We're going to watch that, and we'll give you some time 10:55.300 --> 10:58.163 for the kube-controller-manager to be ready. 11:06.730 --> 11:09.950 It's now in a running state and it's also already. 11:10.030 --> 11:15.540 Let's take a look at the pods, and we see that there are now two pods indeed. 11:15.620 --> 11:19.720 Let's take a look at the deployments, and we see that there are two deployments. 11:19.800 --> 11:25.590 Let's check the status. 11:25.670 --> 11:27.139 We're breaking something now. 11:34.830 --> 11:36.000 Something is wrong with scaling again. 11:36.080 --> 11:39.870 We just tried scaling the deployment to three replicas, but it's not happening. 11:39.950 --> 11:40.890 Let's take a look at that. 11:40.970 --> 11:42.990 Let's do a get pods. 11:43.070 --> 11:44.620 It's only two. 11:44.700 --> 11:50.160 Let's do a get deployment and it's two of three. 11:50.240 --> 11:55.380 It looks like it's gone to three. 11:55.460 --> 11:57.550 Let's check. 11:57.630 --> 12:01.030 Let's look at the deployment. 12:01.110 --> 12:05.940 We see that here we have scaled up the replica set to one, then two, 12:06.020 --> 12:10.860 which we did earlier, but it has not been scaled up to three. 12:10.940 --> 12:17.410 In the previous one, we've learned that if the scaling up isn't happening, 12:17.490 --> 12:24.030 if the changes that we've applied on the deployment isn't going into effect, 12:24.110 --> 12:29.180 then the culprit is usually the controller manager 12:29.260 --> 12:31.570 that is responsible for this. 12:31.650 --> 12:36.780 Let's take a look at the controller manager again. 12:36.860 --> 12:43.699 We're going to look at the logs of this pod, 12:43.779 --> 12:44.699 controller manager. 12:44.779 --> 12:48.790 We're going to do a kubectl logs. 12:48.870 --> 12:51.840 This is the controlplane, controller-manager, 12:51.920 --> 12:56.150 and the kube-system namespace. 12:56.230 --> 13:01.660 We see that it says unable to load client CA file. 13:01.740 --> 13:04.230 The etc/kubernetes/pki/ca.crt is not available. 13:04.310 --> 13:08.080 Let's try and find this file locally. 13:08.160 --> 13:10.949 We see that we do have this file locally. 13:11.029 --> 13:15.880 Yes, this file seems to be here, but it says it's not able to load it. 13:15.960 --> 13:22.449 There's no such file or directory within the kube-controller-manager. 13:22.529 --> 13:24.880 We know that the way that the kube-controller-manager 13:24.960 --> 13:30.140 or any other controlplane component is set up is that the certificate files 13:30.220 --> 13:35.449 are on the controlplane, on the host, and then we use volumes 13:35.529 --> 13:41.530 to mount these directories within the same directory 13:41.610 --> 13:44.210 within the controller manager. 13:44.290 --> 13:45.920 That's set up in the manifest file. 13:46.000 --> 13:47.592 Let's take a look at the manifest file. 13:54.540 --> 13:59.000 We're going to look at kube-controller-manager. 13:59.080 --> 14:03.360 We see that all of these that are configured here, 14:03.440 --> 14:06.879 so etc/kubernetes and etc/kubernetes/pki. 14:06.959 --> 14:11.650 These are all in fact on the host, but they're mapped as a volume through 14:11.730 --> 14:12.650 to these containers. 14:12.730 --> 14:15.970 The way that it's done is through volumeMounts. 14:16.050 --> 14:18.340 We look at volumeMounts here. 14:18.420 --> 14:19.860 We can see there are multiple volumeMounts. 14:19.940 --> 14:25.020 You have /etc/ssl/certs, the CA certs, but what we want is the Kubernetes 14:25.100 --> 14:26.759 and pki a volume. 14:26.839 --> 14:30.210 This is what we are concerned about, because this is what it claims 14:30.290 --> 14:32.690 that is not available. 14:32.770 --> 14:37.000 Here, the mounting looks good, but the volume that is mounted 14:37.080 --> 14:39.030 is called k8s certs. 14:39.110 --> 14:43.920 Let's take a look at the k8s certs volume. 14:44.000 --> 14:47.390 These are the volumes that are configured here. 14:47.470 --> 14:53.160 Here we have the k8s certs volume, and this is the section for that. 14:53.240 --> 14:56.750 Within this, we have the path which is given at the top. 14:56.830 --> 15:00.580 Here, you can see that there's an incorrect path given. 15:00.660 --> 15:03.060 It says WRONG-PKI-DIRECTORY. 15:03.140 --> 15:08.330 We're going to have to get rid of that and add the right pki directory 15:08.410 --> 15:09.940 which is just pki. 15:10.020 --> 15:15.310 Just going to save that. 15:15.390 --> 15:19.600 Let's take a look at the pods' status now. 15:19.680 --> 15:24.870 It seems to be in a running state. 15:24.950 --> 15:28.552 Let's watch it and wait 15:28.632 --> 15:33.196 until it is back up and running. 15:37.330 --> 15:38.850 It's now in a running state. 15:38.930 --> 15:41.470 Let's take a look at the status of pods. 15:41.550 --> 15:44.660 We see that we have three pods indeed. 15:44.740 --> 15:48.000 If we take a look at the status of the deployment, 15:48.080 --> 15:50.420 we see that we have three out of three ready. 15:53.552 --> 15:55.100 That's the end of this lab.