WEBVTT 00:01.333 --> 00:04.673 Let's go through the lab on cluster maintenance. 00:05.513 --> 00:07.826 Let us first explore the environment. 00:08.153 --> 00:10.716 How many nodes do you see in the cluster? 00:11.633 --> 00:15.413 We will set the alias to kubectl. 00:17.333 --> 00:20.363 Let's do get nodes. 00:20.933 --> 00:25.487 We see that there are two nodes, the controlplane and node01. 00:25.567 --> 00:26.567 That's two. 00:30.233 --> 00:32.793 Now, how many applications do you see hosted on the cluster? 00:32.873 --> 00:34.863 Check the number of deployments. 00:34.943 --> 00:40.473 Let's do get deploy and we see that we have one deployment. 00:40.553 --> 00:41.753 That's just one. 00:45.203 --> 00:48.103 Now, which nodes are the applications hosted on? 00:48.293 --> 00:53.603 Let's do get pods and we see there are three pods. 00:53.733 --> 00:58.673 Let's use the wide option to see what nodes they're on. 00:58.763 --> 01:02.003 We see that they're on node01. 01:03.023 --> 01:04.473 All of them are on node01. 01:04.553 --> 01:06.233 That's the answer to this. 01:09.413 --> 01:12.773 Now we need to take node01 out for maintenance. 01:13.073 --> 01:16.733 Empty the node of all applications and mark it unscheduleable. 01:16.913 --> 01:21.723 We learned that the command to take a node down for maintenance 01:21.803 --> 01:26.033 is drain so we're going to do a drain node01. 01:27.473 --> 01:32.153 When we run that, it says it's not drained 01:32.513 --> 01:35.223 because you cannot delete DaemonSet-managed pods. 01:35.303 --> 01:38.043 There are DaemonSet-managed pods on this node. 01:38.123 --> 01:40.387 We could use the ignore-daemondet option. 01:40.467 --> 01:41.520 Let's do that. 01:41.600 --> 01:44.613 On the same command with the ignore DaemonSet option, 01:44.693 --> 01:50.453 and it's going to evict the pods for the blue deployment. 01:52.103 --> 01:53.213 That's done. 01:54.893 --> 01:56.603 Not fully. 01:57.203 --> 02:00.923 Let's wait for the third pod to be evicted. 02:03.713 --> 02:08.183 Okay, so that's done. 02:08.603 --> 02:11.543 Now what nodes are the apps on now? 02:11.843 --> 02:17.543 Let's take a look at the pods and see where they are. 02:17.633 --> 02:21.113 Now you can see that they're all on the controlplane node. 02:22.283 --> 02:25.083 If you look at the status of the node, 02:25.163 --> 02:29.723 we see that node01 is in a scheduling disabled state. 02:31.073 --> 02:34.613 To answer to this question, all of them are on controlplane. 02:38.903 --> 02:41.303 Applying patches on node01, hold on. 02:44.167 --> 02:46.792 The maintenance tasks have been completed. 02:47.333 --> 02:50.753 We now need to configure the node01 to be scheduled again. 02:53.213 --> 02:56.453 We saw that it's in a scheduling disabled state. 02:56.783 --> 02:59.403 To bring a node out of the scheduling disabled state 02:59.483 --> 03:01.233 and back into the cluster, 03:01.313 --> 03:07.733 we do uncordon and then we give this node name. 03:07.883 --> 03:09.563 The node is uncordoned. 03:09.653 --> 03:14.543 When you do get nodes, it's now back to its normal original state. 03:18.503 --> 03:22.223 The next question is how many pods are scheduled on node01 now? 03:22.433 --> 03:25.853 Let's run the get pods command again with the wide option, 03:27.083 --> 03:29.320 and we see that they're all still 03:29.400 --> 03:31.343 on the controlplane node. 03:31.883 --> 03:36.320 There are no pods currently on node01. 03:36.400 --> 03:38.267 That's 0. 03:40.333 --> 03:42.600 Why are there no pods on node01? 03:43.167 --> 03:45.287 Is it because node01 is faulty? No. 03:45.367 --> 03:47.992 Is it cordoned? No, we just uncordoned it. 03:48.473 --> 03:51.273 Then is it because it did not upgrade successfully? No. 03:51.353 --> 03:54.303 It's because only when new pods are created, they'll be scheduled. 03:54.383 --> 03:59.603 We have not deployed any new application after we uncordoned 04:00.263 --> 04:03.623 the node so we don't expect anything to be on that node. 04:03.833 --> 04:06.243 The pods that were moved to controlplane 04:06.323 --> 04:10.403 are not going to move back automatically to node01. 04:15.953 --> 04:20.393 Now, why are the pods placed on the controlplane node? 04:22.193 --> 04:24.693 Check the controlplane node for details. 04:25.943 --> 04:30.723 Usually, we don't expect applications to be deployed on the controlplane 04:30.803 --> 04:33.773 because it usually has a taint of some sort, 04:34.943 --> 04:40.100 but let's look at the node controlplane 04:41.267 --> 04:43.403 and see if it has any taints. 04:46.433 --> 04:48.433 We see that there are no taints. 04:51.023 --> 04:53.643 That's one of the reason that the pods 04:53.723 --> 04:56.223 were scheduled on the controlplane node. 04:57.473 --> 05:02.633 It's not because it's faulty or can never run pods on a master node. 05:03.713 --> 05:05.463 It does not have any taints. 05:07.100 --> 05:10.460 It does not have any taints so that's the correct answer. 05:12.593 --> 05:15.343 Usually, when you have a multi-node cluster, 05:16.013 --> 05:18.633 the master node or the controlplane node would have taints 05:18.713 --> 05:22.043 that would prevent pods from being placed on it. 05:22.133 --> 05:25.133 In this case, it doesn't, and that's why I have it. 05:25.973 --> 05:29.633 Now we're going to time travel to the next maintenance window. 05:33.593 --> 05:36.753 It's time to carry out the maintenance activity on node01, again, 05:36.833 --> 05:39.893 try draining the node again, using the same command. 05:40.073 --> 05:42.253 This is the same kubectl drain node command 05:42.333 --> 05:44.163 with the ignore demand says option. 05:44.243 --> 05:48.083 Let's run that again, but we see that there is an error. 05:48.653 --> 05:53.193 It says that cannot delete pods not managed by ReplicationController, 05:53.273 --> 05:57.503 ReplicaSet, Job, DaemonSet, or StatefulSet so use --force to override. 05:58.043 --> 06:00.333 Did that work, this command? 06:00.413 --> 06:01.538 No, it did not. 06:04.673 --> 06:07.233 Why did the drain command fail on node01? 06:07.313 --> 06:08.763 It worked the first time. 06:08.843 --> 06:10.533 There seems to be a change. 06:10.613 --> 06:13.373 Here it says there's a default hr-app. 06:14.093 --> 06:15.773 Let's take a look at that. 06:16.433 --> 06:18.903 We see that, indeed, there is a hr-app, 06:18.983 --> 06:23.613 which seems to be a new addition and which is a pod that has been added 06:23.693 --> 06:26.381 or an application that is deployed as a pod 06:26.663 --> 06:29.063 since the first maintenance window. 06:29.963 --> 06:32.633 Let's see where that is scheduled. 06:33.233 --> 06:36.353 We see that that one is scheduled on node01. 06:37.167 --> 06:38.733 If you look at the error message here, 06:38.813 --> 06:44.363 you cannot delete pods not managed by any of these, 06:44.813 --> 06:46.413 and when you do a drain. 06:46.493 --> 06:47.223 When you do a drain, 06:47.303 --> 06:51.413 what happens is it first cordons node and marks it unscheduleable, 06:51.803 --> 06:55.223 and then it tries to evict all the pods. 06:55.673 --> 06:58.713 Now it is easy to delete the pods that are managed 06:58.793 --> 07:01.668 by either one of these by a ReplicaSet, or Job, 07:01.763 --> 07:05.963 or ReplicationController because when you delete the pod, 07:06.893 --> 07:10.103 they take care of recreating the pod on another node. 07:11.993 --> 07:16.053 That's less risky, so as part of the draining process, 07:16.133 --> 07:17.973 you can just go ahead and kill the pods. 07:18.053 --> 07:21.663 Then the ReplicaSet or whatever it is that's managing the pod would take 07:21.743 --> 07:24.368 care of recreating them on the other node. 07:24.833 --> 07:27.753 In this case, it's a pod, just a pod. 07:27.833 --> 07:29.987 It's not one that is created as part 07:30.067 --> 07:32.817 of a deployment or ReplicaSet or any of this. 07:34.823 --> 07:38.393 If it deleted this pod as part of this training process, 07:38.783 --> 07:43.053 that pod and anything that part is created, any data that is created 07:43.133 --> 07:46.293 or stored locally on that pod is going to be lost forever. 07:46.373 --> 07:50.033 That's why you have this error message here. 07:50.783 --> 07:54.623 That's why it did not kill or terminate this pod. 07:55.367 --> 07:56.673 The answer to this question 07:56.753 --> 07:59.653 is that this is because there's a pod on node01, 07:59.843 --> 08:02.363 which is not part of a ReplicaSet 08:04.673 --> 08:06.233 or any of these. 08:11.000 --> 08:13.173 What is the name of the pod hosted on node01 08:13.253 --> 08:14.730 that is not part of the ReplicaSet? 08:14.810 --> 08:16.073 It's hr-app. 08:20.963 --> 08:23.193 Now what would happen to hr-app if node01 08:23.273 --> 08:24.753 is drained forcefully? 08:24.833 --> 08:27.573 There is an option to drain the node, as you can see here, 08:27.653 --> 08:29.313 using the force option. 08:29.393 --> 08:32.003 Let's try that out, --force. 08:32.813 --> 08:35.763 What's going to happen is that this pod is going to be deleted 08:35.843 --> 08:38.213 and it's going to be gone forever. 08:39.083 --> 08:42.383 There's nothing here that's going to move 08:42.683 --> 08:45.683 or reschedule this pod on any other worker node. 08:46.103 --> 08:49.883 That's mainly the reason why it's preventing us from doing that. 08:51.563 --> 08:55.343 The answer to this question is that hr-app will be lost forever. 09:01.313 --> 09:03.093 Now we did not want to do that because hr-app 09:03.173 --> 09:06.183 is a critical application that should not be destroyed. 09:06.263 --> 09:08.103 We have now reverted back to the previous state 09:08.183 --> 09:10.683 and redeploy the hr-app as a deployment. 09:11.363 --> 09:15.633 Now, you look at the deployments, 09:15.713 --> 09:19.223 we see that the hr-app is now deployed as a deployment. 09:19.583 --> 09:21.773 If you look at the pods, 09:22.613 --> 09:26.433 we see that the hr-app is indeed deployed as a deployment. 09:26.513 --> 09:30.833 This is the ReplicationController, and then you have the deployment here. 09:32.873 --> 09:34.936 This is also scheduled on node01. 09:37.913 --> 09:41.073 Now hr-app is a critical app and we don't want it to be removed 09:41.153 --> 09:44.613 and we also do not want to schedule any more parts on node01. 09:44.693 --> 09:46.683 Currently, the hr-app is on node01, 09:46.763 --> 09:49.633 and we don't want to schedule anything more 09:49.967 --> 09:55.373 on node01 because apparently, it's a critical app. 09:55.920 --> 09:59.003 We need to mark node01 as unscheduleable 09:59.093 --> 10:01.353 so that no new pods are scheduled on the node, 10:01.433 --> 10:05.093 but we also have to make sure that the hr-app is not affected. 10:05.633 --> 10:07.313 What we want to do is-- 10:07.673 --> 10:10.983 Basically, what we're saying is we have to currently the node, 10:11.063 --> 10:12.987 if a new application is deployed, 10:13.067 --> 10:16.113 it's going to be scheduled here because it's in a ready state 10:16.193 --> 10:18.881 and we want to prevent that from happening. 10:19.073 --> 10:22.263 At the same time, we also don't want to kill this app by doing a drain. 10:22.343 --> 10:24.531 The other option is to just cordon. 10:25.043 --> 10:28.913 We're going to do kubectl cordon node01. 10:29.633 --> 10:34.763 All that's going to do is mark the node as SchedulingDisabled 10:35.903 --> 10:37.478 but at the same time, 10:38.200 --> 10:43.103 the hr-application is just going to be fine. 10:45.833 --> 10:48.173 Okay, so that's the end of this lab.