WEBVTT 00:00.570 --> 00:06.650 Hello and welcome to this lecture. We now look at high availability in kubernetes. 00:06.680 --> 00:11.830 So what happens when you lose the master node in your cluster? 00:12.130 --> 00:18.150 As long as the workers are up and containers are alive, your applications are still running. 00:18.160 --> 00:23.000 Users can access the application until things start to fail. 00:23.020 --> 00:26.420 For example a container or POD on the worker node crashes. 00:26.770 --> 00:32.710 Now, if that pod was part of a replicaset then the replication controller on the master needs to instruct 00:32.710 --> 00:35.150 the worker to load a new pod. 00:35.350 --> 00:40.590 But the Master is not available and so are the controllers and schedulers on the master. 00:40.600 --> 00:44.490 There is no one to recreate the pod and no one to schedule it on nodes. 00:44.530 --> 00:50.260 Similarly, since the kube-api server is not available you cannot access the cluster externally through 00:50.260 --> 00:58.030 kubectl tool or through API for management purposes. Which is why you must consider multiple master nodes in 00:58.030 --> 01:04.510 in a High availability configuration in your production environment. A high availability configuration is 01:04.510 --> 01:10.780 where you have redundancy across every component in the cluster so as to avoid a single point of failure. 01:11.650 --> 01:18.100 The master nodes, the worker nodes, the control plane components, the application of course, which we already 01:18.100 --> 01:21.820 have multiple copies in the form of replicasets and services. 01:21.880 --> 01:28.490 So our focus in this lecture is going to be on the master and control plane components. Let's take a 01:28.490 --> 01:30.390 better look at how it works. 01:30.440 --> 01:34.970 We’ve been looking at a 3 node cluster with 1 master and 2 worker nodes 01:34.970 --> 01:41.840 throughout this course. In this lecture, we will focus on just the master node. As we learned already the 01:41.840 --> 01:48.270 master node hosts the control plane components including the API, Controller Manager, Scheduler and ETCD 01:48.280 --> 01:55.640 server. In a HA setup, with an additional master node, you have the same components running on 01:55.640 --> 01:57.340 the new master as well. 01:57.530 --> 01:59.060 So how does that work. 01:59.090 --> 02:04.430 Running multiple instances of the same components? Are they going to do the same thing twice? 02:04.430 --> 02:07.100 How do they share the work among themselves? 02:07.160 --> 02:10.000 Well that differs based on what they do. 02:10.070 --> 02:16.310 We know that the API server is responsible for receiving requests and processing them or providing information 02:16.370 --> 02:20.650 about the cluster. They work on one request at a time. 02:20.660 --> 02:27.650 So the API servers on all cluster nodes can be alive and running at the same time in an active active 02:27.650 --> 02:30.460 mode. So far in this course 02:30.490 --> 02:35.220 we know that the kubectl utility talks to the API server to get things done 02:35.320 --> 02:42.160 and we point the kubectl utility to reach the master node at port 6443. 02:42.160 --> 02:47.220 That’s where the API server listens and this is configured in the kube-config file. 02:47.260 --> 02:51.740 Well now with 2 masters, where do we point the kubectl to? 02:52.180 --> 02:56.800 We can send a request to either one of them but we shouldn't be sending the same request to both of 02:56.800 --> 02:57.460 them. 02:57.460 --> 03:03.520 So it is better to have a load balancer of some kind configured in the front of the master nodes that 03:03.520 --> 03:07.030 split traffic between the API servers. 03:07.030 --> 03:11.710 So We then point the kubectl utility to that load balancer. 03:11.710 --> 03:18.550 You may use, NGINX or HA proxy or any other load balancer for this purpose. 03:18.620 --> 03:21.490 What about the scheduler and the controller manager. 03:21.560 --> 03:27.220 These are controllers that watch the State of the cluster and take actions. For example the controller 03:27.230 --> 03:32.600 manager consists of controllers like the replication controller that is constantly watching the state 03:32.600 --> 03:36.400 of PODs and taking necessary actions, like creating a new POD 03:36.410 --> 03:44.120 when one fails. If multiple instances of those run in parallel, then they might duplicate actions 03:44.120 --> 03:47.470 resulting in more parts than actually needed. 03:47.510 --> 03:49.700 The same is true with scheduler. 03:49.700 --> 03:53.180 As such they must not run in parallel. 03:53.180 --> 03:56.240 They run in an active standby mode. 03:56.240 --> 04:01.570 So then who decides which among the two is active and which is passive. 04:01.610 --> 04:05.210 This is achieved through a leader election process. 04:05.360 --> 04:06.770 So how does that work. 04:06.770 --> 04:11.880 Let's look at controller manager for instance. When a controller manager process is configured. 04:12.050 --> 04:16.940 You may specify the leader elect option which is by default set to true. 04:16.940 --> 04:23.390 With this option when the controller manager process starts it tries to gain a lease or a lock on an 04:23.520 --> 04:30.500 endpoint object in kubernetes named as kube-controller-manager endpoint. Which ever process first 04:30.620 --> 04:31.760 updates the endpoint. 04:31.760 --> 04:36.250 With this information gains the lease and becomes the active of the two. 04:36.470 --> 04:42.890 The other becomes passive it holds the lock for the lease duration specified using the leader-elect-lease- 04:42.890 --> 04:46.930 duration option which is by default set to 15 seconds. 04:47.180 --> 04:53.030 The active process then renews the lease every 10 seconds which is the default value for the option 04:53.420 --> 05:00.790 leader-elect-renew-deadline. Both the processes tries to become the leader every two seconds, set by 05:00.790 --> 05:03.460 the leader-elect-retry-period option. 05:03.460 --> 05:09.640 That way if one process fails maybe because the first must of crashes then the second process can acquire 05:09.640 --> 05:15.250 the lock and become the leader. The scheduler follows a similar approach and has the same command line 05:15.250 --> 05:19.280 options. Next up, ETCD. 05:19.350 --> 05:22.020 We discussed about ETCD earlier in this course. 05:22.380 --> 05:27.330 It's a good idea to go through that again now just to quickly refresh your memory as we're going to 05:27.330 --> 05:33.270 discuss some topics related to how ETCD works in this lecture. With ETCD, 05:33.270 --> 05:36.750 there are two topologies that you can configure in kubernetes. 05:36.960 --> 05:42.390 One is as it looks here and the same architecture we have been following through out this course, where 05:42.420 --> 05:49.800 ETCD is part of the kubernetes master nodes. It’s called as a stacked control plan nodes topology. 05:49.830 --> 05:56.440 This is easier to setup and easier to manage. And requires fewer nodes. But if one node goes down 05:56.500 --> 06:03.730 both an ETCD member and control plane instance is lost and redundancy is compromised. The other is 06:03.730 --> 06:11.950 where ETCD is separated from the control plane nodes and run on its own set of servers. This is a topology 06:11.980 --> 06:15.870 with external ETCD servers. Compared to the previous topology 06:15.880 --> 06:22.870 this is less risky as a failed control plane node does not impact the ETCD cluster and the data it stores. 06:23.110 --> 06:29.680 However it is harder to setup and requires twice the number of servers for the external etcd nodes. So 06:29.680 --> 06:35.590 remember, the API server is the only component that talks to the ETCD server and if you look into the 06:35.590 --> 06:42.070 API service configuration options, we have a set of options specifying where the ETCD server is. 06:42.100 --> 06:47.980 So regardless of the topology we use and wherever we configure ETCD servers, weather on the same server 06:48.010 --> 06:49.050 or on a separate server. 06:49.210 --> 06:56.810 ultimately we need to make sure that the API server is pointing to the right address of the ETCD servers. 06:56.820 --> 07:03.280 Now remember ETCD is a distributed system, so the API server or any other component that wishes to talk 07:03.280 --> 07:07.510 to it, can reach the ETCD server at any of its instances. 07:07.510 --> 07:12.210 You can read and write data through any of the available ETCD server instances. 07:12.220 --> 07:19.270 This is why we specify a list of etcd-servers in the kube-apiserver configuration. In the next lecture 07:19.270 --> 07:25.090 we discuss more about how ETCD server works in a cluster setup and the best practices around the number 07:25.090 --> 07:28.360 of recommended nodes in a cluster. 07:28.360 --> 07:32.830 So back to our design we had originally planned for a single master node in our cluster. 07:32.950 --> 07:38.450 Now with HA, we decided to configure multiple masters. 07:38.460 --> 07:44.160 We also mentioned about a load balancer for the API server. So we will have that as well. 07:44.160 --> 07:47.400 So we now have a total of five nodes in our cluster.