WEBVTT 00:00.510 --> 00:03.590 Hello and welcome to this lecture. In this lecture 00:03.600 --> 00:07.440 we'll talk about ETCD in a high availability setup. 00:07.440 --> 00:13.020 So this is really a prerequisite lecture for the next lecture where we talk about configuring Kubernetes 00:13.140 --> 00:15.720 in a highly available mode. 00:15.720 --> 00:21.130 Well one portion of that deals with configuring ETCD in HA mode. 00:21.150 --> 00:28.440 So in this lecture we will discuss about ETCD in HA mode in the beginning of this course we took 00:28.440 --> 00:30.510 a quick look at ETCD. 00:30.570 --> 00:36.710 We will now recap real quick and more importantly focus on the cluster configuration on ETCD. 00:36.810 --> 00:41.970 So let’s recap real quick and look at the number of nodes in the cluster, what RAFT protocol is etc. 00:42.870 --> 00:43.980 So what is ETCD? 00:44.070 --> 00:48.580 It's a distributed reliable key value store that is simple secure and fast. 00:48.600 --> 00:49.540 So let's break it up. 00:49.560 --> 00:53.820 Traditionally data was organized and stored in tables like this. 00:53.820 --> 01:00.030 For example to store details about a number of individuals. A key value store stores information in the 01:00.030 --> 01:02.000 form of documents or pages. 01:02.130 --> 01:08.270 So each individual gets a document and all information about that individual is stored within that file. 01:08.370 --> 01:15.090 These files can be in any format or structure and changes to one file does not affect the others. 01:15.090 --> 01:21.470 In this case the working individuals can have their files with salary fields. While you could store and 01:21.470 --> 01:25.540 retrieve simple key and values when your data gets complex. 01:25.550 --> 01:30.090 you typically end up transacting in data formats like JSON or YAML. 01:30.830 --> 01:34.640 So that’s what etcd is and how you quickly get started with it. 01:34.640 --> 01:37.880 We also said that ETC is distributed. 01:37.880 --> 01:38.810 So what does that mean. 01:39.680 --> 01:43.280 And that is what we're going to focus in this lecture. 01:43.280 --> 01:49.340 We had ETCD on a single server. But it’s a database and may be storing critical data. 01:49.340 --> 01:54.090 So it is possible to have your datastore across multiple servers. 01:54.140 --> 02:00.890 Now you have 3 servers, all running etcd, and all maintaining an identical copy of the database. 02:00.890 --> 02:04.470 So if you lose one you still have two copies of your data. 02:04.580 --> 02:05.370 Perfect. 02:05.660 --> 02:09.530 But how does it ensure the data on all the nodes are consistent. 02:10.220 --> 02:18.470 You can write to any instance and read your data from any instance. ETCD ensures that the same consistent 02:18.470 --> 02:23.800 copy of the data is available on all instances at the same time. 02:23.810 --> 02:26.900 So how does it do that? With reads, 02:26.900 --> 02:30.640 its easy. Since the same data is available across all nodes, 02:30.710 --> 02:35.590 you can easily read it from any nodes. But that is not the case with writes. 02:35.600 --> 02:41.180 What if two write requests come in on two different instances? Which one goes through? 02:41.240 --> 02:49.140 For example I have writes coming in for name set to john on one and with the name joe on the other. 02:49.250 --> 02:53.190 Of course we cannot have two different data on two different nodes. 02:53.510 --> 02:58.890 When I said ETCD can write through any instance, I wasn’t a 100% right. 02:58.900 --> 03:01.760 ETCD does not process the writes on each node. 03:01.910 --> 03:08.600 Instead, only one of the instances is responsible for processing the writes. Internally 03:08.600 --> 03:15.620 the two nodes elect a leader among them. Of the total instances one node becomes the leader and the other 03:15.620 --> 03:17.870 node becomes the followers. 03:17.870 --> 03:23.540 If the writes came in through the leader node, then the leader processes the write. The leader makes sure 03:23.870 --> 03:27.410 that the other nodes are sent a copy of the data. 03:27.530 --> 03:33.200 If the writes come in through any of the other follower nodes, then they forward the writes to the leader 03:33.260 --> 03:36.860 internally and then the leader processes the writes. Again 03:36.890 --> 03:43.640 when the writes are processed, the leader ensures that copies of the write and distributed to other instances 03:43.790 --> 03:45.270 in the cluster. 03:45.350 --> 03:53.630 Thus a write is only considered complete, if the leader gets consent from other members in the cluster. 03:53.630 --> 04:00.050 So how do they elect the leader among themselves? And how do they ensure a write is propogated across 04:00.140 --> 04:06.830 all instances? ETCD implements distributed consensus using RAFT protocol. 04:06.830 --> 04:09.560 Let's see how that works in a three node cluster. 04:09.790 --> 04:16.100 When the cluster is setup we have 3 nodes that do not have a leader elected. RAFT algorithm uses 04:16.130 --> 04:19.030 random timers for initiating requests. 04:19.190 --> 04:25.790 For example a random timer is kicked off on the three managers the first one to finish the timer sends 04:25.790 --> 04:32.360 out a request to the other node requesting permission to be the leader. The other managers on receiving 04:32.360 --> 04:39.140 the request responds with their vote and the node assumes the Leader role. Now that it is elected the 04:39.150 --> 04:46.020 leader it sends out notification at regular intervals to other masters informing them that it is continuing 04:46.020 --> 04:53.560 to assume the role of the leader. In case the other nodes do not receive a notification from the leader 04:53.580 --> 04:59.370 at some point in time which could either be due to the leader going down or losing network connectivity 04:59.880 --> 05:07.320 the nodes initiate a re-election process among themselves and a new leader is identified going back 05:07.320 --> 05:13.650 to our previous example where a right come seen it is processed by the leader and is replicated to other 05:13.650 --> 05:20.280 nodes in the cluster the right is considered to be complete only once it is replicated to the other 05:20.280 --> 05:22.500 instances in the cluster. 05:22.500 --> 05:29.190 We said that the ETCD cluster is highly available. So even if we lose a node it should still function. 05:29.250 --> 05:35.550 Say for example a new write comes in but one of the node is not responding. And hence the leader is only 05:35.550 --> 05:41.400 able to write to 2 nodes in the cluster. Is the write considered to be complete? Does it wait for the 05:41.400 --> 05:46.390 third node to be up? Or does it fail? A write is considered to be complete, 05:46.500 --> 05:53.100 if it can be written on the majority of the nodes in the cluster. For example, in this case of the 3 nodes, 05:53.250 --> 05:54.790 the majority is 2. 05:54.840 --> 06:00.550 So if the data can be written on two of the nodes then the right is considered to be complete. 06:00.570 --> 06:05.430 If the third node was to come on line then the data is copied to that as well. 06:05.490 --> 06:07.700 So what is the majority. 06:07.710 --> 06:15.000 Well a more appropriate term to use would be Quorum. Quorum is the minimum number of nodes that must 06:15.000 --> 06:21.720 be available for the cluster to function properly or make a successful right in case of 3. 06:21.720 --> 06:30.460 we know its 2. For any given number of nodes, the quorum is the total number of nodes divided by 2 + 1. 06:30.540 --> 06:36.960 So Quorum of 3 nodes is 3/2 which is 1.5 + 1 equals 2.5. 06:36.960 --> 06:42.280 If there is a .5 consider the whole number only so that's 2. 06:42.390 --> 06:45.310 Similarly quorum of 5 nodes is 3. 06:45.360 --> 06:53.000 So here is a table that shows the quorum of clusters of size 1 to 7. Quorum of 3 and 5 are what 06:53.000 --> 06:56.630 we calculated. Quorum of 1 is 1 itself. 06:56.630 --> 07:00.110 Meaning if you have a single non cluster none of these really apply. 07:00.110 --> 07:05.950 If you lose that note everything's gone if you look at 2 and apply the same formula. 07:06.120 --> 07:10.440 the quorum is 2 itself. 2/2 is 1 and 1 + 1 is 2. 07:10.890 --> 07:17.610 So even if you have 2 instances in the cluster the majority is still 2. If one fails, 07:17.610 --> 07:18.530 There is no quorum. 07:18.540 --> 07:20.540 So rates won't be processed. 07:20.700 --> 07:27.570 So having two instances is like having one instance it doesn't offer you any real value as quorum cannot 07:27.570 --> 07:28.740 be met. 07:28.740 --> 07:35.190 Which is why it is recommended to have a minimum of 3 instances in an ETCD cluster. That way 07:35.220 --> 07:38.380 it offers a fault tolerance of at least 1 node. 07:39.240 --> 07:43.920 If you lose one, you can still have quorum and the cluster will continue to function. 07:44.730 --> 07:50.220 So the first column minus the second column gives you the fault tolerance. The number of nodes that you 07:50.220 --> 07:53.620 can afford to lose while keeping the cluster alive. 07:53.700 --> 07:58.050 So we have 1 to 7 nodes here. 1 and 2 are out of consideration. 07:58.050 --> 08:01.540 So from 3 to 7 what do we consider? 08:01.770 --> 08:07.350 As you can see 3 and 4 have the same fault tolerance of 1 and 5 and 6 have the same fault 08:07.350 --> 08:14.850 tolerance of 2. When deciding on the number of master nodes,it is recommended to select an odd number 08:14.850 --> 08:21.010 as highlighted in the table 3 or 5 or 7. Say we have a 6 node cluster. 08:21.570 --> 08:28.420 So for example due to a disruption in the network it fails and causes the network to partition. 08:28.440 --> 08:31.770 We have 4 nodes on one and 2 on the other. 08:31.770 --> 08:38.980 In this case the group with 4 nodes have quorum and continues to operate normally. However if the 08:38.980 --> 08:45.530 network got partitioned in a different way resulting in nodes being distributed equally between the two, 08:45.610 --> 08:47.860 each group now has 3 nodes only. 08:47.980 --> 08:54.650 But since we originally had 6 manager nodes, the quorum for the cluster to stay alive is 4 08:54.880 --> 08:59.770 But if you look at the groups here neither of these groups have four managers to meet the quorum so 08:59.770 --> 09:01.580 it results in a failed cluster. 09:02.410 --> 09:09.220 So with even number of nodes there is possibility of the cluster failing during a network segmentation. 09:10.210 --> 09:14.190 In case we had odd number of managers originally, say 7, 09:14.260 --> 09:20.390 then after the network segmentation we have 4 on one segmented network and 3 on the other, 09:20.500 --> 09:26.380 and so our cluster still lives on the group with 4 managers as it meets the quorum of 4. 09:26.710 --> 09:33.010 No matter how the network segments there are better chances for your cluster to stay alive in case of 09:33.010 --> 09:35.950 network segmentation with odd number of nodes. 09:36.580 --> 09:44.290 So an odd number of nodes is preferred over even number having 5 is preferred over 6 and having 09:44.290 --> 09:51.750 more than 5 nodes is really not necessary as 5 gives you enough for tolerance. To install ETCD on 09:51.750 --> 09:52.420 a server. 09:52.530 --> 09:55.380 download the latest supported binary. Extract it, 09:55.410 --> 10:02.840 create the required directory structure. Copy over the certificate files generated for ETCD. We 10:02.840 --> 10:08.630 discussed how to generate these certificates in detail in the TLS Certificates section. Then configure 10:08.630 --> 10:10.490 the ETCD service. 10:10.490 --> 10:16.850 What's important here is to note that the initial cluster option where we passing the peer's information 10:17.270 --> 10:24.320 That’s how each etcd service knows that it is part of a cluster and where its peers are. Once installed 10:24.380 --> 10:31.220 and configured use the etcdctl utility to store and retrieve data. ETCDCTL utility has 10:31.220 --> 10:32.810 two API versions. 10:32.810 --> 10:36.850 V2 and V3. So the commands work different in each version. 10:37.010 --> 10:38.540 Version 2 is default. 10:38.870 --> 10:42.680 However we will use version 3. So set an environment variable 10:42.680 --> 10:47.970 ETCDCTL_API to 3, otherwise the below commands won’t work. 10:48.000 --> 10:55.680 Run the etcdctl put command and specify the key as name and value as john. To retrieve data from 10:55.720 --> 10:56.740 the etcdctl 10:56.740 --> 11:04.290 get command with the key /name and it returns the value john.To get all keys run the etcdctl 11:04.330 --> 11:05.650 get –keys-only 11:05.780 --> 11:10.040 command. Going back to our design, 11:10.110 --> 11:14.460 how many nodes, should our cluster have? In an HA environment 11:14.460 --> 11:20.370 as you can see having 1 or 2 instances doesn’t really make any sense. As losing 1 node, in either 11:20.370 --> 11:26.220 case will leave you without quorum and thus render the cluster not functional. Hence the minimum required 11:26.250 --> 11:29.330 nodes in an HA setup is 3. 11:29.340 --> 11:34.980 We also discussed why we prefer odd number of instances over even number. Having even number of instances 11:34.980 --> 11:40.250 can leave the cluster without quorum in certain network partition scenarios. 11:40.500 --> 11:47.340 So all that even number of nodes is out of scope. So we are left with 3 5 and 7 or any odd number above 11:47.340 --> 11:53.820 that 3 is a good start but if you prefer a higher level of full tolerance then 5 is better. But anything 11:53.820 --> 11:56.270 beyond that is just unnecessary. 11:56.310 --> 12:01.730 So considering your environment the fault tolerance requirements and the cost that you can bear you 12:01.740 --> 12:04.450 should be able to choose one number from this list. 12:04.530 --> 12:06.160 In our case we go with 3. 12:07.080 --> 12:10.200 So how does our design look now? With HA, 12:10.200 --> 12:13.440 the minimum required number of nodes for fault tolerance is 3. 12:13.890 --> 12:21.060 Now while it would be great to have 3 master nodes we are limited by our capacity of our laptop. So we 12:21.060 --> 12:22.190 will just go with 2. 12:22.260 --> 12:27.330 But if you're deploying the setup in another environment and have sufficient capacity feel free to go 12:27.330 --> 12:28.390 with 3. 12:28.680 --> 12:35.130 We also chose to go with the stacked topology where we will have the ETCD servers on the master nodes 12:35.220 --> 12:36.390 itself. 12:36.390 --> 12:37.830 Well that's it for this lecture. 12:38.340 --> 12:39.410 Thank you for listening. 12:39.450 --> 12:40.950 I will see you in the next.