WEBVTT

00:00.260 --> 00:04.260
[music]

00:04.740 --> 00:09.050
Hello, and welcome to this lecture and
we are learning advanced Docker concepts.

00:09.590 --> 00:14.150
In this lecture, we're going to talk about
Dockers storage drivers, and file systems.

00:14.930 --> 00:19.670
We're going to see where and how
Docker stores data and how it manages

00:19.670 --> 00:21.560
file systems of the containers.

00:22.940 --> 00:27.320
Let's just start with how Docker
stores data on the local file system.

00:28.320 --> 00:32.100
When you install Docker on a
system, it creates this folder

00:32.100 --> 00:35.610
structure at var/lib/docker.

00:36.150 --> 00:41.970
You have multiple folders under it called
aufs, containers, image, volumes, et cetera.

00:42.660 --> 00:46.200
This is where Docker stores
all its data by default.

00:46.530 --> 00:51.030
When I say data, I mean files
related to images and containers

00:51.060 --> 00:52.410
running on the Docker host.

00:53.160 --> 00:57.260
For example, all files related
to containers are stored under

00:57.260 --> 01:00.810
the containers folder and the
files related to images are

01:00.810 --> 01:02.550
stored under the image folder.

01:03.260 --> 01:06.120
Any volumes created by the
Docker containers are created

01:06.150 --> 01:07.470
under the volumes folder.

01:08.250 --> 01:09.870
Well, don't worry about that for now.

01:10.020 --> 01:11.970
We will come back to that in a bit.

01:12.630 --> 01:18.530
For now, let's just understand where
Docker stores its files and in what format.

01:19.960 --> 01:24.220
So, how exactly does Docker store the
files of an image and a container.

01:24.820 --> 01:29.250
To understand that we need to
understand Dockers layered architecture.

01:30.090 --> 01:33.240
Let's quickly recap something
we learned when Docker builds

01:33.300 --> 01:36.690
images, it builds these
in a layered architecture.

01:37.440 --> 01:41.880
Each line of instruction in the
Docker file creates a new layer

01:41.910 --> 01:45.750
in the Docker image with just the
changes from the previous layer.

01:46.270 --> 01:50.470
For example, the first layer is
a base Ubuntu operating system

01:50.920 --> 01:55.150
followed by the second instruction
that creates a second layer, which

01:55.150 --> 01:59.680
installs all the apt packages and
then the third instruction creates

01:59.770 --> 02:03.970
a third layer, which- uh, with
the Python packages followed by

02:03.970 --> 02:08.020
the fourth layer that copies the
source code over and then finally

02:08.020 --> 02:11.950
the fifth layer that updates
the entry point of the image.

02:13.990 --> 02:18.940
Since each layer only stores the
changes from the previous layer it

02:18.940 --> 02:21.130
is reflected in the size as well.

02:21.670 --> 02:27.040
If you look at the base Ubuntu image,
it is around 120 megabytes in size.

02:27.520 --> 02:31.990
The apt packages that are installed
is around 300 MB and then the

02:31.990 --> 02:33.670
remaining layers are small.

02:34.870 --> 02:39.370
To understand the advantages of
this layered architecture let's

02:39.370 --> 02:41.380
consider a second application.

02:42.370 --> 02:46.180
This application has a different
Docker file, but it's very

02:46.180 --> 02:48.070
similar to our first application.

02:48.610 --> 02:53.550
As in it uses just the same base
image as Ubuntu uses the same Python

02:53.550 --> 02:58.620
and Flask dependencies, but uses
a different source code to create

02:58.680 --> 03:02.940
a different application, and so
at different entry point as well.

03:03.600 --> 03:06.900
When I run the Docker build
command to build a new image

03:06.930 --> 03:10.980
for this application, since
the first three layers of both

03:10.980 --> 03:12.870
the applications are the same.

03:13.230 --> 03:16.800
Docker is not going to build
the first three layers.

03:17.310 --> 03:21.810
Instead it reuses the same three
layers it built for the first

03:21.810 --> 03:27.060
application from the cash and only
creates the last two layers with the

03:27.060 --> 03:29.730
new sources and the new entry point.

03:30.660 --> 03:36.260
This way, Docker builds images faster
and efficiently saves this space.

03:37.000 --> 03:40.840
This is also applicable, if you were
to update your application code.

03:41.650 --> 03:46.390
Whenever you update your application
code, such as the app.py in this

03:46.390 --> 03:51.700
case, Docker simply reuses all
the previous layers from cache and

03:51.700 --> 03:56.200
quickly rebuild the application
image by updating the latest source

03:56.200 --> 04:01.690
code, thus, saving us a lot of
time during rebuilds and updates.

04:03.990 --> 04:08.070
Let's rearrange the layers bottom
up so we can understand it better.

04:08.700 --> 04:12.720
At the bottom, we have the base
Ubuntu layer, then the packages,

04:13.200 --> 04:17.490
then the dependencies, and then
the source code of the application

04:17.880 --> 04:19.110
and then the entry point.

04:20.610 --> 04:24.930
All of these layers are created
when we run the Docker build command

04:25.140 --> 04:27.450
to form the final Docker image.

04:28.200 --> 04:31.350
So, all of these are
the Docker image layers.

04:32.100 --> 04:35.340
Once the build is complete, you
cannot modify the contents of

04:35.340 --> 04:39.270
these layers and so they are
read-only, and you can only modify

04:39.270 --> 04:41.610
them by initiating a new build.

04:42.600 --> 04:46.320
When you run a container based
off of this image, using the

04:46.320 --> 04:50.670
Docker run command Docker creates
a container based off of these

04:50.670 --> 04:55.230
layers and creates a new writeable
layer on top of the image layer.

04:55.870 --> 05:00.340
The writeable layer is used to
store data created by the container

05:00.610 --> 05:04.180
such as log files written by
the applications, any temporary

05:04.180 --> 05:08.380
files generated by the container
or just any file modified by

05:08.380 --> 05:10.210
the user on that container.

05:11.170 --> 05:15.730
The life of this layer though is only
as long as the container is alive.

05:16.210 --> 05:20.020
When the container is destroyed,
this layer and all of the changes

05:20.050 --> 05:22.090
stored in it are also destroyed.

05:22.990 --> 05:27.370
Remember that the same image
layer is shared by all containers

05:27.370 --> 05:29.230
created using this image.

05:30.940 --> 05:35.080
If I were to log into the newly
created container and say, create

05:35.170 --> 05:41.050
a new file called temp.txt, it will
create that file in the container

05:41.050 --> 05:42.760
layer, which is read and write.

05:43.720 --> 05:47.980
We just said that the files in the
image layer are read-only meaning you

05:48.010 --> 05:50.740
cannot edit anything in those layers.

05:51.630 --> 05:53.900
Let's take an example
of our application code.

05:54.480 --> 05:58.980
Since we bake our code into the
image, the code is part of the image

05:59.010 --> 06:00.990
layer and as such, it's read-only.

06:01.890 --> 06:05.550
After running a container, what
if I wish to modify the source

06:05.550 --> 06:07.740
code to say, test a change.

06:08.670 --> 06:12.750
Remember the same image layer
may be shared between multiple

06:12.750 --> 06:14.640
containers created from this image.

06:15.390 --> 06:19.620
So, does it mean that I cannot modify
this file inside the container?

06:20.210 --> 06:20.580
No.

06:21.290 --> 06:25.440
I can still modify this file,
but before I saved the modified

06:25.440 --> 06:29.390
file, Docker automatically
creates a copy of the file in the

06:29.390 --> 06:33.320
read-write layer and I will then
be modifying a different version of

06:33.320 --> 06:35.450
the file in the read/write layer.

06:36.320 --> 06:39.650
All future modifications will
be done on this copy of the

06:39.650 --> 06:41.590
file in the read-write layer.

06:42.100 --> 06:44.390
This is called
copy-on-right mechanism.

06:45.180 --> 06:48.530
The image layer being read
only just means that the files

06:48.570 --> 06:52.380
in these layers will not be
modified in the image itself.

06:52.710 --> 06:57.120
So, the image will remain the same
all the time until you rebuild the

06:57.120 --> 06:59.900
image using the Docker build command.

07:01.530 --> 07:03.690
What happens when we get
rid of the container?

07:04.440 --> 07:07.220
All of the data that was
stored in the container

07:07.220 --> 07:09.060
layer also gets deleted.

07:09.980 --> 07:14.030
The change we made to the
app.py and the new temp file we

07:14.030 --> 07:16.160
created will also get removed.

07:17.520 --> 07:19.790
So, what if we wish
to persist this data?

07:20.090 --> 07:23.900
For example, if we were working
with a database and we would like

07:23.900 --> 07:27.870
to preserve the data created by
the container, we could add a

07:27.870 --> 07:30.090
persistent volume to the container.

07:30.960 --> 07:34.460
To do this first, create
a volume using the Docker

07:34.460 --> 07:35.760
volume, create command.

07:36.600 --> 07:41.610
So, when I run the Docker volume,
create data_volume command, it

07:41.610 --> 07:48.780
creates a folder called data_volume
under the var/lib/docker volumes

07:48.810 --> 07:53.720
directory, then when I run the
Docker container using the Docker

07:53.720 --> 07:57.540
run  command, I could mount
this volume inside the Docker

07:57.540 --> 08:01.490
containers, read/write layer
using the -v option, like this.

08:02.230 --> 08:07.390
So, I would do a Docker run -v then
specify my newly created volume

08:07.390 --> 08:11.890
name, followed by a colon and the
location inside my container, which

08:11.890 --> 08:18.490
is the default location where MySQL
stores data and that is var/libmysql.

08:18.490 --> 08:21.710
And then the image name, MySQL.

08:21.710 --> 08:26.020
This will create a new container
and mount the data volume we

08:26.020 --> 08:30.460
created into var/lib/mysql
folder inside the container.

08:30.870 --> 08:35.150
So, all data written by the
database is in fact, stored on the

08:35.150 --> 08:37.470
volume, created on the Docker host.

08:38.420 --> 08:42.750
Even if the container is destroyed,
the data is still active.

08:43.430 --> 08:46.050
Now, what if you didn't run
the Docker volume create

08:46.050 --> 08:49.380
command to create the volume
before the Docker run command.

08:49.950 --> 08:53.670
For example, if I run the Docker
run commands to create a new

08:53.670 --> 08:59.420
instance of MySQL container
with the volume data_volume 2,

08:59.470 --> 09:04.030
which I have not created yet,
Docker will automatically create

09:04.030 --> 09:09.280
a volume named data _volume 2,
and mount it to the container.

09:10.000 --> 09:15.140
You should be able to see all these
volumes if you list the contents of

09:15.140 --> 09:17.870
the var/lib/docker volumes folder.

09:17.870 --> 09:20.950
This is called volume mounting.

09:21.820 --> 09:26.240
As we are mounting a volume created
by Docker under the var/lib/docker

09:26.240 --> 09:32.010
volumes folder, but what if we had
our data already at another location?

09:32.010 --> 09:35.970
For example, let's say we have
some external storage on the Docker

09:35.970 --> 09:41.850
host at /data and we would like
to store database data on that

09:41.850 --> 09:46.250
volume and not in the default
var/lib/docker volumes folder.

09:46.250 --> 09:50.990
In that case, we would run a
container using the command Docker

09:50.990 --> 09:55.740
run -v, but in this case, we will
provide the complete path to the

09:55.740 --> 09:57.780
folder we would like to mount.

09:57.830 --> 10:04.530
That is /data/MySQL and so it
will create a container and mount

10:04.530 --> 10:06.510
the folder to the container.

10:07.470 --> 10:09.330
This is called bind mountain.

10:09.750 --> 10:13.740
So, there are two types of mounts
of volume mounting and a bind mount.

10:14.280 --> 10:18.150
Volume mount mounts of volume
from the volumes directory and

10:18.180 --> 10:23.210
bind mount mounts indirectly from
any location on the Docker host.

10:24.860 --> 10:27.680
One final point to note
before I'll let you go.

10:28.290 --> 10:30.830
Using the -v is an old style.

10:31.370 --> 10:37.220
The new way is to use -mount option,
but that -mount is the preferred

10:37.220 --> 10:41.920
way as it is more verbose, so you
have to specify each parameter

10:41.950 --> 10:44.140
in a key equals value format.

10:44.800 --> 10:48.630
For example, the previous command
can be written with the -mount

10:48.630 --> 10:52.990
option as this using the type
source and target options.

10:53.680 --> 10:55.570
The type in this case is bind.

10:55.960 --> 11:00.370
The source is the location
on my host and target is the

11:00.370 --> 11:01.960
location on my container.

11:05.140 --> 11:08.350
So, who is responsible for
doing all of these operations?

11:09.010 --> 11:12.790
Maintaining the layered architecture,
creating a writeable layer,

11:12.850 --> 11:16.930
moving files across layers to
enable copy and write, et cetera.

11:17.300 --> 11:18.950
It's the storage drivers.

11:19.000 --> 11:23.170
So, Docker uses storage drivers
to enable layered architecture.

11:23.590 --> 11:30.350
Some of the common storage drivers
are a AUFS, VTRFS,  VFS device-

11:30.350 --> 11:33.180
mapper, overlay,  and overla2.

11:33.180 --> 11:36.520
The selection of the story
is driver depends on the

11:36.550 --> 11:38.290
underlying OS being used.

11:38.290 --> 11:42.560
For example, with Ubuntu the
default storage driver is AUFS,

11:42.560 --> 11:46.360
whereas this storage driver is
not available on other operating

11:46.360 --> 11:48.820
systems like Fedora or CentOS.

11:48.820 --> 11:52.810
In that case device-mapper,
maybe a better option.

11:53.740 --> 11:57.970
Docker will choose the best storage
driver available automatically

11:58.030 --> 11:59.590
based on the operating system.

12:00.330 --> 12:04.170
The different storage drivers
also provide different performance

12:04.200 --> 12:06.210
and stability characteristics.

12:06.870 --> 12:09.870
So, you may want to choose one
that fits the needs of your

12:09.870 --> 12:12.120
application and your organization.

12:12.690 --> 12:16.330
If you would like to read more
on any of these storage drivers,

12:16.380 --> 12:19.380
please refer to the links in
the attached documentation.

12:20.340 --> 12:24.170
For now, that is all from the
Docker architecture concepts,

12:24.170 --> 12:26.440
see you in the next lecture.