Introduction

the Docker terms have recently changed * terms

Docker is an Open Platform containerisation virtualisation system for developing, shipping, and running applications. Docker enables you to separate your applications from your infrastructure so you can deliver software quickly. With Docker, you can manage your infrastructure in the same ways you manage your applications. By taking advantage of Docker’s methodologies for shipping, testing, and deploying code quickly, you can significantly reduce the delay between writing code and running it in production.

Docker is written in Go and takes advantage of several features of the Linux kernel to deliver its functionality. Namespaces

Docker uses a technology called namespaces to provide the isolated workspace called the container. When you run a container, Docker creates a set of namespaces for that container.

These namespaces provide a layer of isolation. Each aspect of a container runs in a separate namespace and its access is limited to that namespace.

Docker Engine uses namespaces such as the following on Linux:

The pid namespace: Process isolation (PID: Process ID).
The net namespace: Managing network interfaces (NET: Networking).
The ipc namespace: Managing access to IPC resources (IPC: InterProcess Communication).
The mnt namespace: Managing filesystem mount points (MNT: Mount).
The uts namespace: Isolating kernel and version identifiers. (UTS: Unix Timesharing System).

Docker Engine on Linux also relies on another technology called control groups (cgroups). A cgroup limits an application to a specific set of resources. Control groups allow Docker Engine to share available hardware resources to containers and optionally enforce limits and constraints. For example, you can limit the memory available to a specific container.

Union file systems, or UnionFS, are file systems that operate by creating layers, making them very lightweight and fast. Docker Engine uses UnionFS to provide the building blocks for containers. Docker Engine can use multiple UnionFS variants, including AUFS, btrfs, vfs, and DeviceMapper. There are slight problems with UnionFs that can cause an EXDEV error (see comments on move below).

Problem renaming UnionFS directories: Calling rename(2) for a directory is allowed only when both the source and the destination path are on the top layer. Otherwise, it returns EXDEV error (“cross-device link not permitted”). Your application needs to be designed to handle EXDEV and fall back to a “copy and unlink” strategy. An alternative strategy inside the docker container is to move the directory somewhere, and then move it back, before an application is permitted to perform updates on the directory. This solves the problem because move handles EXDEV and the result is a top-level read/write directory. I employed that strategy in the iRODS externals build which causes a yum repos update (because the script was designed to run in a VM and not a docker container).

Docker Engine combines the namespaces, control groups, and UnionFS into a wrapper called a container format. The default container format is libcontainer. In the future, Docker may support other container formats by integrating with technologies such as BSD Jails or Solaris Zones, (or HPC singularity).

WARNING: running in a docker container affords the code system privileges. If you include a user in the docker group then they can run code in a docker container and grant themselves root privileges. Obviously security needs to be added to docker (and/or other containerisation provisions) which was going to be my research project.

terms

2Ecom/c/eJwdjksOgyAYhE8jOwj8CMqCha1x2Tu

AYCXFRwAX3r60ySwmmXwz47Tw0i8oaKDAqKIM= qOBcEEbaYRrGpxhBKODTQzYt3Y8Slpu4Y_74ROZjQ6t2inWydUL1yoHgTFkJ3jLlu4UZ21MU9Vr= KmRs-NDBV_ehE4l0t38azcy-UdDLxXOuCSSGH_f3rJuZC5X8L0x5zhu1VI58zLsEnHM21z-sXPA= k5FQ">blog</a> post.

           Thank you,

           The Docker Team

       You're receiving this email because this email address is associate=

d with an active Docker ID

Installation on Debian

Installation is straight forward on Debian 10; all dependencies are met by executing the following sequences:

uninstall the old docker

apt purge docker lxc-docker docker-engine docker.io

install dependencies

apt install apt-transport-https ca-certificates curl gnupg2 software-properties-common

obtain Dockers PGP key and install

curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -

add the Docker repository path

add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/debian buster stable"

update repository lists

apt update

install docker engine, cli and container policies (which depends on SELinux policy)

apt install docker-ce docker-ce-cli containerd.io

verify

docker -v
systemctl status docker

start docker

If docker is not running then execute

systemctl start docker

Installation on Centos/Redhat

OS Pre-requisites

To install Docker Engine, you need a maintained version of CentOS 7. Archived versions aren’t supported or tested. You will also need the yum-utils. If your version of Centos is a redhat version good luck as the repositories are somewhat dated; you may have success like I did by using redhat8.2.

Install the yum-utils package (which provides the yum-config-manager utility) and set up the stable repository.

yum install -y yum-utils

Installation methods

You can install Docker Engine in different ways, depending on your needs:

Most users set up Docker’s repositories and install from them, for ease of installation and upgrade tasks. This is the recommended approach.
Some users download the RPM package and install it manually and manage upgrades completely manually. This is useful in situations such as installing Docker on air-gapped systems with no access to the internet.
In testing and development environments, some users choose to use automated convenience scripts to install Docker.

Install using the repository

Before you install Docker Engine for the first time on a new host machine, you need to set up the Docker repository. Afterward, you can install and update Docker from the repository.

configure the docker repository

yum-config-manager --add-repo=https://download.docker.com/linux/centos/docker-ce.repo

my repositories were after registering and going through the docker isntallation steps:

yum repolist

[root@localhost ralph]# yum repolist
Loaded plugins: product-id, search-disabled-repos, subscription-manager
docker-ce-stable                                                                | 3.5 kB  00:00:00     
(1/2): docker-ce-stable/x86_64/updateinfo                                       |   55 B  00:00:00     
(2/2): docker-ce-stable/x86_64/primary_db                                       |  45 kB  00:00:00     
repo id                                      repo name                                           status
docker-ce-stable/x86_64                      Docker CE Stable - x86_64                               79
rhel-7-server-rpms/7Server/x86_64            Red Hat Enterprise Linux 7 Server (RPMs)            29,413
repolist: 29,492

To enable the test channel, run the following command:

sudo yum-config-manager --enable docker-ce-test

You can disable the nightly or test repository by running the yum-config-manager command with the --disable flag. To re-enable it, use the --enable flag. The following command disables the nightly repository.

sudo yum-config-manager --disable docker-ce-nightly

These repositories are included in the docker.repo file above but are disabled by default. You can enable them alongside the stable repository. The following command enables the nightly repository.

sudo yum-config-manager --enable docker-ce-nightly

To enable the test channel, run the following command:

sudo yum-config-manager --enable docker-ce-test

You can disable the nightly or test repository by running the yum-config-manager command with the --disable flag. To re-enable it, use the --enable flag. The following command disables the nightly repository.

sudo yum-config-manager --disable docker-ce-nightly

You may want to locate files install

yum install -y mlocate

required SELinux policy files

This is very problematic with redhat7.8 trying to find the SELinux polivy file is well hidden. At 8.2 you can tell yum to use the best fit pre-requisite, but on redhat7.8 I could not find that mode switch. Worse, they have moved the docker onwards and upwards, and what worked two weeks ago no longer works!

We can thank NSA contractors for this pollution of dependency sets, and redhat for being so far behind the head.

So I found these sites:

on redhat 7.8

download

http://ftp.riken.jp/Linux/cern/centos/7/extras/x86_64/Packages/container-selinux-2.74-1.el7.noarch.rpm

install container-selinux

yum install container-selinux-2.74-1.el7.noarch.rpm

redhat 8.1 enterprise full image installer

This already installs container-selinux, let's hope it satisfies the requirements. Testing soon.

on redhat 8.2

This might work for redhat 8.2 - good luck!

You will need container-selinux so download from

curl http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm > container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm

or via your browser

http://mirror.centos.org/centos/7/extras/x86_64/Packages/container-selinux-2.119.2-1.911c772.el7_8.noarch.rpm

local install the rpm

yum localinstall -y container-selinux-<tab>

Install Docker Engine

(use this) Install the Docker Engine using the version of containerd that is available:

   yum install -y docker-ce docker-ce-cli containerd.io

If prompted to accept the GPG key, verify that the fingerprint matches 060A 61C5 1B55 8A7F 742B 77AA C52F EB6B 621E 9F35, and if so, accept it.

Got multiple Docker repositories?

If you have multiple Docker repositories enabled, installing or updating without specifying a version in the yum install or yum update command always installs the highest possible version, which may not be appropriate for your stability needs.

To install a specific version of Docker Engine, list the available versions in the repo, then select and install:
This example sorts results by version number, highest to lowest, and is truncated:

   yum list docker-ce --showduplicates | sort -r

   docker-ce.x86_64  3:18.09.1-3.el7                     docker-ce-stable
   docker-ce.x86_64  3:18.09.0-3.el7                     docker-ce-stable
   docker-ce.x86_64  18.06.1.ce-3.el7                    docker-ce-stable
   docker-ce.x86_64  18.06.0.ce-3.el7                    docker-ce-stable

   The list returned depends on which repositories are enabled, and is specific to your version of CentOS (indicated by the .el7 suffix in this example).

Install a specific version by its fully qualified package name, which is the package name (docker-ce) plus the version string (2nd column) starting at the first colon (:), up to the first hyphen, separated by a hyphen (-). For example, docker-ce-18.09.1.

   yum install docker-ce-<VERSION_STRING> docker-ce-cli-<VERSION_STRING> containerd.io

start docker

Docker is installed but not started. The docker group is created, but no users are added to the group.
Start Docker.

   $ sudo systemctl start docker

Verify that Docker Engine is installed correctly by running the hello-world image.

   $ sudo docker run hello-world

The above command downloads a test image and runs it in a container. When the container runs, it prints an informational message and exits.

If it fails to run then see the #Security section - do not follow those instructions on a production server.

Upgrade Docker Engine

To upgrade Docker Engine, follow the installation instructions, choosing the new version you want to install. Install from a package

If you cannot use Docker’s repository to install Docker, you can download the .rpm file for your release and install it manually. You need to download a new file each time you want to upgrade Docker Engine.

Go to https://download.docker.com/linux/centos/ and choose your version of CentOS. Then browse to x86_64/stable/Packages/ and download the .rpm file for the Docker version you want to install.

Note: To install a nightly or test (pre-release) package, change the word stable in the above URL to nightly or test. Learn about nightly and test channels.

Install Docker Engine, changing the path below to the path where you downloaded the Docker package.

   yum install /path/to/package.rpm

Install using the convenience script

Docker provides convenience scripts at get.docker.com and test.docker.com for installing edge and testing versions of Docker Engine - Community into development environments quickly and non-interactively. The source code for the scripts is in the docker-install repository. Using these scripts is not recommended for production environments, and you should understand the potential risks before you use them:

The scripts require root or sudo privileges to run. Therefore, you should carefully examine and audit the scripts before running them.
The scripts attempt to detect your Linux distribution and version and configure your package management system for you. In addition, the scripts do not allow you to customize any installation parameters. This may lead to an unsupported configuration, either from Docker’s point of view or from your own organization’s guidelines and standards.
The scripts install all dependencies and recommendations of the package manager without asking for confirmation. This may install a large number of packages, depending on the current configuration of your host machine.
The script does not provide options to specify which version of Docker to install, and installs the latest version that is released in the “edge” channel.
Do not use the convenience script if Docker has already been installed on the host machine using another mechanism.

This example uses the script at get.docker.com to install the latest release of Docker Engine - Community on Linux. To install the latest testing version, use test.docker.com instead. In each of the commands below, replace each occurrence of get with test.

Warning: Always examine scripts downloaded from the internet before running them locally.

$ curl -fsSL https://get.docker.com -o get-docker.sh
$ sudo sh get-docker.sh

<output truncated>

docker users

If you would like to use Docker as a non-root user, you should now consider adding your user to the “docker” group with something like:

 sudo usermod -aG docker your-user

Remember to log out and back in for this to take effect!

Warning: Adding a user to the “docker” group grants them the ability to run containers which can be used to obtain root privileges on the Docker host. Refer to Docker Daemon Attack Surface for more information.

Docker Engine - Community is installed. It starts automatically on DEB-based distributions. On RPM-based distributions, you need to start it manually using the appropriate systemctl or service command. As the message indicates, non-root users can’t run Docker commands by default.

   Note:
   To install Docker without root privileges, see Run the Docker daemon as a non-root user (Rootless mode).
   Rootless mode is currently available as an experimental feature.

Upgrade Docker after using the convenience script

If you installed Docker using the convenience script, you should upgrade Docker using your package manager directly. There is no advantage to re-running the convenience script, and it can cause issues if it attempts to re-add repositories which have already been added to the host machine. Uninstall Docker Engine

Uninstall the Docker Engine, CLI, and Containerd packages:

   sudo yum remove docker-ce docker-ce-cli containerd.io

Images, containers, volumes, or customized configuration files on your host are not automatically removed. To delete all images, containers, and volumes:

   sudo rm -rf /var/lib/docker

You must delete any edited configuration files manually. Next steps

Continue to Post-installation steps for Linux.
Review the topics in Develop with Docker to learn how to build new applications using Docker.

requirements, apt, installation, centos, rpm, install, uninstall, upgrade, update

volumes

-v --mount volumes ttps://docs.docker.com/storage/volumes/

pruning

prune containers

docker container prune

prune images

docker image prune

Synchronizing yum satellites

cpack

https://cmake.org/cmake/help/latest/module/CPack.html#introduction for making binary packages.

Uninstall old docker versions

Older versions of Docker were called docker or docker-engine. If these are installed, uninstall them, along with associated dependencies.

yum remove docker \
                 docker-client \
                 docker-client-latest \
                 docker-common \
                 docker-latest \
                 docker-latest-logrotate \
                 docker-logrotate \
                 docker-engine

It’s OK if yum reports that none of these packages are installed.

The contents of /var/lib/docker/, including images, containers, volumes, and networks, are preserved.

The Docker Engine package is now called docker-ce.

build cookbook patterns

multi-stage build host read-only volumes
- https://stackoverflow.com/questions/26050899/how-to-mount-host-volumes-into-docker-containers-in-dockerfile-during-build

experimental features

to enable experimental features

sudo nano /etc/docker/daemon.json

This is a new file, so you won’t find anything contained within. Paste the following contents into this file:

{
"experimental": true
}

Security

WARNING: running in a docker container affords the code system privileges. If you include a user in the docker group then they can run code in a docker container and grant themselves root privileges. Obviously security needs to be added to docker (and/or other containerisation provisions) which was going to be my research project.

container security
- https://www.stackrox.com/post/2017/08/hardening-docker-containers-and-hosts-against-vulnerabilities-a-security-toolkit/
- audit https://www.digitalocean.com/community/tutorials/how-to-use-the-linux-auditing-system-on-centos-7
  - https://wiki.archlinux.org/index.php/Audit_framework

I had problems because I put a non-noarch policy file in via yum, and then followed it by a nonarch and installed docker, now my docker can't run. SELinux has sucked the life out of it. So I read this https://phoenixnap.com/kb/how-to-disable-selinux-on-centos-7

you can examine the state of SELinux via

sestatus

you can turn enforcement off via temporarily

setenforce 0

or permanenty turn SELinux off vi /etc/sysconfig.selinux marking SELINIX=desabled

# This file controls the state of SELinux on the system.
# SELINUX= can take one of these three values:
#     enforcing - SELinux security policy is enforced.
#     permissive - SELinux prints warnings instead of enforcing.
#     disabled - No SELinux policy is loaded.
SELINUX=disabled
# SELINUXTYPE= can take one of three values:
#     targeted - Targeted processes are protected,
#     minimum - Modification of targeted policy. Only selected processes are protected.
#     mls - Multi Level Security protection.
SELINUXTYPE=targeted

strace-docker

git clone https://github.com/amrabed/strace-docker && sudo ./strace-docker/install

git clone https://github.com/amrabed/strace-docker && sudo ./strace-docker/install

To check if strace-docker is successfully installed and running, use service strace-docker status

strace-docker is automatically triggered by docker events to monitor any new Docker container. The resulting trace of system calls is written to a new file at /var/log/strace-docker/. File name will be $id-$image-$timestamp where $id is the container ID, $image is the container image, and $timestamp is the time the container started. You can see full log of monitored containers at /var/log/strace-docker/log.

Overlay FS

This content applies to the overlay driver only.

OverlayFS layers two directories on a single Linux host and presents them as a single directory. These directories are called layers and the unification process is referred to as a union mount. OverlayFS refers to the lower directory as lowerdir and the upper directory a upperdir. The unified view is exposed through its own directory called merged.

The diagram below shows how a Docker image and a Docker container are layered. The image layer is the lowerdir and the container layer is the upperdir. The unified view is exposed through a directory called merged which is effectively the containers mount point. The diagram shows how Docker constructs a map to OverlayFS constructs.

Where the image layer and the container layer contain the same files, the container layer “wins” and obscures the existence of the same files in the image layer.

The overlay driver only works with two layers. This means that multi-layered images cannot be implemented as multiple OverlayFS layers. Instead, each image layer is implemented as its own directory under /var/lib/docker/overlay. Hard links are then used as a space-efficient way to reference data shared with lower layers. The use of hardlinks causes an excessive use of inodes, which is a known limitation of the legacy overlay storage driver, and may require additional configuration of the backing filesystem. Refer to the overlayFS and Docker performance for details.

To create a container, the overlay driver combines the directory representing the image’s top layer plus a new directory for the container. The image’s top layer is the lowerdir in the overlay and is read-only. The new directory for the container is the upperdir and is writable.

cleanup cache

https://forums.docker.com/t/how-to-delete-cache/5753
make some aliases

alias docker_clean_images='docker rmi $(docker images -a --filter=dangling=true -q)'
alias docker_clean_ps='docker rm $(docker ps --filter=status=exited --filter=status=created -q)'

cleanup

docker kill $(docker ps -q)
docker_clean_ps
docker rmi $(docker images -a -q)

References

metrics
- https://docs.docker.com/config/containers/runmetrics/

docker stats