Podman, Docker

Podman, Docker

2017-02-20. Category & Tags: Podman, Docker

See also:
cnvrg.io How to setup Docker and Nvidia-Docker 2.0 on Ubuntu 18.04 (bak)

Docker container lifecycle (commands events states status)

Timing: time spent to install & test (for the 1st time): 15 tomatos/hours.

Comparing Container Engines #

Tip: podman has exactly the same CLI as Docker, so previous Docker users can move to Podman by:
alias Docker=Podman.

Podman advantages compared to Docker:

  • doesn’t need daemon
  • doesn’t need root privileges (which has been long-standing concern with Docker)
  • can also manage pods

Other alternavtive engines:

  • LXD: (LinuX container manager Daemon) for LXC (LinuX Containers) [few users, ‘system’ containers like VMs]
  • CRI-O: more like a runtime instead of en engine, only for k8s not for normal end-users // “runtime” which is responsible for running containers
  • rkt (as ‘rocket’): by CoreOS [deprecated]

ref: TowardsDataScience, (also in cn)

Install #

Tested on Ubuntu 16 with Docker 18.
Make sure NIX kernel >=3.8 by uname -a.

Uninstall old versions first !!!
Note: older versions named “docker” or “docker-engine” should be uninstalled first.

WARN! install & update docker should use the same way (script vs. repo).

option 1 - convenience script #

For test only, fully automatic, NON-interactive, NOT for production (due to security reasons).
For security reasons, plz verify the script content before installation.

export DOWNLOAD_URL="http://mirrors.tuna.tsinghua.edu.cn/docker-ce" # for CN users
curl -fsSL https://get.docker.com -o get-docker.sh
sudo sh get-docker.sh # silent & automatic

alt. curl -sSL https://get.daocloud.io/docker | sh [ref]

ref: TsingHua mirror, Tencnet blog 2023, zhihu for 2204 & older system.

option 2 - repo (official & private) # 

sudo apt-get update && \
sudo apt-get install -y apt-transport-https ca-certificates curl software-properties-common && \
curl -fsSL https://download.docker.com/linux/ubuntu/gpg | sudo apt-key add - && \
sudo apt-key fingerprint 0EBFCD88 && \
sudo add-apt-repository \
    "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
    $(lsb_release -cs) \
    stable" && \
sudo apt-get update && \
sudo apt-get install -y docker-ce && \
sudo systemctl status docker && \
sudo usermod -aG docker ${USER} && \
sudo su - ${USER} && \
id -nG && \
docker ps

To add another user into docker group: sudo usermod -aG docker $USER (may need the group before this cmd: sudo groupadd docker), then may need to sudo chown root:docker /var/run/docker.sock; sudo chmod g+rw /var/run/docker.sock.

option 3 - apt repo (aliyun mirror) # 

curl -fsSL https://mirrors.aliyun.com/docker-ce/linux/ubuntu/gpg | sudo gpg --dearmor -o /etc/apt/keyrings/aliyun-docker.gpg
echo \
  "deb [arch=$(dpkg --print-architecture) signed-by=/etc/apt/keyrings/aliyun-docker.gpg] https://download.docker.com/linux/ubuntu \
  $(lsb_release -cs) stable" | sudo tee /etc/apt/sources.list.d/docker.list > /dev/null
sudo apt update
sudo apt install -y docker-ce

ref: zhihu for 2204 & older system.

option 4 - deb package #

Download .deb here and install by:

sudo dpkg -i /path/to/package.deb && \

Post-install Config #

ustc mirror #

sudo vi /etc/docker/daemon.json

{
        "registry-mirrors": ["https://docker.mirrors.ustc.edu.cn/"]
}

usermod # 

# note: usermod needs re-login/new terminal/su to take effect
sudo usermod -aG docker ${USER} && \
sudo su - ${USER} && \
id -nG && \
docker ps # which should show a table (maybe empty, but teable-headers should be shown)

WARN: adding a user to the group docker gives them root privilege (except using non-root docker).

verify # 

docker run hello-world && \
docker container ls --all # '-all' shows also stopped ones
docker image ls
docker system df -v # check image sizes

network dns #

In many organizations, the available DNS servers are only internal ones, but the docker files are usually using Google DNS.
Modify /etc/docker/daemon.json to use internal DNS servers (overwriting docker’s network.)
For example:

{
  "insecure-registries": ["192.168.1.1:44453"],
  "dns": ["10.1.1.1", "10.2.1.1", "104.104.104.104"],
  "experimental": true,
  "mtu": 1450
}

and restart docker daemon:

service docker restart

Note insecure-registries are optional; experimental features are optional; mtu is usually optional, but manadatory when using OpenStack [ref1, ref2].

permission #

Add non-root users to the docker group to give permission.

sudo usermod -aG docker <non_root_username>

Note: the user should re-login the terminal/SSH to make it take effect, which can be verified by groups W/O giving the username as param. (groups print current situation, groups <username> print the situation in its DB/config).

Next - Get Started #

See the official doc. [bak part 1], [bak part 2]

Compose #

“Compose” is a tool for defining and running multi-container apps, i.e., applications with more than one container/service. With Compose, you use a .yml file to configure your application’s services. Then, using a single command, you create and start all the services from your configuration. [Github]

Each service needs to point to an image or build directory; all other keywords (links, ports, environment, restart) correspond to Docker options. [codeship]

Install:

sudo curl -L "https://github.com/docker/compose/releases/download/1.23.1/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose && \
sudo chmod +x /usr/local/bin/docker-compose && \
docker-compose --version

[ref]

Example of docker-compose.yaml using Pulsar [ref]:

version: '3.5'
services:
    pulsar-standalone:
        image: apachepulsar/pulsar
        container_name: pulsar-standalone
        ports:
            - 6650:6650
            - 8080:8080
        environment:
            - PULSAR_MEM=" -Xms512m -Xmx512m -XX:MaxDirectMemorySize=1g"
        command: /bin/bash -c "bin/apply-config-from-env.py conf/standalone.conf && bin/pulsar standalone"
    pulsar-dashboard:
        image: apachepulsar/pulsar-dashboard
        container_name: pulsar-dashboard
        ports:
          - "80:80"
        environment:
            - SERVICE_URL=http://pulsar-standalone:8080

Docker Client in Windows Subsystem/Ubuntu #

When Docker Desktop is running in windows, the control commands can be sent from within Win-subsys. I.e, running a “client” in Win-subsys to control the Docker “server” (Docker Desktop) in win.
See: ref.
Summary: export localhost:2375 from Win, and install docker in subsys. Give -H <host:port> param to all docker commands in subsys.

Scaling #

The levels of docker architecture is:

  • Container. An isolated environment for an app.
  • Service. A load-balanced container-cluster with online/inplace/hot scaling via .yml file re-loading triggered by docker stack deploy.
  • (Swarm). Similar to a service, a swarm is a cluster of machines running Docker (i.e., a “Dockerized” cluster by joining multiple physical or virtual machines). After joining a swarm, the machines are referred to as nodes. (Docker Machine is needed.)
  • Stack. A stack is a group of interrelated services that share dependencies, and can be orchestrated and scaled together. A stack can define and coordinate an entire application (if not very complex).

Performance #

No influence/overhead on CPU, RAM and local disk IO (both sequence and random access) [IBM14, BlueData17 pp27-28].

Data locality is not so important any more databricks18 (see PPT p27 for storage solutions), though network (e.g. no data locality, on remote HDFS) has influence. jd.com

Management Tools #

e.g. lazy-docker, protainer etc.

FAQ & TIPS #

data sharing #

Problem: need to share files and mount volumnes from host to containers.
Solution: -v /absolute/host/src_dir/:/container/dest_dir/. (WARN： non-absolute path fails w/o displaying any info.)
Tip: –volume (automatically creates dest_dir) vs. –mount: [official doc], [stackoverflow].

auto-completion #

Problem: need cmd/CLI auto-completion.
Solution:
apt install bash-completion -y

or a bigger client package (which includes bash-completion)

apt install docker-ce-cli -y

[ref: containerd.io vs docker-ce-cli vs docker-ce]

See also: completion for compose & machine.

network/firewall #

Problem: DockerHub network issue/slow.
Solution 1: modify mirrors in /etc/docker/daemon.json using sudo tee /etc/docker/daemon.json <<-'EOF' followed by:

{
  "registry-mirrors": ["https://registry.docker-cn.com", "https://docker.mirrors.ustc.edu.cn", "https://hub-mirror.c.163.com", "https://mirror.baidubce.com"]
}

Note: aliyun’s server requires registration.

Solution 2: docker alternatives: e.g. OpenEuler.

port forwarding #

Problem: need port forwarding, but sometimes not working.
Reason: [possibily] the local port has been used, but docker will not warn.
Solution: check netstat -tulvpn first then do port forwarding:

docker run -p [local-ip:]<local-port>:<container-port> \
...[other options]... -it <image>

going in/keep running #

Problem: going in
Solution: docker exec -it <running_container_id> /bin/bash.

Problem: the docker runs and quits without giving me time to check what is inside it.
Reason: TODO (a container quits/stops for many reasons.)
Solution:

docker run <image> bash

or

docker run --entrypoint /bin/bash \
...[other options]... -it <image>

This will temporarily overwrite the entrypoint.
Ref: medium, github discussion.

Problem: want to keep the docker running without modifying the entrypoint.
Solution: (We can do below, but not sure if it overwrites the entrypoint).

docker run -d <image>[:tag] /bin/sh -c "while true; do echo 'hello world'; sleep 2; done" # Note: this cmd MAY not display anything

Or:

docker run -d <image>[:tag] /bin/sh -c "tail -f /dev/null"

Problem: want to see full command of a running/stopped container. docker inspect <CONTAINER> (inc. env variables) or docker ps --no-trunc (NO env variables). ref

Docker Mirrors in CN #

sudo vim /etc/docker/daemon.json

{
  "registry-mirrors": ["https://<personal_key>.mirror.aliyuncs.com", "https://docker.mirrors.ustc.edu.cn/", "https://ustc-edu-cn.mirror.aliyuncs.com/"]
}

sudo systemctl daemon-reload && sudo systemctl restart docker

Ali personal key is available in panel > mirror accelerator. Please note that according to official documents, Docker Hub is limitting the access, so the mirros via accelerators may not be the latest.

We may also host our own images on Aliyun.

Ref #

docker.com
More:
docker labs
cn docker accelerators & test
energy-physics-workloads

docker.com
More:
docker labs
cn docker accelerators & test