Setting Up a Remote Podman Service

Setting Up a Remote Podman Service

Overview

Setup up a Linux instance to act as a podman server allowing you to run podman and podman-compose commands remotely. So for example you can develop on a Mac, but run containers via podman and podman-compose at least some of the time. This article will show you how to:

  • Setup a remote podman server (RHEL, Fedora, CentOS - Ubuntu should just be package changes)

  • Setup podman remote client on your Mac (Should work for Windows and Linux too)

Background

For an increasing number of reasons lately I’ve been defaulting to podman more and more on my Mac though I still also have docker installed and use both day to day. I’m not going to go into the pros and cons of both in this article but there are some resources at the end of this article.

Setting up podman remote client has been covered before, for example Podman remote clients for macOS and Windows and in the repo’s Podman remote-client tutorial itself however I found a couple of gotchas hence this article. The goal is for this to be a comprehensive HOWTO. which should work if followed on both x86 and M1 Macs.

Pre-Requisites

  • Linux host e.g. a home-lab machine or Linux VM

  • Pretty much any Mac, M1 or x86 (the instructions below should work with few changes for Windows or another Linux client if you have a use case)

Part 1: Setting up your Podman Host

So recently I built a powerful home-lab machine to run Fedora and planned to use it to learn KVM better and run VMs. In my case, and this is way overkill for typical needs, I built an 8 core AMD Ryzen 3800 based system with a generous with 128GB RAM, and a 1GB M2 drive. Currently it runs Fedora 34 and Podman 3.1.2. This article should work fine with far more modest configurations.

  • Install podman

  • Enable podman daemon to listen for remote request

  • setup ssh keys - careful here, it’s a little fussy

  • Add the remote podman service to the Mac

Install and Enable podman

Assumptions: You already have a running fedora (RHEL, CentOS, etc should work fine), with sshd already configured.

  1. Install podman and the optional compose components

    fedora $ sudo dnf install -y podman docker-compose podman-compose
  2. Enable and start the service for your user by enabling the podman.socket

    fedora $ systemctl --user --now enable podman.socket
  3. Keep the socket alive with linger will ensure the podman service is always available irrespective of whether you are logged in or not

    fedora $ sudo loginctl enable-linger $USER

Verify the podman.socket is listening

There are a number of ways we can confirm the socket is now listening.

  1. Easiest is to query with systemctl

    fedora $ systemctl status podman.socket
    ● podman.socket - Podman API Socket
         Loaded: loaded (/usr/lib/systemd/system/podman.socket; enabled; vendor preset: disabled)
         Active: active (listening) since Sat 2021-05-29 07:01:11 EDT; 1 day 3h ago
       Triggers: ● podman.service
           Docs: man:podman-system-service(1)
         Listen: /run/podman/podman.sock (Stream)
         CGroup: /system.slice/podman.socket
  2. You can also simply curl the podman.socket

    fedora $ curl  --unix-socket /run/user/1000/podman/podman.sock http://localhost/_ping
    OK%
    Note
    The path above contains your User ID /run/user/<USER_ID>/… which you can obtain with id Remember this as you’ll need during the setup section on your Mac.

Setup ssh authentication on your Mac

This was the part that initially caught me out. I’d been using RSA keys for years so I initially tried it with my id_rsa key. This was also how I had seen examples both in the docs and in at least one blog article. Turns out at some point, I’m assuming relatively recently, support for RSA keys has either been dropped in favor of the more recent, and secure, ed25519 keys or I simply caught a bug/issue. Certainly there were other users caught by this.

Note

Reading through closed Issues in the Podman Repo it may be that RSA keys do now again work. Testing this is an exercise for the reader, and hey ed25519 keys are stronger!

Important
This section is performed on your Mac. (should also work on a Windows/Linux host)
  1. Generate your ed25519 key if you don’t already have one

    mac $ ssh-keygen -t ed25519 -C "jdoe@example.com"
  2. Copy your new key from your Mac to your podman host, in my case fedora

    mac $ ssh-copy-id -i ~/.ssh/id_ed25519 fedora
  3. Validate by ssh’ing to your podman machine explicitly using the new key

    mac $ ssh tok@fedora -i ~/.ssh/id_ed25519
    Last login: Sun May 30 09:40:05 2021 from 192.168.1.54
    fedora $

Part 2: Connect to the podman Service

Setting up your Mac. I’m a sort of latest and greatest person so today this is running on the latest Big Sur 11.4 and I mix an older MBP 16” with a new MacBook Air M1 which is an awesome portable rocket ship. Despite using Ansible for most things lets keep this simple:

  1. Install podman on your Mac

    mac $ brew install podman
    Note

    I’m assuming your PATH is setup correctly /usr/local/bin on x86 and /opt/homebrew/bin on the M1. Incidentally it’s worth noting if you are installing on an M1 based system that it is actually an ARM based package:

    Pouring podman—​3.1.2.arm64_big_sur.bottle.tar.gz

  2. Add the remote podman service

    mac $ podman system connection add development --identity ~/.ssh/id_ed25519 ssh://homelab/run/user/1000/podman/podman.sock
    Note
    • The development in the above command should be a meaningful name for your remote system as you can have multiple remote podman services

    • Remember you need your remote User ID which if you’ve forgotten can easily be retrieved eg:

    ssh tok@fedora -i ~/.ssh/id_ed25519 id
    uid=1000(tok) gid=1000(tok) groups=1000(tok),36(kvm)
    context=unconfined_u:unconfined_r:unconfined_t:s0-s0:c0.c1023

Test your system

Now you should be able to access and use podman from your Mac

  1. List the podman system connection services

    mac $ podman system connection list
    Name          Identity                    URI
    development*  /Users/tok/.ssh/id_ed25519  ssh://tok@fedora:22/run/user/1000/podman/podman.sock
    production    /Users/tok/.ssh/id_ed25519  ssh://tok@production:22/run/user/1000/podman/podman.sock
    Tip

    Here there are 2 remote systems which can both be used, including our new development service we just created. To set the default, here development, on your Mac: podman system connection default development

  2. Now test from your Mac

    mac $ podman system info | head
    host:
      arch: amd64
      buildahVersion: 1.20.1
      cgroupManager: systemd
      cgroupVersion: v2
      conmon:
        package: conmon-2.0.27-2.fc34.x86_64
        path: /usr/bin/conmon
        version: 'conmon version 2.0.27, commit: '
      cpus: 16
  3. Launch your first image, let’s use alpine since it’s nice and small

    mac $ podman run -it --rm docker.io/alpine sh
    Trying to pull docker.io/library/alpine:latest...
    Getting image source signatures
    Copying blob sha256:540db60ca9383eac9e418f78490994d0af424aab7bf6d0e47ac8ed4e2e9bcbba
    Copying config sha256:6dbb9cc54074106d46d4ccb330f2a40a682d49dda5f4844962b7dce9fe44aaec
    Writing manifest to image destination
    Storing signatures
    / #  uname -a
    Linux 21bd9e863f71 5.12.7-300.fc34.x86_64 #1 SMP Wed May 26 12:58:58 UTC 2021 x86_64 Linux
  4. In another terminal on your Mac check what’s running

    mac $ podman ps
    CONTAINER ID  IMAGE                         COMMAND  CREATED        STATUS          PORTS   NAMES
    21bd9e863f71  docker.io/library/alpine:latest  sh      4 minutes ago  Up 4 minutes ago        beautiful_ishizaka
  5. And also from your Linux host try the same podman ps

    fedora $ podman ps
    CONTAINER ID  IMAGE                         COMMAND  CREATED        STATUS          PORTS   NAMES
    21bd9e863f71  docker.io/library/alpine:latest  sh      4 minutes ago  Up 4 minutes ago        beautiful_ishizaka
    Note

    podman and docker typically default to different registries so for example docker run -d httpd will default to docker.io/httpd whilst podman will not. So make sure to qualify your image names or you may see unexpected results.

Cheatsheet

Table 1. podman system commands
List Command

podman system info

podman system info

Add podman remote system production

podman system connection add production --identity ~/.ssh/id_ed25519 \ ssh://production/run/user/1000/podman/podman.sock

Remove podman remote system production

podman system connection remove production

List podman systems

podman system connection list

Make podman system production default

podman system connection default production

Conclusion

So now you can seamlessly run containers using podman on one, or more, remote systems. This HOWTO can easily be extended to allow access to a similar remote service running as root if necessary. <INSERT usual security warnings!!>

In future articles I want to explore topics including:

  • Creating podman based systemd services

  • Running your own long lived registry

  • docker-compose and podman-compose

Resources