You might be wondering on what are we going to discuss in this article as it is most likely the case that you are seeing the error for the first time. But I found it very interesting, especially the whole debugging process I followed to figure out the actual root cause triggering above error. Going forward there are few software and command-line utilities that I mention to explain the context in detail. I have managed to provide hyperlinks to manual pages or websites for those utilities/terminologies I find may be useful for readers to go through and familiarize themselves with more insights.

Preamble

Most of you might have heard about Samba software suite, which is an implementation of SMB protocol providing file and print services. As part of my daily job we have setup Continuous Integration(CI) nightly jobs to create RPMs out of upstream master branch. These jobs make use of mock software to build RPMs for Fedora, CentOS etc. At present we have jobs for building master code base, v4.14 and 4.15 of Samba for Fedora 34, 35 and CentOS 7,8. RPMs created out of these jobs are pushed to CI artifacts server as YUM repositories for easier consumption. Every job is run on a dedicated CentOS Linux 8(now CentOS Stream 8) host allocated using specific tools from CentOS CI infrastructure. Following is a sample mock command that we use to create master branch RPMs for Fedora 35:

# mock --root /etc/fedora-35-x86_64.cfg --resultdir /tmp/samba-build/rpms/master/fedora/35/x86_64 --rebuild /tmp/samba-build/srpms/<samba.src.rpm>

Stoppage

Ever since we started creating RPMs for Fedora 35(on a CentOS Linux 8 host), we always encountered the following error from mock:

Errors during downloading metadata for repository ‘fedora’:

Curl error (6): Couldn’t resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo ‘fedora’: Cannot prepare internal mirrorlist: Curl error (6): Couldn’t resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]
ERROR: Command failed:
# /usr/bin/systemd-nspawn -q -M 60d173130f8442799d437b8a889b5062 -D /var/lib/mock/fedora-35-x86_64-bootstrap/root -a –capability=cap_ipc_lock –bind=/tmp/mock-resolv.u51p8qri:/etc/resolv.conf –setenv=TERM=vt100 –setenv=SHELL=/bin/bash –setenv=HOME=/var/lib/mock/fedora-35-x86_64/root/installation-homedir –setenv=HOSTNAME=mock –setenv=PATH=/usr/bin:/bin:/usr/sbin:/sbin –setenv=PROMPT_COMMAND=printf “\033]0;\007” –setenv=PS1= \s-\v\$ –setenv=LANG=C.UTF-8 –setenv=LC_MESSAGES=C.UTF-8 –resolv-conf=off /usr/bin/dnf –installroot /var/lib/mock/fedora-35-x86_64/root/ –releasever 35 –setopt=deltarpm=False –allowerasing –disableplugin=local –disableplugin=spacewalk –disableplugin=versionlock install @buildsys-build –setopt=tsflags=nocontexts
No matches found for the following disable plugin patterns: local, spacewalk, versionlock
fedora 0.0 B/s | 0 B 00:00
Errors during downloading metadata for repository ‘fedora’:

Curl error (6): Couldn’t resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]
Error: Failed to download metadata for repo ‘fedora’: Cannot prepare internal mirrorlist: Curl error (6): Couldn’t resolve host name for https://mirrors.fedoraproject.org/metalink?repo=fedora-35&arch=x86_64 [getaddrinfo() thread failed to start]

It is basically a curl error and we all normally doubt resolv.conf problems. But curl worked fine on the host system without any errors. We also checked other jobs and just Fedora 35 jobs were failing. So it has something to do with new Fedora 35. Additionally the very same job completed successfully on a Fedora 35 host(instead of CentOS Linux 8 host). We became curious on why it only fails on CentOS Linux 8.

Behind the scene

In order to further debug the issue, we have to get into the container systemd-nspawn creates for a Fedora 35 environment. For those of you who don’t know what systemd-nspawn containers are, read through the manual page for better clarity. In a nutshell it can be used as an enhanced chroot. Mock by default uses systemd-nspawn to create namespace containers for specific distro environments. As we can see from the error pasted in the above section one of the main argument to systemd-nspawn is the directory to be used as file system root for the container. There are many other optional arguments used to create a virtual file system hierarchy with process tree, IPC subsystems, domain etc. For our debugging purpose we could omit most of them to have a minimal Fedora 35 container environment without much complexities. To start with we need to have a directory to be used as file system root. Let’s use dnf to create such a root directory for Fedora 35 using –installroot argument as follows:

# dnf --installroot /var/tmp/f35-root --disablerepo='*' --enablerepo f35 --repofrompath=f35,https://kojipkgs.fedoraproject.org/repos/f35-build/latest/x86_64/ --nogpgcheck install strace

Please note that we install strace utility to analyze behind the scene calls while running curl command. On executing above dnf command we are presented with a Fedora 35 root at /var/tmp/f35-root. Now we instruct systemd-nspwan to spin up a namespace container with /var/tmp/f35-root as file system root with minimal arguments and present us with a bash shell.

# systemd-nspawn -q -D /var/tmp/f35-root -a --capability=cap_ipc_lock --bind=/etc/resolv.conf:/etc/resolv.conf --setenv=SHELL=/bin/bash --setenv=LC_MESSAGES=C.UTF-8 --resolv-conf=off bash

bash-5.1#

TA..DA! We are inside a Fedora 35 container. How about trying out curl command?

The culprit

bash-5.1# curl fedoraproject.org

curl: (6) getaddrinfo() thread failed to start

Oops.. there you go. The very same error we saw in mock output. What do we do now? Why is it failing? We now have strace utility to trace the calls. Let’s see what it tells:

bash-5.1# strace -o strace.log curl fedoraproject.org

curl: (6) getaddrinfo() thread failed to start

Searching through strace.log we could find a failure for the following syscall.

clone3({flags=CLONE_VM|CLONE_FS|CLONE_FILES|CLONE_SIGHAND|CLONE_THREAD|CLONE_SYSVSEM|CLONE_SETTLS|CLONE_PARENT_SETTID|CLONE_CHILD_CLEARTID, child_tid=0x7f1b29620910, parent_tid=0x7f1b29620910, exit_signal=0, stack=0x7f1b28e20000, stack_size=0x7ffe00, tls=0x7f1b29620640}, 88) = -1 EPERM (Operation not permitted)

Woot..!! We have the culprit as clone3() syscall not allowed to execute inside the container. Further reading led us to the syscall filtering mechanism by systemd using seccomp filters. You can read more from kernel.org documentation.

Particulars

Fedora 35 comes with newer glibc v2.34. Curl utility when run tries to create threads which internally boils down to glibc calling clone3() syscall to achieve the purpose. This clone3() syscall is not in the allowed list of syscalls configured by systemd-nspawn. As per systemd-nspawn GitHub sources, it creates an allow-list containing different sets of syscalls defined by systemd. In recent versions of systemd, clone3() is part of @process set and is included within allow-list. For verification purpose we do have systemd-analyze command-line utility to display various sets of syscalls. But on an updated CentOS Linux 8 host we don’t see clone3() under @process set.

# systemd-analyze syscall-filter | grep clone3

systemd also maintains a @known set of syscalls. However the seccomp filtering rules are added inside systemd-nspawn in such a way that it returns EPERM(instead of returning ENOSYS by default) for all from @known set but not from allow-list. Thus on receiving EPERM for clone3(), glibc doesn’t fallback to others to deal with thread creation resulting in curl failure.

The end

Now that we know what is going wrong let’s see if we can deal with the issue using some workarounds or additional options to various components involved until fixed systemd version is available on CentOS Linux 8. We managed to find out following solutions to the problem:

• Use –system-call-filter=clone3 to systemd-nspawn which adds the space separated list of syscalls or sets to the allow-list

# systemd-nspawn -q -D /var/tmp/f35-root -a --system-call-filter=clone3 --capability=cap_ipc_lock --bind=/etc/resolv.conf:/etc/resolv.conf --setenv=SHELL=/bin/bash --setenv=LC_MESSAGES=C.UTF-8 --resolv-conf=off bash

• Export SYSTEMD_SECCOMP=0 to disable seccomp filtering within systemd

# export SYSTEMD_SECCOMP=0

# systemd-nspawn -q -D /var/tmp/f35-root -a --capability=cap_ipc_lock --bind=/etc/resolv.conf:/etc/resolv.conf --setenv=SHELL=/bin/bash --setenv=LC_MESSAGES=C.UTF-8 --resolv-conf=off bash

• Refrain from using systemd-nspawn by mock. Instead use legacy isolation mechanism

# mock --root /etc/fedora-35-x86_64.cfg --isolation=simple --resultdir /tmp/samba-build/rpms/master/fedora/35/x86_64 --rebuild /tmp/samba-build/srpms/<samba.src.rpm>

I hope this helps someone who might encounter similar error in related environments. All commands from this article have been tried on an updated(till date) CentOS Linux 8 host. Feel free to add your suggestions as comments. Let this debugging saga be a motivation for others to get into details while trying to solve various issues. Bye..

Update: mock v2.16-1 got released yesterday with the second workaround mentioned towards the end of article which basically disables seccomp filtering within systemd. Updates are already in place and hopefully will be available with EPEL 8 stable repositories soon.