Kenny's Blog

03 Mar 2021

Ubuntu and systemd-resolved

I’ve been slowly setting up my homelab environment, and one problem I ran into recently was my salt minion wasn’t connecting to my salt master

Looks like DNS errors:

 ubuntu@kvm02:~$ sudo systemctl status salt-minion
 ● salt-minion.service - The Salt Minion
      Loaded: loaded (/lib/systemd/system/salt-minion.service; enabled; vendor preset: enabled)
      Active: active (running) since Wed 2021-03-03 16:20:53 UTC; 8h ago
        Docs: man:salt-minion(1)
    Main PID: 2246 (salt-minion)
       Tasks: 4 (limit: 4730)
      Memory: 70.9M
      CGroup: /system.slice/salt-minion.service
              ├─2246 /usr/bin/python3 /usr/bin/salt-minion
              ├─2251 /usr/bin/python3 /usr/bin/salt-minion
              └─2285 /usr/bin/python3 /usr/bin/salt-minion

 Mar 04 01:01:19 kvm02 salt-minion[2251]: [ERROR   ] DNS lookup or connection check of 'salt' failed.
 Mar 04 01:01:19 kvm02 salt-minion[2251]: [ERROR   ] Master hostname: 'salt' not found or not responsive. Retrying in 30 seconds

I hadn’t run into this problem with my debian machines, so I was a bit confused:

The default salt-minion configuration uses the salt hostname, so I was thinking there was a DNS configuration issue. I’ve configured my edgerouter to resolve the salt hostname to the salt master’s ip address.

 # Set the location of the salt master server. If the master server cannot be
 # resolved, then the minion will fail to start.
 #master: salt

Turns out, systemd-resolved (which ubuntu uses), won’t pass simple names: https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu

The page links to the following github issue: https://github.com/systemd/systemd/issues/2514

Basically, single-label hostnames are considered “private”, something that should be made sense of locally only, and as such should never end up on the public Internet.

Bleh, so I have two options here:

  1. Disable systemd-resolved (see this https://askubuntu.com/questions/907246/how-to-disable-systemd-resolved-in-ubuntu)
  2. Use a fully qualified domain name (fqdn) for my salt master’s hostname

I ended up using a fqdn instead of trying to mess around with disabling systemd-resolved.