In Part 1, I built a small Ansible environment with one control node and two managed Ubuntu web servers. Then I decided to break and fix it.
First, My Nodes Got Stuck
At one point, my managed nodes were taking forever to restart in Multipass. They were stuck long enough that it was obvious something was wrong. Then I force-stopped them, deleted them and purged them.
Then I recreated:
node1
node2
with the same names.
In my head, I had replaced the machines and given the replacements the same names, so I initially expected the environment to mostly continue where I left off. But apparently, machine identity doesn’t work that way.
The new node1 and node2 were not the old ones.
They now had:
different IP addresses
different SSH host keys
empty authorized_keys files
clean filesystems
none of my previous packages
none of my previous configuration
Same Name Does Not Mean Same Machine
This reminded me of replacing a computer at work.
If somebody gets a new laptop and we name it exactly what the old laptop was called, that does not somehow copy the old laptop’s identity, software, certificates, keys and configuration onto the new one.
It is still a different machine. SSH makes that very obvious. My control node already had information about the old nodes and the new nodes had their own SSH identities.
So even though I recreated machines called node1 and node2, I had to rebuild the relationship between the control node and those machines.
First, I checked their new IP addresses:
multipass list
Then I updated my Ansible inventory.
My recreated nodes were now using addresses ending in:
node1 -> .14
node2 -> .15
I needed to set up SSH access again.
authorized_keys
This file exists on the machine I am trying to log into.
For example, on node1:
~/.ssh/authorized_keys
It answers:
Which public keys are allowed to authenticate as this user?
The control node’s public key goes here, and the private key stays on the control node.
known_hosts
This file exists on the SSH client.
In my case, that was the control node.
It answers:
Which server identities have I seen and trusted before?
My control node could still remember an old SSH fingerprint for an IP address, while the newly created VM at that address had a completely different host key.
SSH sees that and gets suspicious. If I connect to a server and SSH previously knew that address as one machine, but it suddenly presents a different cryptographic identity, SSH warns me, and that can happen innocently in a lab because I deleted and recreated a VM.
In a real environment, blindly ignoring that warning would be a terrible habit.
After confirming that I really had rebuilt the machine, I could remove the stale entry:
ssh-keygen -R <ip-address>
Then I could reconnect and trust the new host key. I also had to add the control node’s public key to the new machine’s:
~/.ssh/authorized_keys
again.
This was a good distinction:
authorized_keys
Who is allowed to connect TO this machine?
known_hosts
Which machines does this SSH client trust?
Public Key Here. Private Key There.
I also confused myself at one point about where the keys should go.
CONTROL NODE
Private key -> stays here
Public key -> copied to managed nodes
On the target:
~/.ssh/authorized_keys
contains the public key of the client that should be allowed to connect.
I do not put my control node’s private key on every managed server.
If I were managing hundreds of machines, manually opening authorized_keys on every server and pasting a key would obviously be a bad process.
At scale, I obviously wouldn’t manually SSH into hundreds of servers to add my public key. The initial access would normally be handled during provisioning, for example by configuring SSH access when Terraform creates the servers.
But for this lab, doing it manually helped me understand what Ansible was depending on.
Then Ansible Started Working Again
After:
- checking the new IP addresses
- updating
inventory.ini - fixing stale SSH host-key entries
- adding the control node’s public key to the new nodes
- testing SSH manually
I could go back to:
ansible all -m ansible.builtin.ping
and get:
pong
again.
Then I could rerun my main playbook. This is where idempotent automation became useful in a slightly different way. I had clean machines again. So instead of trying to remember every command I had manually run before deleting them, I already had the desired configuration written down.
Run the playbook.
Install the packages.
Start Nginx.
Deploy the template.
Put the machines back into the state I wanted.
That is a much better recovery story than:
I think I remember what I did to the old server.
Then Multipass Itself Broke
Just when I thought I was back to troubleshooting my Ansible lab, Multipass started behaving strangely.
At one point I got:
list failed: cannot connect to the multipass socket
Now this was interesting because Ansible was not really the problem anymore.
If Multipass cannot properly manage the VMs, then there may not even be a healthy machine for Ansible to connect to.
I started checking the layer underneath Ansible.
On macOS, I inspected the Multipass service:
sudo launchctl print system/com.canonical.multipassd
and:
sudo launchctl list | grep multipass
I also checked that Multipass was installed:
brew list | grep multipass
The daemon appeared to be there, so I kept going.
The Error That Was Causing the Problem
The useful clue eventually came from the logs:
Failed to get shared "write" lock
Is another process using the image?
That pointed me away from Ansible completely.
An orphaned QEMU process was still holding a lock on a VM image.
So I had something closer to:
Ansible
↓
SSH
↓
Ubuntu VM
↓
Multipass
↓
QEMU
↓
macOS
I was initially interacting with the problem from the top of that stack, but the failure was lower down.
Restarting the Mac eventually cleared the stale process and allowed the environment to recover.
I would not turn “restart the computer” into my troubleshooting strategy, but in this particular case it cleared the orphaned virtualization process that was holding the image.
The bigger lesson was knowing which layer was actually broken.
Not Every Ansible Error Is an Ansible Problem
If I run:
ansible-playbook playbooks/site.yml
and something fails, my first instinct should not always automatically be:
My YAML is wrong.
There are several things underneath Ansible that need to work.
For my lab:
Ansible
↓
Inventory
↓
SSH
↓
Authentication
↓
Guest operating system
↓
Virtual machine
↓
Multipass
↓
QEMU
↓
Host operating system
A failure lower in the stack can surface as something that looks like an Ansible problem.
Somehow My Server Was Living in the Future
Another issue showed up while using APT.
I ran:
sudo apt update
and Ubuntu complained that a release file was:
not valid yet
Why would valid repository metadata not be valid yet?
Time.
The VM clock was wrong, so I checked:
date
and:
timedatectl
The guest’s time had drifted enough that the timestamps from the repository did not make sense from the VM’s point of view.
Once the time synchronization issue was corrected, APT behaved normally again.
Time Is Infrastructure Too
That sounds dramatic for a clock, but time matters everywhere.
Think about:
TLS certificates
authentication
package repositories
logs
distributed systems
scheduled jobs
tokens
monitoring
If two systems disagree badly enough about what time it is, strange things start happening.
So when I saw:
not valid yet
I knew I had to check what assumptions the failing system is making.
In this case, APT assumed the machine knew what time it was, but it did not.
Troubleshooting Ansible Errors Instead of Fighting Them
I also got better at reading what Ansible was actually telling me.
Earlier, while learning the user module, I tried something like:
ansible.builtin.user:
name: "{{ user_details.username }}"
password: "{{ user_details.password }}"
email: "{{ user_details.email }}"
Ansible told me:
Unsupported parameters for (ansible.builtin.user) module: email
The answer was basically in the error.
I could then run:
ansible-doc ansible.builtin.user
and inspect what the module actually supports.
The same idea applies more broadly.
Before changing five things at once:
- Read the error.
- Identify the layer producing it.
- Check the documentation for that component.
- Change one thing that actually relates to the failure.
- Test again.
I wish I could say I followed that perfectly throughout this lab because I did not. But I got better at it.
The Playbook Became My Recovery Mechanism
One thing I appreciated more after rebuilding the nodes was that my configuration was no longer trapped inside the machines.
I could dispose of the machines, but I needed the configuration.
My playbook described what I wanted:
packages installed
Nginx running
service enabled
HTML template deployed
My variables described the values, the template described the webpage, and the inventory described the machines.
If a managed node disappeared, I could create another machine, establish the initial SSH access, add it to inventory and apply the configuration again.
What Breaking the Lab Actually Taught Me
Breaking the lab taught me how Ansible pieces depend on each other and the rest of the system.
The biggest lessons I am keeping are:
Same hostname != same machine
Public key -> managed node
Private key -> stays on control
authorized_keys -> who can authenticate to me?
known_hosts -> which servers do I trust?
Ansible failing != Ansible is necessarily broken
Check the layer underneath the error
Correct time is part of a healthy system
A repeatable playbook makes rebuilding much easier
And probably the biggest one:
Automation does not remove the need to understand the systems underneath it.
If anything, automation makes that understanding more important.
When everything works, one command can configure several machines. When something breaks, I still need to know whether I am looking at Ansible, SSH, Linux, Multipass, QEMU or the host operating system.
That was probably the most useful part of this project.