Silly thing to happen really.
Some system on the instance detected too many connections in a short amount of time – likely due to some automated tasks driven by ansible.
That resulted in iptables dropping all connections to port 22 🙁
Thankfully, with google compute, one can access the serial console via the web UI.
Unfortunately, all of my users have no passwords – this is to ensure that ssh logins are via keypairs only.
The work around was eventually found on https://medium.com/google-cloud/resolving-getting-locked-out-of-a-compute-engine-85800251890b
TLDR;
- Edit the instance
- Tick “Enable connecting to serial ports”
- Scroll down to “Custom metadata”
- Add a new item with the key “startup-script”:
#!/bin/bash
useradd --groups google-sudoers tempuser
echo "tempuser:password" | chpasswd
- Save the instance and hit “Reset”
- When the instance is back up, connect to it via the serial console and login as the “tempuser”
- Fix things!
- Remove the tempuser
- Remove the startup-script from custom metadata