A freshly provisioned VPS gets scanned within minutes of receiving an IP address. Most of what matters can be done before you install anything else at all.
Keys, then no passwords
Create a non-root user, copy your key across, and confirm the key works in a second terminal before you disable password authentication. Locking yourself out of a fresh box is an avoidable rite of passage.
adduser deploy
usermod -aG sudo deploy
rsync --archive --chown=deploy:deploy ~/.ssh /home/deploy
# verify in a SECOND terminal, then:
sed -i 's/^#*PasswordAuthentication.*/PasswordAuthentication no/' /etc/ssh/sshd_config
systemctl restart sshDefault deny
Set the firewall to deny inbound by default and open only what you actually serve. Re-read the rule list a week later. The ports you opened just to test something are still open, and one of them is a database.
Unattended upgrades
Automatic security updates are the highest-value fifteen seconds you will spend on a new server. The failure mode people fear, an update breaking production, is far rarer than the one they ignore: a six-month-old unpatched CVE in a public-facing service.
Backups before you need them
A backup on the same disk is not a backup. It protects you from a bad deploy, not from a dead server. Get a copy off the machine, then restore it once to prove it works.