R
G

Back to list2020-052

Install Proxmox on a Hetzner Root Server

This article explains how to set up Proxmox on a Hetzner Root server using an LVM-thin pool , Nginx and Letsencrypt. Additionally, support for additional Hetzner IPs is added.

Hardware server Hetzner EX51 running Debian 9.12.

Services

  • Proxmox 5 KVM / LXC

Installation Process

OS installation using LVM

  • Activate Rescue System and login as root
  • Run installimage:
    • Choose Debian -> Debian-97-stretch-64-minimal (or latest Proxmox compatible version)
    • Change HOSTNAME to your desired hostname
    • Comment the active PART lines (should be four)
    • Replace with:
PART /boot ext4 1024M
PART lvm pve all

LV pve swap swap swap 8G
LV pve root / ext4 100G

This will create an 8GB swap partition (maybe that's overkill) and a 100GB system partition. The rest will be used by proxmox directly.

Reboot after successful installation.

Install Proxmox

Add Proxmox repository and install

echo "deb http://download.proxmox.com/debian stretch pve-no-subscription" > /etc/apt/sources.list.d/pve-install-repo.list
wget http://download.proxmox.com/debian/proxmox-ve-release-5.x.gpg -O /etc/apt/trusted.gpg.d/proxmox-ve-release-5.x.gpg

apt-get update
apt-get -y dist-upgrade
apt-get -y install proxmox-ve ssh postfix ksm-control-daemon open-iscsi systemd-sysv

Use postfix in Internet Configuration mode.

Reboot after installation.

After the reboot, delete the Proxmox enterprise repository:

rm /etc/apt/sources.list.d/pve-enterprise.list && apt-get update

IMPORTANT: Proxmox may be up and running at this point, but it's advisable to NOT access it directly. The certificate is not verified at this point. Continue until nginx is installed and an official letsencrypt certificate has been installed.

Storage configuration

Next is the creation of an LVM-thin pool for Proxmox to use as a block storage:

lvcreate -l +99%FREE -n data pve
lvconvert --type thin-pool pve/data

Only 99% of the free space is used. LVM needs some space for LVM-thin.

Network configuration

IMPORTANT These are bogous IP addresses. Insert your own IP addresses and device names as applicable.

Create /etc/sysctl.d/99-networking.conf:

net.ipv4.ip_forward=1
net.ipv4.conf.enp0s31f6.send_redirects=0
net.ipv6.conf.all.forwarding=1

Edit /etc/network/interfaces to use the bridges later used by Proxmox.

source /etc/network/interfaces.d/*

auto lo
iface lo inet loopback
iface lo inet6 loopback

auto enp0s31f6
iface enp0s31f6 inet static
      # route 93.120.23.193/26 via 93.120.23.193
      up route add -net 93.120.23.193 netmask 255.255.255.192 gw 94.120.23.194 dev enp0s31f6

iface enp0s31f6 inet6 static
      address 2a01:4f9:2b:1ac7::2
      netmask 64
      gateway fe80::1

# Public IP bridge. Use this for Hetzner IPs.
# Add the IP's MAC address in the setup with vmbr0 and your
# public IP will be set automatically.
auto vmbr0
iface vmbr0 inet static
      address  94.120.23.233
      netmask  255.255.255.192
      gateway  94.120.23.194
      bridge_ports enp0s31f6
      bridge_stp off
      bridge_fd 0

# Internal IP address range.
auto vmbr1
iface vmbr1 inet static
      bridge_ports none
      bridge_fd 0
      bridge_maxwait 0
      address 10.44.0.1
      netmask 255.255.0.0
      bridge_stp off

IMPORTANT: At this point vmbr1 doesn't connect to anywhere but itself. Clients will not have internet connectivity. This will come later during the final firewall setup. However, in case connectivity is needed at this point temporarily add the following two lines to the vmbr1 configuration:

      post-up iptables -t nat -A POSTROUTING -s '10.44.0.0/16' -o vmbr0 -j MASQUERADE
      post-down iptables -t nat -D POSTROUTING -s '10.44.0.0/16' -o vmbr0 -j MASQUERADE

Reverse proxy setup and certificate

This step explains the installation of nginx which is used to provide verification for letsencrypt and will work as a reverse proxy for services which don't have public IPs.

It was attempted to reverse proxy proxmox, however the noVNC console failed to connect through this method.

apt-get -y install nginx certbot

openssl dhparam -out /etc/ssl/dhparams.pem 2048

mkdir -p /srv/http/00_template/{acme,bin,htdocs,log,tmp}
chown -R www-data:www-data /srv/http/00_template/

cp -a /srv/http/00_template/ /srv/http/myserver.domain.com
chown -R www-data:www-data /srv/http/myserver.domain.com

rm /etc/nginx/sites-enabled/default

Create file /etc/nginx/sites-available/myserver.domain.com:

server {
    listen 80;
    server_name myserver.domain.com;

    access_log /srv/http/myserver.domain.com/log/nginx_access.log;
    error_log /srv/http/myserver.domain.com/log/nginx_error.log;

    location /.well-known/acme-challenge {
        root /srv/http/myserver.domain.com/acme/;
        allow all;
    }

    location / {
        rewrite ^ https://$server_name$request_uri? permanent;
    }
}

Start nginx and get a certificate:

cd /etc/nginx/sites-enabled/
ln -s ../sites-available/myserver.domain.com

systemctl restart nginx
certbot certonly -a webroot --webroot-path=/srv/http/myserver.domain.com/acme -d myserver.domain.com

Once the certificates are in, replace the stock certificates of Proxmox:

rm /etc/pve/nodes/atlas/{pve-ssl.key,pve-ssl.pem}

cp /etc/letsencrypt/live/myserver.domain.com/privkey.pem /etc/pve/nodes/atlas/pve-ssl.key
cp /etc/letsencrypt/live/myserver.domain.com/fullchain.pem /etc/pve/nodes/atlas/pve-ssl.pem

systemctl restart pveproxy

Important: Sadly linking is not possible as these files seem to sit in some FUSE based mount. Make sure to copy updated certificates over the expired ones every time when the renew command is run.

Proxmox should now be available securely via https://myserver.domain.com:8006.

Finalize nginx host

Append the SSL block to /etc/nginx/sites-available/myserver.domain.com:

server {
    listen 443;
    server_name myserver.domain.com;
    root /srv/http/myserver.domain.com/htdocs/;

    access_log /srv/http/myserver.domain.com/log/nginx_access_ssl.log;
    error_log /srv/http/myserver.domain.com/log/nginx_error_ssl.log;

    ssl on;
    ssl_certificate /etc/letsencrypt/live/myserver.domain.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/myserver.domain.com/privkey.pem;
    include snippets/ssl.conf;

    # Optiona: Allow only internal networks (e.g. VPN)
    # allow 10.240.0.0/16;
    # deny all;

    index index.html index.htm index.php;

    location / {
        autoindex on;
    }

    # Protect Apache .ht* files
    location ~ /\.ht {
        deny  all;
    }

    location /.well-known/acme-challenge {
        root /srv/http/myserver.domain.com/acme/;
        allow all;
    }

}

Create file /etc/nginx/snippets/ssl.conf:

ssl_prefer_server_ciphers on;
ssl_ciphers 'ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-DSS-AES128-GCM-SHA256:kEDH+AESGCM:ECDHE-RSA-AES128-SHA256:ECDHE-ECDSA-AES128-SHA256:ECDHE-RSA-AES128-SHA:ECDHE-ECDSA-AES128-SHA:ECDHE-RSA-AES256-SHA384:ECDHE-ECDSA-AES256-SHA384:ECDHE-RSA-AES256-SHA:ECDHE-ECDSA-AES256-SHA:DHE-RSA-AES128-SHA256:DHE-RSA-AES128-SHA:DHE-DSS-AES128-SHA256:DHE-RSA-AES256-SHA256:DHE-DSS-AES256-SHA:DHE-RSA-AES256-SHA:AES128-GCM-SHA256:AES256-GCM-SHA384:AES128-SHA256:AES256-SHA256:AES128-SHA:AES256-SHA:AES:CAMELLIA:DES-CBC3-SHA:!aNULL:!eNULL:!EXPORT:!DES:!RC4:!MD5:!PSK:!aECDH:!EDH-DSS-DES-CBC3-SHA:!EDH-RSA-DES-CBC3-SHA:!KRB5-DES-CBC3-SHA';
ssl_session_timeout 1d;
ssl_session_cache shared:SSL:10m;
ssl_stapling on;
ssl_stapling_verify on;
add_header Strict-Transport-Security max-age=15768000;
ssl_dhparam /etc/ssl/dhparams.pem;

Restart and enable nginx:

systemctl restart nginx
systemctl enable nginx

Finalize

updatedb

apt-get -y autoremove
apt-get -y clean

Troubleshooting

Broken network Configuration

It's possible that after messing with the network configuration the system is no longer available via network. If that happens, activate the Hetzner Rescue system and order a reboot in their management system.

Login and fix the config:

mount /dev/mapper/pve-root /mnt
vim /mnt/etc/network/interfaces
...

Using Proxmox

Storing OS ISOs

ISOs can be stored under /var/lib/vz/template/iso. Example:

cd /var/lib/vz/template/iso
wget https://cdimage.debian.org/debian-cd/current/amd64/iso-cd/debian-9.7.0-amd64-netinst.iso

Container Images

Proxmox has it's own set of tools described here.

List available container images with:

pveam available --section system

Download with:

pveam download local debian-9.0-standard_9.7-1_amd64.tar.gz

Check locally stored images:

pveam list local

VMs and containers with multiple interfaces

Having mutliple interfaces (e.g. one public, one internal) causes problems when trying to connect to a VM/container from another internal network. Got it working using the first interface to be the public one (via Hetzner DHCP) and the second to be internal. This works fine to get the hypervisor to connect, however not the rest of the network.

Adding routes for those networks helped:

ip route add 10.111.0.0/16 dev eth1 via 10.44.0.1 ; ip route add 10.200.0.0/16 dev eth1 via 10.44.0.1

With Debian it's easy to automate by adding it to the /etc/network/if-up.d/ directory (e.g. custom-routes):

#!/bin/sh -e
#Establish custom routes needed to make this VM talk with other networks

if [ "$IFACE" = "eth1" ]; then
    ip route add 10.111.0.0/16 dev eth1 via 10.44.0.1
    ip route add 10.200.0.0/16 dev eth1 via 10.44.0.1
fi

exit 0

Remember to set chmod 755 for the file.

External documentation

Proxmox