Jume - My Virtualization Blog
Access VM console in a secured environment - part 1
Accessing the VM console always requires a direct connection to the ESXi host running the VM. If you've got a user/insecure network and you firewalled your ESXi/vCenter servers, you cannot access the console without opening the port for that whole network (you never know who requires a connection to any of the hosts for that console). This article explains how you can build your own HTML5 console and reroute the WebSockets through HAProxy as a jump host for extra security and logging so you can block all client traffic to your ESXi network.
As a team at my client, we developed this solution during a one-day hackathon. This setup has been running for almost a year now. With over 11.000 VMs, this feature is used the most next to power actions every day. I would like to thank Gijs de Wachter for helping out with the HAProxy config. This article describes not all, but the most important part of the solution.
We're going to create the following situation:
Lab setup with HAProxy
1: Client connects to vCenter, request a ticket for a VM console
2: vCenter generates a ticket and presents to the client including the host where VM is
3: Rewrite the URL to the HAProxy and add the host information in the request
4: Connect to HAProxy to set up the WebSocket tunnel
5: HAProxy forwards WebSockets and present the data back to the client
Lab Setup
- OPNsense v20.1 firewall
- Ubuntu 19.10 LTS client
- VCSA 6.7
- 3 nested ESXi 6.7 hosts
- PhotonOS 3.0 running the HAProxy service
I've made 4 networks to keep all functionalities separated. All portgroups are in a different VLAN and have their own dedicated subnets.
- client = VLAN 101 - subnet 192.168.179.0/26
- vc = VLAN 110 - subnet 192.168.179.64/26
- vesxi = VLAN 120 - subnet 192.168.179.128/26
- proxy = VLAN 130 - subnet 192.168.179.192/26
- ESXiPorts (443, 902 and 9080) are used for communication between vCenter and ESXi
- WebTraffic is enabled from all networks to WAN (to patch and update OS)
- vCenters and vESXiHosts are 'Host(s)' groups to do easier management in the firewall
Custom Console
Request a ticket
The SDK PDF describes a couple of methods to request a ticket for the console. I like to use PowerShell with PowerCLI:
$vmname = "tinycore01" Connect-VIServer -server secvcsa01.sec.lab $vmview = get-view -ViewType "VirtualMachine" -Filter @{"Name" = "^$($vmname)$"} -Property "Name" $AcquireTicketResult = $vmview.AcquireTicket("webmks") $AcquireTicketResult
Line 1: Define vm name variable
Line 3: Connect to vCenter
Line 4: Get the view object of the VM
Line 5: Acquire the ticket
Line 7: Print the ticket result:
Ticket : fd809cc7d35c314b
CfgFile : /vmfs/volumes/5e4b0cdd-35efa4db-a655-005056be05df/tinycore01/tinycore01.vmx
Host : vesxi01.sec.lab
Port : 443
SslThumbprint : 0D:68:3E:6C:EF:FB:D3:90:1B:4E:9F:B5:00:84:27:A3:B2:DE:7E:92
This works great out of the box. However, we're connecting to an ESXi host directly. So let's close the port. On the firewall, I chose to block (reject) any traffic from my client LAN. I picked reject so you don't have to wait until a timeout occurs (in production environments, I would put this to block).
When connecting to the console it will report:
createWMKS successfully!
connect succeeded
onErrorHandler - error type websocketerror
onConnectionStateChange - connectionState: disconnected
reason is , code is1006
Setup HAProxy
Now we've got this up and running. We can setup HAProxy and block the port. I'm using PhotonOS to setup HAProxy, but you can use any OS / Container of your liking. Ensure the machine has a fixed IP address, a hostname (registered in DNS), and fully updated with the latest security patches, etc.
Installing HAProxy on PhotonOS is easy:
# tdnf install haproxy
Then create some self-signed certificates:
# cd ~
# openssl genrsa -out haproxy01.sec.lab.key 2048
# openssl req -new -key haproxy01.sec.lab.key -out haproxy01.sec.lab.csr
Enter the requested information for validation of the certificate.
# openssl x509 -req -days 365 -in haproxy01.sec.lab.csr -signkey haproxy01.sec.lab.key -out haproxy01.sec.lab.crt
And put the certificate in your private SSL folder:
# cat haproxy01.sec.lab.key haproxy01.sec.lab.crt >> /etc/ssl/private/haproxy01.sec.lab.pem
Open up OS firewall 443 port with IPTABLES:
# iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED -j ACCEPT
HAProxy config file
The HAProxy configuration file requires some tweaking. Since I've got 3 ESXi hosts in my lab, I need to define all of them. Here is an example of my configuration file:
global defaults timeout client 30s timeout connect 30s timeout server 30s frontend https-in bind *:443 ssl crt /etc/ssl/private/haproxy01.sec.lab.pem use_backend console_redirect if { ssl_fc_sni haproxy01.sec.lab } { path_beg /ticket } backend console_redirect use-server vesxi01.sec.lab if { urlp(host) -i vesxi01.sec.lab } use-server vesxi02.sec.lab if { urlp(host) -i vesxi02.sec.lab } use-server vesxi03.sec.lab if { urlp(host) -i vesxi03.sec.lab } server vesxi01.sec.lab vesxi01.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi01.sec.lab server vesxi02.sec.lab vesxi02.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi02.sec.lab server vesxi03.sec.lab vesxi03.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi03.sec.lab
1: global: nothing in here
3-6: default timeouts
8-10: frontend part, listen to 443 with ssl certificate, redirect if hostname is haproxy01.sec.lab and path begins with /ticket (will come to that later)
12-15: tells to use a server based on the host parameter in the url (will come to that later)
17-19: connects to the correct esxi server
Creating a new URL
So by default, the 'wmks' code works with a URL like: wss://vesxi01.sec.lab:443/ticket/fd809cc7d35c314b
But we need to reroute it through the HAProxy, we also need to add the host and the ticket, so we should get a URL like: wss://haproxy01.sec.lab:443/ticket/55f58e1f75ef7c58?host=vesxi01.sec.lab
This allows the Web Sockets run through the HAProxy, and adding the parameter ?host ensures HAProxy knows where to route the traffic to. The HAProxy routes the whole URL to the ESXi host, but it only reads the 'ticket' part. The host parameter is ignored.
I use the following code in PowerShell to create this URL:
$vmname = "tinycore01" $proxy = "haproxy01.sec.lab" Connect-VIServer -server secvcsa01.sec.lab $vmview = get-view -ViewType "VirtualMachine" -Filter @{"Name" = "^$($vmname)$"} -Property "Name" $AcquireTicketResult = $vmview.AcquireTicket("webmks") $FullUrl = "wss://$($proxy):443/ticket/$($AcquireTicketResult.Ticket)?host=$($AcquireTicketResult.Host)" $FullUrl
8: most important part: rewrites the url
Change the example HTML and JS
On line 84 in 'wmks-sdk-example.html', I replaced:
<tr> <td>host:</td> <td><input id="host" type="text" value="127.0.0.1"></td> </tr> <tr> <td>port:</td> <td><input id="port" type="text" value="8180"></td> </tr> <tr> <!-- added row for MKS ticket --> <td>ticket:</td> <td><input id="ticket" type="text" value=""></td> </tr>
with:
<tr> <td>link from script:</td> <td><input id="url" type="text" value=""></td> </tr>
And to use this new 'id' url I update the 'wmks-sdk-example.js' file line 245:
var host = $("#host")[0].value; var port = $("#port")[0].value; var ticket = $("#ticket")[0].value; var url = "ws://" + host + ":" + port; if (ticket) { var url = "wss://" + host + ":" + port + "/ticket/" + ticket ; } try {
with:
try { var url = $("#url")[0].value
This gives a nice new box allowing us to paste the URL generated by out PowerShell script:
And if you've configured everything as planned, you should be able to connect!!!
Hurray!!! You are now running your console through the HAProxy!!!
This concludes this post for now (it has become way longer than originally planned). In the next article, I'll explain some basic troubleshooting steps and I'll present some extra scripts and insight.
When you subscribe to the blog, we will send you an e-mail when there are new updates on the site so you wouldn't miss them.
Comments 5
Hi,
i have the same situation here and tried to setup this with haproxy. i have all up and running but on doing the wmks connect i always get an ssl handshake error on haproxy log. are there any certs needed on haproxy besides the self sigend ones?
Hi, i finally got i working with the following config for HAProxy
global
log 127.0.0.1:514 local0
tune.ssl.default-dh-param 2048
defaults
log global
mode http
option httplog
option http-server-close
option dontlognull
option redispatch
option contstats
retries 3
backlog 10000
timeout connect 5s
timeout client 25s
timeout server 25s
timeout tunnel 3600s
timeout http-keep-alive 1s
timeout http-request 15s
timeout queue 30s
timeout tarpit 60s
default-server inter 3s rise 2 fall 3
option forwardfor
frontend https-in
bind *:443 ssl crt /etc/ssl/private/mypem.pem
use_backend console_redirect if { ssl_fc_sni haproxy01.test.de } { path_beg /ticket }
backend console_redirect
use-server test1.test.de if { urlp(host) -i test1.test.de }
use-server test2.test.de if { urlp(host) -i test2.test.de }
use-server test3.test.de if { urlp(host) -i test3.test.de }
server test1.test.de test1.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test1.test.de
server test2.test.de test2.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test2.test.de
server test3.test.de test3.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test3.test.de
Additionally i added the Root CA used by the ESXi/vSphere Servers on the HAProxy OS and ensured that from the client the certificate chain was correctly closed and trusted (use a Public Cert or import the selfsigned into your trusted root).
Also i customized the example page to pass a URL parameter "connectionString=" where i can pass the base64 encoded value of the haproxy URL.
The page is automatically then using this to create the WMKS and starting the connect immediately.
If someone is interested in details, then please let me know.
That's awesome you figured it out. Thanks for sharing!
Hi, do you have some git repo with your solution ? I would very much like to look at it. Also, is there a way to make it without setting the backend host redirect as fixed ? Let's say I have a dynamic environment where the ESXi hosts get created and deleted at random...