Access VM console in a secured environment - part 1

​Accessing the VM console always requires a direct connection to the ESXi host running the VM. If you've got a user/insecure network and you firewalled your ESXi/vCenter servers, you cannot access the console without opening the port for that whole network (you never know who requires a connection to any of the hosts for that console). This article explains how you can build your own HTML5 console and reroute the WebSockets through HAProxy as a jump host for extra security and logging so you can block all client traffic to your ESXi network.

As a team at my client, we developed this solution during a one-day hackathon. This setup has been running for almost a year now. With over 11.000 VMs, this feature is used the most next to power actions every day. I would like to thank Gijs de Wachter for helping out with the HAProxy config. This article describes not all, but the most important part of the solution.

We're going to create the following situation:

Lab setup with HAProxy
1: Client connects to vCenter, request a ticket for a VM console
2: vCenter generates a ticket and presents to the client including the host where VM is
3: Rewrite the URL to the HAProxy and add the host information in the request
4: Connect to HAProxy to set up the WebSocket tunnel
5: HAProxy forwards WebSockets and present the data back to the client

Lab Setup

Lab setup: VMs

• OPNsense v20.1 firewall
  
• Ubuntu 19.10 LTS client
  
• VCSA 6.7
  
• 3 nested ESXi 6.7 hosts
• PhotonOS 3.0 running the HAProxy service

I've made 4 networks to keep all functionalities separated. All portgroups are in a different VLAN and have their own dedicated subnets.

• client = VLAN 101 - subnet 192.168.179.0/26
• vc = VLAN 110 - subnet 192.168.179.64/26
• vesxi = VLAN 120 - subnet 192.168.179.128/26
• proxy = VLAN 130 - subnet 192.168.179.192/26

• ESXiPorts (443, 902 and 9080) are used for communication between vCenter and ESXi
• WebTraffic is enabled from all networks to WAN (to patch and update OS)
• vCenters and vESXiHosts are 'Host(s)' groups to do easier management in the firewall

Custom Console

Request a ticket

The SDK PDF describes a couple of methods to request a ticket for the console. I like to use PowerShell with PowerCLI:

$vmname = "tinycore01"

Connect-VIServer -server secvcsa01.sec.lab

$vmview = get-view -ViewType "VirtualMachine" -Filter @{"Name" = "^$($vmname)$"} -Property "Name"
$AcquireTicketResult = $vmview.AcquireTicket("webmks")
$AcquireTicketResult 

​Line 1: Define vm name variable
Line 3: Connect to vCenter
Line 4: Get the view object of the VM
Line 5: Acquire the ticket
Line 7: Print the ticket result:

Ticket        : fd809cc7d35c314b
CfgFile       : /vmfs/volumes/5e4b0cdd-35efa4db-a655-005056be05df/tinycore01/tinycore01.vmx
Host          : vesxi01.sec.lab
Port          : 443
SslThumbprint : 0D:68:3E:6C:EF:FB:D3:90:1B:4E:9F:B5:00:84:27:A3:B2:DE:7E:92

​This works great out of the box. However, we're connecting to an ESXi host directly. So let's close the port. On the firewall, I chose to block (reject) any traffic from my client LAN. I picked reject so you don't have to wait until a timeout occurs (in production environments, I would put this to block).

When connecting to the console it will report:

createWMKS successfully!
connect succeeded
onErrorHandler - error type websocketerror
onConnectionStateChange - connectionState: disconnected
reason is , code is1006

Setup HAProxy 

Now we've got this up and running. We can setup HAProxy and block the port. I'm using PhotonOS to setup HAProxy, but you can use any OS / Container of your liking. Ensure the machine has a fixed IP address, a hostname (registered in DNS), and fully updated with the latest security patches, etc.

Installing HAProxy on PhotonOS is easy:

# tdnf install haproxy

Then create some self-signed certificates:

# cd ~
# openssl genrsa -out haproxy01.sec.lab.key 2048
# openssl req -new -key haproxy01.sec.lab.key -out haproxy01.sec.lab.csr

Enter the requested information for validation of the certificate.

# openssl x509 -req -days 365 -in haproxy01.sec.lab.csr -signkey haproxy01.sec.lab.key -out haproxy01.sec.lab.crt

And put the certificate in your private SSL folder:

# cat haproxy01.sec.lab.key haproxy01.sec.lab.crt >> /etc/ssl/private/haproxy01.sec.lab.pem

Open up OS firewall 443 port with IPTABLES:

# iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED -j ACCEPT

HAProxy config file

​The HAProxy configuration file requires some tweaking. Since I've got 3 ESXi hosts in my lab, I need to define all of them. Here is an example of my configuration file:

global

defaults
	timeout client	30s
	timeout connect	30s
	timeout server	30s

frontend https-in
	bind	*:443 ssl crt /etc/ssl/private/haproxy01.sec.lab.pem
	use_backend console_redirect if { ssl_fc_sni haproxy01.sec.lab } { path_beg /ticket }

backend console_redirect
	use-server vesxi01.sec.lab if { urlp(host) -i vesxi01.sec.lab }
	use-server vesxi02.sec.lab if { urlp(host) -i vesxi02.sec.lab }
	use-server vesxi03.sec.lab if { urlp(host) -i vesxi03.sec.lab }
	
	server vesxi01.sec.lab vesxi01.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi01.sec.lab
	server vesxi02.sec.lab vesxi02.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi02.sec.lab
	server vesxi03.sec.lab vesxi03.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi03.sec.lab
 

1: global: nothing in here
3-6: default timeouts
8-10: frontend part, listen to 443 with ssl certificate, redirect if hostname is haproxy01.sec.lab and path begins with /ticket (will come to that later)
12-15: tells to use a server based on the host parameter in the url (will come to that later)
17-19: connects to the correct esxi server

Creating a new URL

So by default, the 'wmks' code works with a URL like: wss://vesxi01.sec.lab:443/ticket/fd809cc7d35c314b

But we need to reroute it through the HAProxy, we also need to add the host and the ticket, so we should get a URL like: wss://haproxy01.sec.lab:443/ticket/55f58e1f75ef7c58?host=vesxi01.sec.lab

This allows the Web Sockets run through the HAProxy, and adding the parameter ?host ensures HAProxy knows where to route the traffic to. The HAProxy routes the whole URL to the ESXi host, but it only reads the 'ticket' part. The host parameter is ignored.

I use the following code in PowerShell to create this URL:

$vmname = "tinycore01"
$proxy = "haproxy01.sec.lab"

Connect-VIServer -server secvcsa01.sec.lab
$vmview = get-view -ViewType "VirtualMachine" -Filter @{"Name" = "^$($vmname)$"} -Property "Name"
$AcquireTicketResult = $vmview.AcquireTicket("webmks")

$FullUrl = "wss://$($proxy):443/ticket/$($AcquireTicketResult.Ticket)?host=$($AcquireTicketResult.Host)"
$FullUrl 

8: most important part: rewrites the url

Change the example HTML and JS

 On line 84 in 'wmks-sdk-example.html', I replaced:

        <tr>
          <td>host:</td>
          <td><input id="host" type="text" value="127.0.0.1"></td>
        </tr>
        <tr>
          <td>port:</td>
          <td><input id="port" type="text" value="8180"></td>
        </tr>
        <tr>
          <!-- added row for MKS ticket -->
          <td>ticket:</td>
          <td><input id="ticket" type="text" value=""></td>
        </tr> 

with:

        <tr>
          <td>link from script:</td>
          <td><input id="url" type="text" value=""></td>
        </tr> 

And to use this new 'id' url I update the 'wmks-sdk-example.js' file line 245:

  var host = $("#host")[0].value;
  var port = $("#port")[0].value;
  var ticket = $("#ticket")[0].value;
  var url = "ws://" + host + ":" + port;
  if (ticket) {
    var url = "wss://" + host + ":" + port + "/ticket/" + ticket ;
  }
  try { 

with:

  try {
    var url = $("#url")[0].value 

​This gives a nice new box allowing us to paste the URL generated by out PowerShell script:

And if you've configured everything as planned, you should be able to connect!!!

Hurray!!! You are now running your console through the HAProxy!!!

This concludes this post for now (it has become way longer than originally planned). In the next article, I'll explain some basic troubleshooting steps and I'll present some extra scripts and insight.