Jume - My Virtualization Blog

My personal and professional virtualization blog. Everything about VMware, PowerCLI, Powershell, Agile, Scrum, VSAN and Cloud related.

Access VM console in a secured environment - part 1

​Accessing the VM console always requires a direct connection to the ESXi host running the VM. If you've got a user/insecure network and you firewalled your ESXi/vCenter servers, you cannot access the console without opening the port for that whole network (you never know who requires a connection to any of the hosts for that console). This article explains how you can build your own HTML5 console and reroute the WebSockets through HAProxy as a jump host for extra security and logging so you can block all client traffic to your ESXi network.

As a team at my client, we developed this solution during a one-day hackathon. This setup has been running for almost a year now. With over 11.000 VMs, this feature is used the most next to power actions every day. I would like to thank Gijs de Wachter for helping out with the HAProxy config. This article describes not all, but the most important part of the solution.

We're going to create the following situation:

Lab setup with HAProxy
1: Client connects to vCenter, request a ticket for a VM console
2: vCenter generates a ticket and presents to the client including the host where VM is
3: Rewrite the URL to the HAProxy and add the host information in the request
4: Connect to HAProxy to set up the WebSocket tunnel
5: HAProxy forwards WebSockets and present the data back to the client

Lab Setup

Lab setup: VMs
​First, let's take a look at how the lab is set up. I've got all virtual VMs:
  • OPNsense v20.1 firewall
  • Ubuntu 19.10 LTS client
  • VCSA 6.7
  • 3 nested ESXi 6.7 hosts
  • PhotonOS 3.0 running the HAProxy service

I've made 4 networks to keep all functionalities separated. All portgroups are in a different VLAN and have their own dedicated subnets.

  • client = VLAN 101 - subnet 192.168.179.0/26
  • vc = VLAN 110 - subnet 192.168.179.64/26
  • vesxi = VLAN 120 - subnet 192.168.179.128/26
  • proxy = VLAN 130 - subnet 192.168.179.192/26
In OPNsense, I've ensured all networks are labeled and aliases are created:
  • ESXiPorts (443, 902 and 9080) are used for communication between vCenter and ESXi
  • WebTraffic is enabled from all networks to WAN (to patch and update OS)
  • vCenters and vESXiHosts are 'Host(s)' groups to do easier management in the firewall
Full lab setup with all IPs. Domain is 'sec.lab' - all hostnames are registrered in OPNSense.

Custom Console

VMware provides an SDK for the HTML console. You can find it here: https://code.vmware.com/web/sdk/2.1.0/html-console​. You require the SDK so you can use a custom URL (in our case the HAProxy server).
Besides the ZIP file containing the wmks.min.js​ and some css, right in the bottom of that page, there is an "HTML Console demonstration program​" - download that one too, including the documentation. To be able to get the example working, ensure to copy the contents of the SDK to the same folder. Open 'wmks-sdk-example.html​' in your browser and you should see something like this:

Request a ticket

The SDK PDF describes a couple of methods to request a ticket for the console. I like to use PowerShell with PowerCLI:

$vmname = "tinycore01"

Connect-VIServer -server secvcsa01.sec.lab

$vmview = get-view -ViewType "VirtualMachine" -Filter @{"Name" = "^$($vmname)$"} -Property "Name"
$AcquireTicketResult = $vmview.AcquireTicket("webmks")
$AcquireTicketResult 

​Line 1: Define vm name variable
Line 3: Connect to vCenter
Line 4: Get the view object of the VM
Line 5: Acquire the ticket
Line 7: Print the ticket result:

Ticket        : fd809cc7d35c314b
CfgFile       : /vmfs/volumes/5e4b0cdd-35efa4db-a655-005056be05df/tinycore01/tinycore01.vmx
Host          : vesxi01.sec.lab
Port          : 443
SslThumbprint : 0D:68:3E:6C:EF:FB:D3:90:1B:4E:9F:B5:00:84:27:A3:B2:DE:7E:92

First click 'createWMKS', fill in the details in the host, port and ticket and click connect.

​This works great out of the box. However, we're connecting to an ESXi host directly. So let's close the port. On the firewall, I chose to block (reject) any traffic from my client LAN. I picked reject so you don't have to wait until a timeout occurs (in production environments, I would put this to block).

When connecting to the console it will report:

createWMKS successfully!
connect succeeded
onErrorHandler - error type websocketerror
onConnectionStateChange - connectionState: disconnected
reason is , code is1006

Setup HAProxy 

Now we've got this up and running. We can setup HAProxy and block the port. I'm using PhotonOS to setup HAProxy, but you can use any OS / Container of your liking. Ensure the machine has a fixed IP address, a hostname (registered in DNS), and fully updated with the latest security patches, etc.

Installing HAProxy on PhotonOS is easy:

# tdnf install haproxy

Then create some self-signed certificates:

# cd ~
# openssl genrsa -out haproxy01.sec.lab.key 2048
# openssl req -new -key haproxy01.sec.lab.key -out haproxy01.sec.lab.csr

Enter the requested information for validation of the certificate.

# openssl x509 -req -days 365 -in haproxy01.sec.lab.csr -signkey haproxy01.sec.lab.key -out haproxy01.sec.lab.crt

And put the certificate in your private SSL folder:

# cat haproxy01.sec.lab.key haproxy01.sec.lab.crt >> /etc/ssl/private/haproxy01.sec.lab.pem

Open up OS firewall 443 port with IPTABLES:

# iptables -A INPUT -p tcp --dport 443 -m conntrack --ctstate NEW,ESTABLISHED -j ACCEPT
# iptables -A OUTPUT -p tcp --sport 443 -m conntrack --ctstate ESTABLISHED -j ACCEPT

HAProxy config file

​The HAProxy configuration file requires some tweaking. Since I've got 3 ESXi hosts in my lab, I need to define all of them. Here is an example of my configuration file:

global

defaults
	timeout client	30s
	timeout connect	30s
	timeout server	30s

frontend https-in
	bind	*:443 ssl crt /etc/ssl/private/haproxy01.sec.lab.pem
	use_backend console_redirect if { ssl_fc_sni haproxy01.sec.lab } { path_beg /ticket }

backend console_redirect
	use-server vesxi01.sec.lab if { urlp(host) -i vesxi01.sec.lab }
	use-server vesxi02.sec.lab if { urlp(host) -i vesxi02.sec.lab }
	use-server vesxi03.sec.lab if { urlp(host) -i vesxi03.sec.lab }
	
	server vesxi01.sec.lab vesxi01.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi01.sec.lab
	server vesxi02.sec.lab vesxi02.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi02.sec.lab
	server vesxi03.sec.lab vesxi03.sec.lab:443 check check-ssl verify none ssl sni req.hdr(host) check-sni vesxi03.sec.lab
 

1: global: nothing in here
3-6: default timeouts
8-10: frontend part, listen to 443 with ssl certificate, redirect if hostname is haproxy01.sec.lab and path begins with /ticket (will come to that later)
12-15: tells to use a server based on the host parameter in the url (will come to that later)
17-19: connects to the correct esxi server

Creating a new URL

So by default, the 'wmks' code works with a URL like: wss://vesxi01.sec.lab:443/ticket/fd809cc7d35c314b

But we need to reroute it through the HAProxy, we also need to add the host and the ticket, so we should get a URL like: wss://haproxy01.sec.lab:443/ticket/55f58e1f75ef7c58?host=vesxi01.sec.lab

This allows the Web Sockets run through the HAProxy, and adding the parameter ?host ensures HAProxy knows where to route the traffic to. The HAProxy routes the whole URL to the ESXi host, but it only reads the 'ticket' part. The host parameter is ignored.

I use the following code in PowerShell to create this URL:

$vmname = "tinycore01"
$proxy = "haproxy01.sec.lab"

Connect-VIServer -server secvcsa01.sec.lab
$vmview = get-view -ViewType "VirtualMachine" -Filter @{"Name" = "^$($vmname)$"} -Property "Name"
$AcquireTicketResult = $vmview.AcquireTicket("webmks")

$FullUrl = "wss://$($proxy):443/ticket/$($AcquireTicketResult.Ticket)?host=$($AcquireTicketResult.Host)"
$FullUrl 

8: most important part: rewrites the url

Change the example HTML and JS

 On line 84 in 'wmks-sdk-example.html', I replaced:

        <tr>
          <td>host:</td>
          <td><input id="host" type="text" value="127.0.0.1"></td>
        </tr>
        <tr>
          <td>port:</td>
          <td><input id="port" type="text" value="8180"></td>
        </tr>
        <tr>
          <!-- added row for MKS ticket -->
          <td>ticket:</td>
          <td><input id="ticket" type="text" value=""></td>
        </tr> 

with:

        <tr>
          <td>link from script:</td>
          <td><input id="url" type="text" value=""></td>
        </tr> 

And to use this new 'id' url I update the 'wmks-sdk-example.js' file line 245:

  var host = $("#host")[0].value;
  var port = $("#port")[0].value;
  var ticket = $("#ticket")[0].value;
  var url = "ws://" + host + ":" + port;
  if (ticket) {
    var url = "wss://" + host + ":" + port + "/ticket/" + ticket ;
  }
  try { 

with:

  try {
    var url = $("#url")[0].value 

​This gives a nice new box allowing us to paste the URL generated by out PowerShell script:

And if you've configured everything as planned, you should be able to connect!!!

Hurray!!! You are now running your console through the HAProxy!!!

This concludes this post for now (it has become way longer than originally planned). In the next article, I'll explain some basic troubleshooting steps and I'll present some extra scripts and insight.

×
Stay Informed

When you subscribe to the blog, we will send you an e-mail when there are new updates on the site so you wouldn't miss them.

Passed: 2V0-21.19 Professional VMware vSphere 6.7 ...
PowerShell 7.0 GA

Related Posts

 

Comments 5

Guest
Guest - Andreas Braidt on Friday, 05 February 2021 13:50

Hi,
i have the same situation here and tried to setup this with haproxy. i have all up and running but on doing the wmks connect i always get an ssl handshake error on haproxy log. are there any certs needed on haproxy besides the self sigend ones?

Hi, i have the same situation here and tried to setup this with haproxy. i have all up and running but on doing the wmks connect i always get an ssl handshake error on haproxy log. are there any certs needed on haproxy besides the self sigend ones?
Guest
Guest - Andreas Braidt on Wednesday, 24 February 2021 08:51

Hi, i finally got i working with the following config for HAProxy

global
log 127.0.0.1:514 local0
tune.ssl.default-dh-param 2048

defaults
log global
mode http
option httplog
option http-server-close
option dontlognull
option redispatch
option contstats
retries 3
backlog 10000
timeout connect 5s
timeout client 25s
timeout server 25s
timeout tunnel 3600s
timeout http-keep-alive 1s
timeout http-request 15s
timeout queue 30s
timeout tarpit 60s
default-server inter 3s rise 2 fall 3
option forwardfor

frontend https-in
bind *:443 ssl crt /etc/ssl/private/mypem.pem
use_backend console_redirect if { ssl_fc_sni haproxy01.test.de } { path_beg /ticket }

backend console_redirect
use-server test1.test.de if { urlp(host) -i test1.test.de }
use-server test2.test.de if { urlp(host) -i test2.test.de }
use-server test3.test.de if { urlp(host) -i test3.test.de }

server test1.test.de test1.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test1.test.de
server test2.test.de test2.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test2.test.de
server test3.test.de test3.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test3.test.de

Additionally i added the Root CA used by the ESXi/vSphere Servers on the HAProxy OS and ensured that from the client the certificate chain was correctly closed and trusted (use a Public Cert or import the selfsigned into your trusted root).

Also i customized the example page to pass a URL parameter "connectionString=" where i can pass the base64 encoded value of the haproxy URL.
The page is automatically then using this to create the WMKS and starting the connect immediately.
If someone is interested in details, then please let me know.

Hi, i finally got i working with the following config for HAProxy global log 127.0.0.1:514 local0 tune.ssl.default-dh-param 2048 defaults log global mode http option httplog option http-server-close option dontlognull option redispatch option contstats retries 3 backlog 10000 timeout connect 5s timeout client 25s timeout server 25s timeout tunnel 3600s timeout http-keep-alive 1s timeout http-request 15s timeout queue 30s timeout tarpit 60s default-server inter 3s rise 2 fall 3 option forwardfor frontend https-in bind *:443 ssl crt /etc/ssl/private/mypem.pem use_backend console_redirect if { ssl_fc_sni haproxy01.test.de } { path_beg /ticket } backend console_redirect use-server test1.test.de if { urlp(host) -i test1.test.de } use-server test2.test.de if { urlp(host) -i test2.test.de } use-server test3.test.de if { urlp(host) -i test3.test.de } server test1.test.de test1.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test1.test.de server test2.test.de test2.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test2.test.de server test3.test.de test3.test.de:443 check check-ssl verify none ssl sni req.hdr(host) check-sni test3.test.de Additionally i added the Root CA used by the ESXi/vSphere Servers on the HAProxy OS and ensured that from the client the certificate chain was correctly closed and trusted (use a Public Cert or import the selfsigned into your trusted root). Also i customized the example page to pass a URL parameter "connectionString=" where i can pass the base64 encoded value of the haproxy URL. The page is automatically then using this to create the WMKS and starting the connect immediately. If someone is interested in details, then please let me know.
Bouke Groenescheij on Wednesday, 24 February 2021 10:27

That's awesome you figured it out. Thanks for sharing!

That's awesome you figured it out. Thanks for sharing!
Guest
Guest - vladoportos on Monday, 14 February 2022 07:39

Hi, do you have some git repo with your solution ? I would very much like to look at it. Also, is there a way to make it without setting the backend host redirect as fixed ? Let's say I have a dynamic environment where the ESXi hosts get created and deleted at random...

Hi, do you have some git repo with your solution ? I would very much like to look at it. Also, is there a way to make it without setting the backend host redirect as fixed ? Let's say I have a dynamic environment where the ESXi hosts get created and deleted at random...
Guest
Guest - ammarloo on Saturday, 11 December 2021 11:14
Guest
Sunday, 24 September 2023