{"id":270,"date":"2020-02-03T09:40:32","date_gmt":"2020-02-03T09:40:32","guid":{"rendered":"https:\/\/bene.webtopia.ch\/?p=270"},"modified":"2021-10-26T07:29:07","modified_gmt":"2021-10-26T07:29:07","slug":"https-www-altaro-com-vmware-directpath-io-esxi","status":"publish","type":"post","link":"https:\/\/bene.webtopia.ch\/?p=270","title":{"rendered":"https:\/\/www.altaro.com\/vmware\/directpath-io-esxi\/"},"content":{"rendered":"\n

I was recently toying with the idea of setting up a FreeNAS virtual machine on my ESXi 6.5<\/a>\n host. This would give me an alternative shared storage resource to add \nto my testing environment. With that in mind, I sat down and started \nlooking around for recommendations and what not, on how to accomplish \nthe task. Many of the posts I came across, recommended to go for \npassthrough, or passthru, technically known as DirectPath I\/O. My \noriginal thought was to use RDM or a plain old vmdk and attach it to the\n FreeNAS VM.<\/p>\n\n\n\n

The problem with the latter solution is mainly one of performance \nregardless of this being a testing environment. With DirectPath enabled,\n I can give the FreeNAS VM direct access to a couple of unused drives \ninstalled on the host. So, without giving it further thought, I went \nahead and enabled pass-through on the storage controller on my testing \nESXi host.<\/p>\n\n\n\n

And here comes the big disclaimer. DO NOT DO THIS ON A LIVE SYSTEM<\/strong>. You have been warned!<\/p>\n\n\n\n

This post exists for the sole reason of helping some poor soul, out \nthere, to recover from what turned out to be a silly oversight. More to \ncome on this.<\/p>\n\n\n\n

What went wrong?<\/strong><\/h2>\n\n\n\n
\n\n\n\n

The physical host I chose for my grandiose plan, has a single IBEX \nSATA controller with 3 disks attached to it. Nothing fancy. It\u2019s just \nsoftware emulated RAID which ESXi doesn\u2019t even recognize hence the \nnon-utilized drives. I\u2019m only using one drive which I\u2019ve also used to \ninstall ESXi on. Remember, this is a testing environment. On \nthis one drive, there\u2019s also the default local datastore and a couple of\n VMs.<\/p>\n\n\n\n

The drive, by default, is split into a number of partitions during the ESXi installation. Two of the partitions, called bootbank<\/em> and altbootbank<\/em>, hold the ESXi binaries that are loaded into memory during the booting process. I\u2019ll come back to this later.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

Back to the business of enabling DirectPath I\/O<\/a>\n on a PCI card such as a storage controller. In the next screenshot, \nI\u2019ve highlighted the ESXi host in question and labeled the steps taken \nto enable DirectPath for a PCI device. This is enabled by simply ticking\n a box next to the device name. In this case, I enabled pass-through on \nthe Ibex Peak SATA controller. Keep in mind that this is my one and only\n storage controller.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

After enabling DirectPath , you are reminded that the host must be rebooted for the change to take root.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

Again, without giving it much thought, I rebooted the host. Five \nminutes later and the host was back on line. I proceeded to create the \nnew VM for FreeNAS but I quickly found out that I no longer had a \ndatastore where to create it. And it suddenly dawned on me! With \nDirectPath enabled, the hypervisor will no longer have access to the \ndevice. Darn!<\/p>\n\n\n\n

And that\u2019s when I ended up with a bunch of inaccessible VMs on a datastore that was no more.<\/p>\n\n\n\n

At this point, panic would have quickly ensued if this were a live \nsystem but thankfully it was not. I started digging around in vSphere \nclient, and one of first things noticed was that the Ibex storage \nadapter was not listed under Storage Adapters<\/em> irrespective of the number of rescans carried out.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

The logical solution that first came to mind was, disable DirectPath \nI\/O on the storage controller, reboot the host and restore everything to\n its former glory. So I did, rebooted the host and  waited for another 5\n minutes only to discover the host was still missing the datastore and \nstorage adapter.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

The Fix<\/strong><\/h2>\n\n\n\n
\n\n\n\n

The first fix I found on the net is the one described in KB1022011<\/a>. Basically, you SSH to ESXi and inspect \/etc\/vmware\/esx.conf<\/em> looking for the line where a device\u2019s owner<\/em> value is set to passthru<\/em>. This must be changed to vmkernel <\/em>to disable pass-through. To verify the device is the correct one, look at the \/device <\/em>value and compare it to the Device ID<\/em> value in vSphere Web client; Configure -> PCI Devices<\/em> -> Edit<\/em>.<\/p>\n\n\n\n

In this example, the device ID, for the storage controller, is 3b25<\/em> meaning I know I\u2019m changing the correct one. You can see this in the next screenshot.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

To replace the entry, use vi<\/em>, substituting passthru<\/em> with vmkernel<\/em>, which\n is what I did. I rebooted the host a third time and much to my \nannoyance it was still missing the datastore. After some more digging \naround, I came across these two links; The wrong way to use VMware DirectPath<\/a> and KB2043048<\/a>.<\/p>\n\n\n\n

The first link is what helped me fix the issue and also provided me \nwith the inspiration to write this post, so kudos goes to the author.<\/p>\n\n\n\n

So, why did the fix fail?<\/strong><\/h2>\n\n\n\n
\n\n\n\n

The simple explanation is that the changes done, given my setup, \nwon\u2019t persist because everything is running off volatile memory, in RAM \nthat is. This means that once ESXi is rebooted, the change done to esx.conf <\/em>is lost too and it\u2019s back to square one. During the booting process, the ESXi binaries are loaded from partition sda5<\/em> (bootbank) or sda6<\/em> (altbootbank) depending on which one is currently marked active.<\/p>\n\n\n\n

The host\u2019s configuration files are automatically and periodically backed up via script to an archive called state.tgz<\/em>\n which is copied to both partitions depending on which one is active at \nthe time. This backup mechanism is what allows you to revert to a \nprevious state using Shift-R<\/a> while\n ESXi is booting. Unfortunately, in my case, the pass-through change was\n backed up as well and copied to both partitions or so it seemed.<\/p>\n\n\n\n

ESXi has full visibility of the drive while booting up, which \nexplains why it manages to in the first place despite the pass-through \nsetting. It is only when esx.conf<\/em> is read, that the pass-through setting is enforced\/<\/p>\n\n\n\n

Reverting to a previous state using Shift-R<\/em> did not work for me, so I went down the GParted route.<\/p>\n\n\n\n

The GParted Fix<\/strong><\/h2>\n\n\n\n
\n\n\n\n

GParted, GNOME Partition Editor, is a free Linux-based disk \nmanagement utility that has saved my skin on countless occasions. You \ncan create a GParted bootable USB stick by downloading the ISO from here<\/a> and using Rufus<\/a>.<\/p>\n\n\n\n

As mentioned already, we need to change the passthru<\/em> setting in the backed up esx.conf<\/em> file found in the state.tgz<\/em> archive. To do this, we must first uncompress state.tgz<\/em> which contains a second archive, local.tgz.<\/em><\/p>\n\n\n\n

Uncompressing local.tgz<\/em> yields \/etc<\/em> folder where we find esx.conf. <\/em>Once there, open esx.conf<\/em> in an editor. <\/em>Find and change the passthru<\/em> entry and compress the \/etc<\/em> folder back to state.tgz. <\/em><\/p>\n\n\n\n

Finally we overwrite the original state.tgz<\/em> files under \/sda5 and \/sda6 and reboot the host.<\/p>\n\n\n\n

Here\u2019s the whole procedure in step form.<\/p>\n\n\n\n

Step 1<\/strong> \u2013 Boot the ESXi server off the GParted USB stick. Once it\u2019s up, open a terminal window.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

Step 2<\/strong> \u2013 Run the following commands.\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\tcd \/\nmkdir \/mnt\/hd1 \/mnt\/hd2 \/temp\nmount -t vfat \/dev\/sd5 \/mnt\/hd1\nmount -t vfat \/dev\/sd6 \/mnt\/hd2\n\t\t\t\n\t\t\t\t<\/p>\n\n\n\n

\n\t\t\t\t\t1234\n\t\t\t\t<\/td>cd \/mkdir \/mnt\/hd1 \/mnt\/hd2 \/tempmount -t vfat \/dev\/sd5 \/mnt\/hd1mount -t vfat \/dev\/sd6 \/mnt\/hd2<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Step 3<\/strong> \u2013 Determine which state.tgz<\/em> file is \nmost current. Run the following commands and look at the time stamps. \nYou\u2019ll need to uncompress the most recent one to keep using the current \nhost configuration save for the changes we\u2019ll be making.\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\tls -l \/mnt\/hd1\/state.tgz\nls -l \/mnt\/hd2\/state.tgz\n\t\t\t\n\t\t\t\t<\/p>\n\n\n\n

\n\t\t\t\t\t12\n\t\t\t\t<\/td>ls -l \/mnt\/hd1\/state.tgzls -l \/mnt\/hd2\/state.tgz<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Here I\u2019m showing the file while SSH\u2019ed to the host<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n

Step 4<\/strong> \u2013 Copy the most current state.tgz<\/em> to \/temp<\/em>. Here, I\u2019ve assumed that the most current file is the one under \/mnt\/hd1<\/em>. Yours could be different.\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\tcp \/mnt\/hd1\/state.tgz \/temp\n\t\t\t\n\t\t\t\t<\/p>\n\n\n\n

\n\t\t\t\t\t1\n\t\t\t\t<\/td>cp \/mnt\/hd1\/state.tgz \/temp<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n

Step 5<\/strong> \u2013 Uncompress state.tgz<\/em> and the resulting local.tgz<\/em> using tar. You should end up with an etc<\/em> directory as shown.\n\n\t\t\n\t\t\n\t\t\t\n\t\t\t\n\t\t\tcd \/temp\ntar -xf state.tgz\ntar -xf local.tgz\n\t\t\t\n\t\t\t\t<\/p>\n\n\n\n

\n\t\t\t\t\t123\n\t\t\t\t<\/td>cd \/temptar -xf state.tgztar -xf local.tgz<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
\"\"\/<\/figure><\/div>\n\n\n\n

Step 6<\/strong> \u2013 Navigate to \/temp\/etc\/vmware<\/em> and open esx.conf<\/em> in vi<\/em>.<\/p>\n\n\n\n

\"\"\/<\/figure><\/div>\n\n\n\n