VMs with Virtual SCSI Bus Sharing

If you’ve ever spent time patching a set of ESXi hosts, the following scenario will likely feel all too familiar.

You kick off the deployment of a new cluster image in a VMware HA cluster. Everything starts smoothly, updates are rolling out, hosts are entering maintenance mode as expected, and for a moment it feels like a smooth upgrade. But then, suddenly, progress comes to a halt. One of the hosts refuses to enter maintenance mode, and the entire update process gets stuck.

In most cases, the root cause isn’t immediately obvious. You might start checking DRS settings, VM migrations, or resource constraints. However, more often than not, the issue comes down to a virtual machine that cannot be migrated due to a dependency on the host. A common cause here is a VM configured with Virtual SCSI Bus Sharing.

From my experience, when this happens, there’s usually one or multiple VMs configured with Virtual SCSI Bus Sharing enabled. Because of this configuration, vMotion is restricted, and the VM becomes “pinned” to its current host. As a result, the host cannot evacuate all workloads and is therefore unable to enter maintenance mode, blocking your patching workflow.

To avoid running into this situation mid-update, it would be incredibly helpful to have visibility into these VMs beforehand. Imagine having a simple report that lists all VMs with Virtual SCSI Bus Sharing enabled across your environment. You could proactively address potential blockers, scheduling downtime, or planning the update sequence more carefully. With that insight and with a bit of help from Claude AI, this script was created.

In short, a bit of preparation goes a long way. Identifying these special cases in advance can save you from frustrating interruptions and ensure your ESXi patching process runs smoothly from start to finish.

The Find-SharedSCSI-BusOnly.ps1 script generates this overview and works with vCenter read-only permissions.


<# 
.SYNOPSIS
  Report VMs that have SCSI controllers with Bus Sharing enabled (Virtual/Physical).

.DESCRIPTION
  - Scans VMs (optionally scoped by Datacenter(s) and Cluster(s)).
  - Detects only SCSI controllers with BusSharing set to Virtual or Physical.
  - Outputs a dark-themed HTML report and opens it in Edge (preferred), or Chrome/Firefox.

.NOTES
    Script  : Find-SharedSCSI-BusOnly.ps1
    Version : 1.0
    Author  : Vincent Jansen
    Blog    : https://www.vrmware.nl
    Tested  : VMware vCenter 8.0.x

.CHANGELOG
      v1.0.0  2026-05-19  VJ  Initial release

.PARAMETER vCenter
  One or more vCenter FQDN/IPs to connect to. If omitted, uses any existing PowerCLI session.

.PARAMETER Cluster
  One or more cluster names to scope the check. If omitted, all clusters.

.PARAMETER Datacenter
  One or more datacenter names to scope the check. If omitted, all datacenters.

.PARAMETER HtmlPath
  Output path for the HTML report (default: .\SharedSCSIReport.html)

.PARAMETER Credential
  PSCredential used to connect to vCenter(s). Overrides UserName/Password if supplied.

.PARAMETER UserName
  vCenter username (used if -Credential not provided). If provided without -Password, a prompt will appear.

.PARAMETER Password
  SecureString password for the given -UserName.

.EXAMPLE
  .\Find-SharedSCSI-BusOnly.ps1 -vCenter vcsa01.lab.local -Cluster "Prod-HA-01" -Username "User01@lab.local"

.NOTES
  Requires VMware.PowerCLI. Run: Install-Module VMware.PowerCLI
#>

[CmdletBinding()]
param(
  [string[]] $vCenter,
  [string[]] $Cluster,
  [string[]] $Datacenter,
  [string]   $HtmlPath = ".\SharedSCSIReport.html",

  [Parameter(ValueFromPipelineByPropertyName=$true)]
  [System.Management.Automation.PSCredential] $Credential,

  [string] $UserName,
  [SecureString] $Password
)

# --- Preconditions -----------------------------------------------------------
if (-not (Get-Module -ListAvailable -Name VMware.PowerCLI)) {
  Write-Error "VMware.PowerCLI module not found. Install with: Install-Module VMware.PowerCLI"
  exit 1
}

Set-PowerCLIConfiguration -Scope User -ParticipateInCEIP $false -InvalidCertificateAction Ignore -Confirm:$false | Out-Null

# --- Build credential if needed ----------------------------------------------
if (-not $Credential) {
  if ($UserName) {
    if (-not $Password) {
      $Password = Read-Host ("Enter password for {0}" -f $UserName) -AsSecureString
    }
    try {
      $Credential = New-Object System.Management.Automation.PSCredential ($UserName, $Password)
    } catch {
      Write-Error ("Failed to construct PSCredential for {0}: {1}" -f $UserName, $_.Exception.Message)
      exit 1
    }
  }
}

# --- vCenter connections (optional) ------------------------------------------
if ($vCenter) {
  foreach ($vc in $vCenter) {
    $already = $null
    try { $already = Get-VIServer -Server $vc -ErrorAction Stop } catch {}
    if (-not $already) {
      try {
        Write-Verbose ("Connecting to {0} ..." -f $vc)
        if ($Credential) {
          Connect-VIServer -Server $vc -Credential $Credential -ErrorAction Stop | Out-Null
        } else {
          Connect-VIServer -Server $vc -ErrorAction Stop | Out-Null
        }
      } catch {
        Write-Warning ("Failed to connect to {0}: {1}" -f $vc, $_.Exception.Message)
      }
    }
  }
}

# --- Scope selection ---------------------------------------------------------
$dcScope = if ($Datacenter) { Get-Datacenter -Name $Datacenter -ErrorAction SilentlyContinue } else { Get-Datacenter -ErrorAction SilentlyContinue }
if (-not $dcScope) { Write-Warning "No datacenters found in the current session/scope." }

$clusterScope = @()
foreach ($dc in $dcScope) {
  if ($Cluster) {
    $clusterScope += Get-Cluster -Location $dc -Name $Cluster -ErrorAction SilentlyContinue
  } else {
    $clusterScope += Get-Cluster -Location $dc -ErrorAction SilentlyContinue
  }
}

$vms = if ($clusterScope) { $clusterScope | Get-VM -ErrorAction SilentlyContinue } else { Get-VM -ErrorAction SilentlyContinue }
if (-not $vms) { Write-Warning "No VMs found in the current scope."; $vms = @() }

# --- Helper: SCSI controller info only ---------------------------------------
function Get-ScsiControllerInfo {
  param([VMware.VimAutomation.ViCore.Impl.V1.Inventory.VirtualMachineImpl] $Vm)

  $hw = $Vm.ExtensionData.Config.Hardware
  if (-not $hw) { return @() }

  $controllers = $hw.Device | Where-Object {
    $_ -is [VMware.Vim.ParaVirtualSCSIController]     -or
    $_ -is [VMware.Vim.VirtualLsiLogicController]     -or
    $_ -is [VMware.Vim.VirtualLsiLogicSASController]  -or
    $_ -is [VMware.Vim.VirtualBusLogicController]
  }

  foreach ($c in $controllers) {
    $busSharingPretty = switch ($c.SharedBus) {
      'noSharing'       {'NoSharing'}
      'virtualSharing'  {'Virtual'}
      'physicalSharing' {'Physical'}
      default           { [string]$c.SharedBus }
    }

    [pscustomobject]@{
      VM              = $Vm.Name
      ControllerKey   = $c.Key
      ControllerType  = $c.GetType().Name
      BusNumber       = $c.BusNumber
      BusSharing      = $busSharingPretty
    }
  }
}

# --- Analysis: only BusSharing findings --------------------------------------
Write-Verbose "Inspecting SCSI controllers (bus sharing only)..."
$results = New-Object System.Collections.Generic.List[object]

foreach ($vm in $vms) {
  try {
    $ctrls = Get-ScsiControllerInfo -Vm $vm
    $sharedBusCtrls = $ctrls | Where-Object { $_.BusSharing -and $_.BusSharing -ne 'NoSharing' }
    if ($sharedBusCtrls) {
      $dcName      = ($vm | Get-Datacenter -ErrorAction SilentlyContinue | Select-Object -ExpandProperty Name -First 1)
      $clusterName = ($vm | Get-Cluster -ErrorAction SilentlyContinue | Select-Object -ExpandProperty Name -First 1)
      foreach ($c in $sharedBusCtrls) {
        $results.Add([pscustomobject]@{
          Datacenter     = $dcName
          Cluster        = $clusterName
          VM             = $vm.Name
          PowerState     = $vm.PowerState
          ControllerType = $c.ControllerType
          BusNumber      = $c.BusNumber
          BusSharing     = $c.BusSharing
        })
      }
    }
  } catch {
    Write-Warning ("Error analyzing VM '{0}': {1}" -f $vm.Name, $_.Exception.Message)
  }
}

# --- Build HTML (dark theme) -------------------------------------------------
$now = Get-Date
$findings      = $results | Sort-Object Cluster, VM, BusNumber
$totalFindings = $findings.Count
$totalVMs      = ($findings | Select-Object -ExpandProperty VM -Unique).Count
$totalClusters = ($findings | Select-Object -ExpandProperty Cluster -Unique).Count

$clusterAgg = $findings | Group-Object Cluster | ForEach-Object {
  [pscustomobject]@{
    Cluster  = $_.Name
    VMs      = ($_.Group | Select-Object -ExpandProperty VM -Unique).Count
    Findings = $_.Count
  }
} | Sort-Object Cluster

$css = @'
:root {
  color-scheme: dark;
  --bg: #0f1115; --panel: #141821; --text: #e5e7eb; --muted: #9ca3af;
  --accent: #60a5fa; --border: #1f2937; --row: #0b0e14; --ok: #34d399; --danger:#f87171;
}
*{box-sizing:border-box}
body{margin:0;padding:24px;background:var(--bg);color:var(--text);font-family:system-ui,-apple-system,Segoe UI,Roboto,Ubuntu,Cantarell,Noto Sans,Arial,sans-serif}
h1,h2{margin:0 0 12px 0} h1{font-size:22px} h2{font-size:18px;color:var(--muted)}
.panel{background:var(--panel);border:1px solid var(--border);border-radius:10px;padding:16px;margin-bottom:16px}
.grid{display:grid;gap:16px;grid-template-columns:repeat(auto-fit,minmax(240px,1fr))}
.stat{display:flex;align-items:center;justify-content:space-between;padding:14px;background:#0c1220;border:1px solid var(--border);border-radius:10px}
.stat .label{color:var(--muted);font-size:12px} .stat .value{font-size:22px;font-weight:700}
.table-wrap{overflow:auto;max-height:60vh;border:1px solid var(--border);border-radius:10px}
table{width:100%;border-collapse:collapse}
thead th{position:sticky;top:0;background:#111827;color:var(--muted);text-align:left;padding:10px;font-weight:600;font-size:12px;cursor:pointer;user-select:none;border-bottom:1px solid var(--border)}
tbody td{padding:10px;border-bottom:1px solid var(--border);font-size:13px}
tbody tr:nth-child(even){background:var(--row)}
.badge{padding:2px 8px;border-radius:999px;font-size:12px;border:1px solid var(--border)}
.badge.on{color:var(--ok);background:rgba(52,211,153,.08);border-color:rgba(52,211,153,.4)}
.badge.off{color:var(--danger);background:rgba(248,113,113,.08);border-color:rgba(248,113,113,.4)}
.small{font-size:12px;color:var(--muted)}
.controls{display:grid;gap:12px;grid-template-columns:repeat(auto-fit,minmax(220px,1fr))}
'@

$js = @'
(function(){
  const q = s => document.querySelector(s);
  const rows = Array.from(document.querySelectorAll("tbody tr"));
  let sortDir = 1, sortCol = -1;
  function text(el){ return (el?.textContent || "").trim().toLowerCase(); }
  function applyFilter(){
    const fCluster = q("#fCluster").value.toLowerCase();
    const fVM      = q("#fVM").value.toLowerCase();
    let visible = 0;
    rows.forEach(tr=>{
      const c = tr.children;
      const cluster = text(c[1]);
      const vm      = text(c[2]);
      const ok = (fCluster==="" || cluster.includes(fCluster))
              && (fVM===""      || vm.includes(fVM));
      tr.style.display = ok ? "" : "none";
      if(ok) visible++;
    });
    q("#visibleCount").textContent = visible;
  }
  function sortBy(colIdx){
    const tbody = rows[0]?.parentElement; if(!tbody) return;
    if(sortCol === colIdx){ sortDir = -sortDir; } else { sortCol = colIdx; sortDir = 1; }
    const sorted = rows.slice().sort((a,b)=>{
      const A = text(a.children[colIdx]), B = text(b.children[colIdx]);
      const nA = parseFloat(A.replace(/[^0-9.\-]/g,"")), nB = parseFloat(B.replace(/[^0-9.\-]/g,""));
      const bothNum = !isNaN(nA) && !isNaN(nB) && A.match(/^[\d .\-]+$/) && B.match(/^[\d .\-]+$/);
      return (bothNum ? (nA-nB) : A.localeCompare(B)) * sortDir;
    });
    sorted.forEach(tr=>tbody.appendChild(tr));
  }
  ["fCluster","fVM"].forEach(id => q("#"+id).addEventListener("input", applyFilter));
  document.querySelectorAll("thead th").forEach((th,i)=> th.addEventListener("click", ()=>sortBy(i)));
  applyFilter();
})();
'@

function Escape-Html {
  param([AllowNull()] [string] $s)
  if ($null -eq $s) { return "" }
  ($s -replace '&','&amp;' -replace '<','&lt;' -replace '>','&gt;' -replace '"','&quot;' -replace "'","&#39;")
}

# Build rows (Bus-sharing only)
$tbody = New-Object System.Text.StringBuilder
foreach ($r in $findings) {
  $pstate = if ($r.PowerState -eq 'PoweredOn') { '<span class="badge on">On</span>' } else { '<span class="badge off">Off</span>' }
  [void]$tbody.AppendLine( ("<tr>" +
    "<td>{0}</td>" +   # Datacenter
    "<td>{1}</td>" +   # Cluster
    "<td>{2}</td>" +   # VM
    "<td>{3}</td>" +   # Power
    "<td>{4}</td>" +   # ControllerType
    "<td>{5}</td>" +   # Bus#
    "<td>{6}</td>" +   # BusSharing
    "</tr>"
  ) -f (
    (Escape-Html $r.Datacenter),
    (Escape-Html $r.Cluster),
    (Escape-Html $r.VM),
    $pstate,
    (Escape-Html $r.ControllerType),
    (Escape-Html $r.BusNumber),
    (Escape-Html $r.BusSharing)
  ))
}

# Cluster summary rows
$clusterRows = New-Object System.Text.StringBuilder
foreach ($c in $clusterAgg) {
  [void]$clusterRows.AppendLine( ("<tr><td>{0}</td><td>{1}</td><td>{2}</td></tr>" -f (Escape-Html $c.Cluster), $c.VMs, $c.Findings) )
}

# HTML
$html = @"
<!doctype html>
<html lang="en">
<head>
  <meta charset="utf-8">
  <title>Shared SCSI Bus Report</title>
  <meta name="viewport" content="width=device-width, initial-scale=1">
  <style>$css</style>
</head>
<body>
  <div class="panel">
    <h1>Shared SCSI Bus Report</h1>
    <h2>Generated: $($now) </h2>
    <div class="grid" style="margin-top:12px;">
      <div class="stat"><div><div class="label">Findings</div><div class="value" id="visibleCount">$totalFindings</div></div></div>
      <div class="stat"><div><div class="label">Unique VMs</div><div class="value">$totalVMs</div></div></div>
      <div class="stat"><div><div class="label">Clusters</div><div class="value">$totalClusters</div></div></div>
    </div>
  </div>

  <div class="panel">
    <h2>Filters</h2>
    <div class="controls">
      <input id="fCluster" type="text" placeholder="Filter by Cluster...">
      <input id="fVM" type="text" placeholder="Filter by VM name...">
    </div>
  </div>

  <div class="panel">
    <h2>Cluster Summary</h2>
    <div class="table-wrap">
      <table>
        <thead>
          <tr><th>Cluster</th><th>Unique VMs</th><th>Findings</th></tr>
        </thead>
        <tbody>
          $clusterRows
        </tbody>
      </table>
    </div>
  </div>

  <div class="panel">
    <h2>Detailed Findings</h2>
    <div class="table-wrap">
      <table>
        <thead>
          <tr>
            <th>Datacenter</th>
            <th>Cluster</th>
            <th>VM</th>
            <th>Power</th>
            <th>ControllerType</th>
            <th>Bus#</th>
            <th>BusSharing</th>
          </tr>
        </thead>
        <tbody>
          $tbody
        </tbody>
      </table>
    </div>
    <div class="small">Click a column header to sort. Use filters above to narrow results.</div>
  </div>

  <script>$js</script>
</body>
</html>
"@

# Write HTML
try {
  $fullPath = (Resolve-Path -Path $HtmlPath -ErrorAction SilentlyContinue)
  if (-not $fullPath) {
    $dir = Split-Path -Path $HtmlPath -Parent
    if ($dir -and -not (Test-Path $dir)) { New-Item -ItemType Directory -Path $dir -Force | Out-Null }
    $html | Out-File -FilePath $HtmlPath -Encoding UTF8
    $fullPath = Resolve-Path -Path $HtmlPath
  } else {
    $html | Out-File -FilePath $fullPath -Encoding UTF8
  }
  Write-Host ("HTML report saved to: {0}" -f $fullPath.Path)
} catch {
  Write-Warning ("Failed to write HTML to '{0}': {1}" -f $HtmlPath, $_.Exception.Message)
  return
}

# --- Open in browser ---------------------------------------------------------
function Open-InBrowser {
  param([string]$Path)
  $candidates = @(
    "$Env:ProgramFiles (x86)\Microsoft\Edge\Application\msedge.exe",
    "$Env:ProgramFiles\Microsoft\Edge\Application\msedge.exe",
    "$Env:LocalAppData\Microsoft\Edge\Application\msedge.exe",
    "$Env:ProgramFiles\Google\Chrome\Application\chrome.exe",
    "$Env:ProgramFiles (x86)\Google\Chrome\Application\chrome.exe",
    "$Env:LocalAppData\Google\Chrome\Application\chrome.exe",
    "$Env:ProgramFiles\Mozilla Firefox\firefox.exe",
    "$Env:ProgramFiles (x86)\Mozilla Firefox\firefox.exe",
    "$Env:LocalAppData\Mozilla Firefox\firefox.exe"
  )
  foreach ($exe in $candidates) {
    if (Test-Path $exe) { Start-Process -FilePath $exe -ArgumentList ("`"{0}`"" -f $Path); return $true }
  }
  try { Start-Process $Path; return $true } catch { return $false }
}
$ok = Open-InBrowser -Path $fullPath.Path
if (-not $ok) { Write-Warning ("Could not open the report automatically. Please open: {0}" -f $fullPath.Path) }

Once the data is gathered, it generates a HTML report that provides a complete overview at a glance.


The script provides a clear, proactive, and repeatable way to identify VMs with Virtual SCSI Bus Sharing, preventing maintenance mode issues and keeping your ESXi patching process running smoothly.

This script is also available in my GitHub repositiry.

Automating NIC Firmware Inventory Across VMware Clusters with PowerShell

Keeping firmware versions consistent across VMware hosts is crucial for stability, performance, and security. Yet, identifying which ESXi hosts are running which network interface firmware versions can quickly become a tedious task, especially in environments with multiple clusters and dozens of servers.

Recently, I faced exactly this challenge. I needed a clear and reliable overview of the installed NIC firmware versions across all hosts in specific VMware clusters. Rather than performing the checks manually (or clicking through countless vSphere UI tabs), I decided to automate the process.

VMware environments often contain a mix of hardware models, driver versions, and firmware levels. These variables play a major role in network reliability and performance. When a firmware update is released or when troubleshooting network issues, it’s important to know:

  • Which hosts are running outdated firmware?
  • Which clusters might be at risk due to inconsistent versions?

However, VMware doesn’t provide a quick, consolidated overview of NIC firmware versions per cluster out of the box.

To streamline this task, I wrote a PowerShell script that retrieves the NIC firmware versions for each host in a VMware cluster. Using PowerCLI, the script collects:

  • vCenter & Cluster name on top of the report
  • Hostname
  • Server Model
  • NIC
  • NIC Model
  • Firmware Version

Once the data is gathered, it generates a HTML report that provides a complete overview at a glance.

Below is the script used to build this overview.


#Version 1.0
#2026-03-28
#Check NIC Firmware script with HTML output
#Adds vCenter, Cluster & Server Model to HTML report

# ============================
#  Ask user for vCenter & Cluster
# ============================
$vCenter = Read-Host "Enter vCenter Server name or IP"
$cluster  = Read-Host "Enter Cluster name"

Write-Host "Using vCenter: $vCenter" -ForegroundColor Cyan
Write-Host "Using Cluster: $cluster" -ForegroundColor Cyan

# ============================
#  Function: Check NIC Firmware
# ============================
function CheckNICFirmware {
    Param (
        $ESXi
    )

    $esxcli = Get-EsxCli -V2 -VMHost $ESXi
    $result = ""

    # Get Server Model
    $serverModel = (Get-View -Id $ESXi.ExtensionData.MoRef).Hardware.SystemInfo.Model

    # Get NIC list
    $nicList = $esxcli.network.nic.list.Invoke()

    foreach ($nic in $nicList) {

        $nicName = $nic.Name
        $nicGet  = $esxcli.network.nic.get.Invoke(@{nicname=$nicName})

        $model = $nic.Description
        $firmwareVersion = $nicGet.DriverInfo.FirmwareVersion

        $rowColor = "#e6f0ff"   # Light blue row

        $result += "<tr style='background-color: $rowColor;'>
                        <td>$($ESXi.Name)</td>
                        <td>$serverModel</td>
                        <td>$nicName</td>
                        <td>$model</td>
                        <td>$firmwareVersion</td>
                    </tr>"
    }

    return $result
}

# ============================
#  Connect to vCenter
# ============================
Try {Disconnect-VIServer * -Confirm:$false -ErrorAction SilentlyContinue | Out-Null}
Catch {}

Connect-VIServer $vCenter

$ESXis = Get-Cluster -Name $cluster | Get-VMHost | Sort-Object Name | Where-Object {
    $_.ConnectionState -eq 'Connected' -or $_.ConnectionState -eq 'Maintenance'
}

# ============================
#  HTML: Header
# ============================
$html = @"
<html>
<head>
    <title>NIC Firmware Report - vCenter: $vCenter | Cluster: $cluster</title>
    <style>
        table { width: 100%; border-collapse: collapse; margin-bottom: 30px; }
        th, td { border: 1px solid black; padding: 8px; text-align: left; }
        th { background-color: #f2f2f2; }
        h1 { margin-bottom: 5px; }
        .subheader { color: #555; font-size: 18px; margin-bottom: 20px; }
    </style>
</head>
<body>
    <h1>NIC Firmware Report</h1>
    <div class="subheader">
        vCenter Server: <b>$vCenter</b><br>
        Cluster: <b>$cluster</b>
    </div>

    <table>
        <tr>
            <th>Host</th>
            <th>Server Model</th>
            <th>NIC</th>
            <th>NIC Model</th>
            <th>Firmware Version</th>
        </tr>
"@

# ============================
#  NIC Firmware Section
# ============================
foreach ($ESXi in $ESXis) {
    $html += CheckNICFirmware -ESXi $ESXi
}

$html += "</table>"

# ============================
#  HTML Footer
# ============================
$creationDate = Get-Date -Format "yyyy-MM-dd HH:mm:ss"

$html += @"
<p>Report generated on: $creationDate</p>
</body>
</html>
"@

# ============================
#  Output Report
# ============================
$outputPath = "C:\Scripts\NIC_Firmware_Report.html"
$html | Out-File -FilePath $outputPath

Start-Process "msedge.exe" $outputPath

Write-Host "Disconnecting from $vCenter..."
Disconnect-VIServer -Confirm:$False | Out-Null

Write-Host "Report generated at $outputPath" -ForegroundColor Green

The final output is an automatically generated HTML page containing a complete NIC firmware inventory per VMware cluster.


It provides a quick, accurate, and repeatable way to check firmware versions across an environment, saving time and avoiding configuration drift.

VxRail 7.0.300 GA

What’s new in VxRail 7.0.300

VxRail software version 7.0.300 includes VMware ESXi 7.0 Update 3, VMware vSAN 7.0 Update 3 and VMware vCSA 7.0
Update 3a with support for external storage and introduction to satellite nodes.

New features

Operationalize the edge with VxRail satellite nodes:
You can deploy the E660, E660F, and V670F as single VMware vSphere nodes with no VMware vSAN to address VxRail edge deployments that require a smaller footprint. You can configure satellite nodes with an optional PowerEdge RAID controller to add resiliency for local disks. The satellite nodes are managed by a new or existing standard cluster with VMware vSAN running 7.0.300.

Control satellite nodes from a central location:
You can deploy a VxRail Manager VM that can control all satellite nodes from a centralized host management location in VMware vCenter. You can add, remove, and update satellite nodes from one access point using VxRail Manager.

Expanded storage option for VxRail dynamic nodes:
You can deploy VxRail dynamic nodes as part of a PowerFlex 2-layer architecture. Deploy VxRail dynamic nodes cluster as compute only node leveraging PowerFlex storage for hosting the workload VMs.

Protocol support for VxRail dynamic nodes:
NVMe-FC is supported with PowerStore and PowerMax storage arrays that are attached to dynamic nodes.

VMware ESXi 7.0 Update 3, VMware vSAN 7.0 U3, VMware vCSA 7.0 Update 3a support. The major changes for VxRail include:
Support upgrade of the VMware vSAN Witness Host (dedicated) in vLCM as part of the coordinated cluster remediation workflow for VMware vSAN 2-Node and Stretched Clusters.

  1. Stretched Cluster Enhancement to allow the ability to tolerate planned or unplanned downtime of a site and the witness in a stretched cluster deployment.
  2. Nest Fault Domain in a 2-node configuration
  3. Easy VMware vSAN cluster shutdown and start-up
  4. Upgrade note for VxRail with external storage

Source: https://dl.dell.com/content/docu98130

VMware vCLS datastore selection part 2

Last year I wrote an blog post about the VMware vCLS datastore selection. This blog post is one of the most read articles on my website. This does indicate that there is a need to be able to choose a datastore on which the vCLS vms are placed.

Today VMware announced vSphere 7.0 update 3. In this update there is also an improvement on the vCLS datastore selection. It’s now possible to choose the datastore on which the vCLS vms should be located.

In the following video on the VMware vSphere YouTube channel move on to 20 minutes to learn more about the vCLS vms datastore selection improvement.

Another improvement is that the vCLS vms now have a unique identifier. This is useful when you have multiple clusters managed by the same vCenter.

It’s always good to see that a vendor is listening to the customers’ needs to further improve a product.

Cannot install the vCenter Server agent(HA) service. Unknown installer error

It had been a while since I have installed a non HCI VMware cluster. After installing the ESXi hosts, the updates and multipath software were installed. The storage team has made the datastores available. Nothing special. After installation, the host has been taken out of maintenance mode. Then there was an error “Error: “Cannot install the vCenter Server agent service. Unknown installer error“. See VMware KB #2083945 and VMware KB #2056299.

I have followed all standard procedures to resolve HA errors:

  • Right click the affected host. Reconfigure for vSphere HA
  • Reconfigure HA on a cluster level.  Turn Off vSphere HA and  Turn ON vSphere HA
  • Disconnect and reconnect the affected host

After performing the above options, the issue was still unsolved. Next I wanted to know if the HA (fdm) agent is installed or not. I ssh to the host and ran the following command:

Esxcli software vib list | grep fdm

The output was empty. I realized that the HA agent was not installed. In VMware KB #2056299 is written about a vib dependency. That made me realize that besides the VMware updates also multipath software was installed, Dell EMC PowerPath/VE. This turned me out to the right direction to solve the problem.

Solution:

  • Ssh to the affected host(in maintenance mode)
  • Esxcli software vib list or Esxcli software vib list | grep power. The results are three vibs: powerpath.plugin.esx, powerpath.cim.esx and powerpath.lib.esx
  • Uninstall the three vibs running the following command: esxcli software vib remove –vibname=powerpath.plugin.esx –vibname=powerpath.cim.esx –vibname=powerpath.lib.esx
  • Reboot the host
  • Esxcli software vib list | grep power The output shlould be empty.
  • Leaving maintenance mode. The HA agent is now installing. After the HA agent is installed enter maintenance mode again
  • Esxcli software vib list | grep fdm The output should be similar like: vmware-fdm VMware VMwareCertified 2021-02-16
  • Reinstall Dell EMC PowerPath/VE. Installing the same version PowerPath/VE gave a VUM error even after restarting the host. To resolve this error I’ve installed a newer version of PowerPath/VE. This version was installed succesful.
  • Leaving maintenance mode

In my case the PowerPath/VE vibs dependecies were causing the issue. Another dependency can also cause this problem. I am aware that looking for the right dependence can be a difficult job. I hope I have at least been able to help you start the search in the right direction.

November 2022. An update about this issue can be read here.

VMware vCLS datastore selection

Recently I noticed that after updating a VMware vCenter from 6.7 to 7.0 u1 the new VMware vCLS VMs where placed on datastores that are not meant for VMs.

Starting with vSphere 7.0 Update 1, vSphere Cluster Services (vCLS) is enabled by default and runs in all vSphere clusters.
vCLS ensures that if vCenter Server becomes unavailable, cluster services remain available to maintain the resources and health of the workloads that run in the clusters.

The datastore for vCLS VMs is automatically selected based on ranking all the datastores connected to the hosts inside the cluster. A datastore is more likely to be selected if there are hosts in the cluster with free reserved DRS slots connected to the datastore. The algorithm tries to place vCLS VMs in a shared datastore if possible before selecting a local datastore. A datastore with more free space is preferred and the algorithm tries not to place more than one vCLS VM on the same datastore. You can only change the datastore of vCLS VMs after they are deployed and powered on.

You can perform a storage vMotion to migrate vCLS VMs to a different datastore.

If you want to move vCLS VMs to a different datastore or attach a different storage policy, you can reconfigure vCLS VMs. A warning message is displayed when you perform this operation.

Conclusion: If datastores used that are intended for e.g. repository purposes, it is possible that the vCLS files are placed on that datastores. You can tag vCLS VMs or attach custom attributes if you want to group them separately.

Reference: docs.vmware.com

VMware vSphere 7 first impression

Yesterday VMware released Version 7 of vSphere. After downloading the necessary software, I built a nested vSAN 7 cluster in my lab. This is not a deep technical blogpost just my first impression.

vSphere logo 2020


I chose a fresh installation instead of an upgrade. This has to do with the available resources in my lab. The installation was simple as usual.

  • Deploy 4 nested ESXi hosts
  • Install vCSA
  • Create a cluster
  • Configure networks
  • Create vSAN
  • Deploy vm’s
  • Setup Skyline
  • Setup Backup

Deploying nested ESXi

When creating the nested ESXi hosts don’t forget to check the CPU option “Expose hardware assisted virtualization to the guest OS”. This is required if you want a working nested ESXi.

CPU hardware assisted virtualization enabled

After spinning up the ESXi installation and just before the deployment, the following warning occurred.

CPU Warning during ESXi setup

This message is due to the obsolete CPU type of the physical ESXi host. Because it’s a lab we ignore the warning and start the deployment. After a few minutes the installation is finished.

Hooray!

vCenter vCSA

The first thing that is noticed, is the absence of the vSphere-Client. Nobody used the vSphere-client either. So only the native HTML5 client is available.

vSphere UI

vSAN cluster

I’ve manually created a local vSAN cluster. I prefer this method because it gives more flexibility than the Cluster quickstart wizard. There are a lot of new and enhanced features.

New:

  • Simplify Cluster Updates with vSphere Lifecycle Manager
  • Native File Services for vSAN

Enhancements:

  • Integrated DRS awareness of Stretched Cluster configurations
  • Immediate repair operation after a vSAN Witness Host is replaced
  • Stretched Cluster I/O redirect based on an imbalance of capacity across sites
  • Accurate VM level space reporting across vCenter UI for vSAN powered VMs
  • Improved Memory reporting for ongoing optimization
  • Visibility of vSphere Replication objects in vSAN capacity views
  • Support for larger capacity devices
  • Native support for planned and unplanned maintenance with NVMe hotplug
  • Removal of Eager Zero Thick (EZT) requirement for shared disk in vSAN
  • The complete information can be found here:

The vSAN capacity monitoring has also been greatly improved. It gives a good overview of the current and historical capacity usage.

Capacity Usage
Capacity History

Virtual Machines

Windows 2019 is now available as Guest OS.

Windows 2019 available as Guest OS

Skyline

Skyline gives a daily overview of security findings and recommendation from VMware environments. That is why I immediately added this cluster to Skyline. I wonder if there are any findings and recommendations after the first collection of data.

Update Skyline April 4, 2020

vSphere7 lab is connected to VMware Skyline. Already two recommendations. Good to see it works.

vSphere 7 connected to VMware Skyline

Backup

The vm’s in this environment must also be backed up. I have choose to use the backup solution from Veeam, V10. I don’t know if Veeam currently supports vSphere 7, but it works in my lab.

Conclusion

VMware has released multiple enhancements and improvements with vSphere 7. vSphere 7 remains the strong engine of a modern SDDC. In addition to vSphere7, VMware has also released VMware Cloud Foundation 4.0 and VMware Tanzu. There is a lot to read and learn about all the new and enhanced VMware products.

WSFC on vSAN, backup & restore

After a week in Barcelona for VMworld Europe 2019 I got home with a lot of new information and ideas. This post is about Windows Server Failover Cluster(WSFC) on vSAN and how to backup and restore. WSFC is now fully supported on vSphere 6.7 update3 and for the Dell VxRail users, code 4.7.300.

I started with reading VMware KB74786. It’s a good start and describes the straight forward deployment.

First I have deployed two Windows Server 2016 vm’s in a vSAN cluster. After the initial deployment I have added the failover cluster file server role should on both vm’s. Now it was time to power-off both vm’s and add a Paravirtual SCSI controller with a physical bus sharing to both vm’s.

The next step is reconfiguring vm1 and add two new disks. The first disk is 5GB(Quorum) and the second disk is 50GB(Fileserver data). After reconfigure the vm’s it’s time to power-on them again.

At vm1 I brought the new disks online and format them as NTFS. The next step is crucial before the cluster can be created. If you forgot this steps the disks are not detected in the cluster configuration. Power-off vm2 and add the two existing disks from vm1 to the Paravirtual SCSI controller. Power-on vm2 after reconfiguring.

The creation of the cluster is now straight forward as on physical hardware. You need a cluster-core FQDN and for the fileserver role you need a cluster-cap FQDN. There is a lot of documentation available about configure a Windows failover cluster and otherwise ask your favourite Windows admin :-).

After the deployment I did some failover and failback tests. I was surprised of the speed of the failover. I know there were not many client connections, but I am really impressed.

Backup and restore

I was already convinced that WSFC on vSAN should work. But how to backup and restore the cluster and the data on it? I was thinking about this because snapshots are unsupported with WSFC on vSAN. See VMware KB74786.

I’ve performed the backup and restore tests in my testlab with Veeam B&R 9.5 update 4b.

The backup and restore test configuration:

First I have excluded the two vm’s from snapshot backup. The next step is create a new protection group for virtual failover clusters in the inventory view. In the active directory tab, search and add the two nodes and the cluster-core. In the exclusion tab of the new protection group I have unmarked “Exclude All virtual machines”. This is important otherwise the the cluster nodes can’t be added to the protection group. Use a service account with enough permissions and keep the defaults in options tab. After completing the new protection group wizard, the Veeam Agent for Windows will be deployed on the cluster nodes. A reboot is needed. Using Veeam Agent for Windows is the trick in this test. I considered the cluster and nodes as if they were physical. That’s why I used Veeam Agent for Windows. The final step is configure a backup job and backup! After this initial backup I created the recovery iso for both nodes for a bare metal restore(bmr).

I’ve succesful do the following restores from a Veeam Windows ReFS landingzone server.

  • File / folder
  • Volume restore
  • Bare metal restore

Everything went normal. Only a bmr restore with recovery iso is a bit different then bmr a physical server. You have to keep the following in mind. Normally when you create a recovery iso all the network drivers are included in the iso. VMware VMXNET3 driver is not included. I’ve asked Veeam support if it’s possible to add the VMXNET3 driver? It’s not possible. There is an option to load a driver during the startup of the recovery iso. During my test I was able to browse the the driver in the Windows folder: C:\Windows\System32\DriverStore\FileRepository\vmxnet3.inf_amd64_583434891c6e8231. And load it succesful. In the future maybe there are other ways of achieve this.

During the bmr restore I was only able to recover the system volumes only. This by design, I guess, because normally the other cluster node, including the data volumes are online. Finally I’ve succesful tested a recovery of an entire cluster data volume.

Conclusion:

The test deployment WSFC on vSAN helped me better to understand how it works. I see definitely possibilities for WSFC on vSAN.

The backup and restore tests helped me to find an answer how to backup and restore a WSFC on vSAN cluster. The tested backup configuration is supported by Veeam. I logged a case and asked them and they confirmed! Keep in mind that your guest-os is supported. See the Veeam release notes document.

Cheers!

Unable to login VAMI vCSA 6.7 update 2a

Recently we ran into a strange issue. After upgrading to vCenter vCSA 6.7 update 2a we we were no longer able to login the vCSA VAMI. The message we see was “Unable to authenticate user”. vCenter was working fine for daily use.

So we started some investigation. It was impossible for us to enable SSH because we couldn’t login into the VAMI. So we tried to log in to the vCSA vm-console with the root account. After 4 attempts the root account was locked. The used password was the correct one. In the vCSA System Configuration, Manage Tab we saw an alert “The appliance management service on this node is not running”

We went to the vCSA system services and notify that that “Appliance Management Service” was not started. After starting the service the appliance management is back online.

Next thing was enable SSH and Bash so we were able to log in to the vCSA with SSH and the root account. We used the same root account and password as before when it was locked out.

Our final test was login to the VAMI with root account. The login succeeded but we were surprised by what we saw after we logged in.

It looks the update wasn’t finished. So now we had an delayed “Hooray” moment because the update installation was succeeded. We don’t know if this was an incident or a bug?