For a customer, I deployed a new vRealize Log Insight 8.10 cluster as a test. After initial installation and configuration of the first vRLI node I experienced a performance that was particularly bad. It wasn’t always this bad, but it frequently occurred.
The bad performance initially revealed itself when choosing one of the options in the menu on the left side of the screen. It seemed like the whole screen was frozen. After 10 till 30 seconds the requested information appeared.
I decided to first complete the cluster setup. After adding some vCenters and importing Content Packs, technically everything worked as designed. The bad performance was still there even after temporarily adding additional memory to the nodes.
During one of the moments that the performance was decreased I saw something in the lower left-hand corner. There were the following messages displayed:
Read <vRLI FQDN>
Transferring data from <vRLI FQDN>
Connecting to cdn.pendo.io
The last bullet was the key to resolve this issue. I searched some information about “cdn.pendo.io” and found this article about “Join or Leave the VMware Customer Experience Improvement Program“. At the bottom of the article is a section “What to do next”
After CEIP is enabled, when a user logs in to vRealize Log Insight, they see a banner at the top of their window that asks whether they want Pendo to collect data based on their interaction with the user interface.
If the user clicks Accept, Pendo collects their data and sends it to VMware
If the user clicks Decline, Pendo does not collect their data
In the General Configuration section of the vRLI configuration I deselected the Usage Reporting (Join the VMware Customer Experience Improvement Program). This resolved the issue for me.
Finally, in this case there was no need to join VMware CEIP. So, therefore it is an acceptable solution for me.
Recently, We missed an alert notification that had been generated in VMware vRealize Log Insight (vRLI) outside of office hours. This had caused a disruption that we could have avoided if we had been informed in time. The alert notifcation had been sent via email but email is not always accessed outside of business hours. Which I can well understand. In this blog, I will explain what I have come up with to notice these types of alerts earlier.
Use Case – Increase the ability to notice prio 1 alerts outside of office hours with the available technical resources.
Goal – In addition to the standard vRLI alerts, we also want to have the option available to receive alerts through Microsoft (MS) Teams.
Solution – Use vRLI Webhook to send alerts to MS Teams
Setup – In order to have vRLI alerts sent to MS Teams, we need to set up two things.
Setup a MS Teams Connector to receive alerts
Setup the vRLI Webhook configuration to push alerts
Setup a MS Teams Connector to receive alerts
First, decide in which Teams Channel you want to receive the vRLI Alerts or add a new Teams Channel. I have created a new Channel called VRMware VMware Alerts.
Click on the 3 dots on the right side and select Connectors.
Select Configure Incoming Webhook.
Provide a friendly name, upload an image and create the connector.
After creation copy the url to the clipboard. We need this URL later to configure the vRLI Webhook.
Before we move on to vRLI we need to enable the channel notifications. Click once again on the 3 dots on the right side and select Channel notifications > All activity.
Setup the vRLI Webhook configuration to push alerts
Go to the Administration section and open Configuration > Webhook > New Webhook. Choose a name. From the Endpoint drop down menu select Custom. Copy the Webhook URL that was copied from MS Teams connector. From the Content Type drop down menu select JSON and from the Action drop down menu select POST. The Webhook Payload will be described under the picture.
The Webload Payload was the hardest part to configure. Thanks to my colleague Roger who has figured out how the Webhook Payload layout should look like.
As far as I know, from vRLI Webhook only clear text can be used to send notifications to MS Teams. It’s possible to use one or more parameters in the script. For an overview of the parameters see the picture above here. Because the notictions are send in clear text it’s not possible to use all parameters. In our case not a problem because MS Teams is not used to replace monitoring software. It is just an additional option to be informed in a timely manner.
I wouldn’t go indepth how we found out the layout of the Webhook Payload code. That’s why I’m only sharing the code with you, so you can start testing for yourself.
After completing the Webhook configuration you may want test the Webhook configartion. Press the Send Test button.
Finally Save the Webhook configuration.
Open the MS Teams Channel where the connector was created earlier. You should see here the Test Alert.
The last part is sending a notification to MS Teams when a ESXi host have entered Maintenance Mode.
I have created an vRLI alert with the name “TEST VRMware VRLI Alert: vSphere Host entered Maintenance in vCenter“.
I have decided that I would like to be notified by both email and MS Teams. This can be set under the Trigger Conditions.
If everything is configured correctly we should receive the Send Test Alert Results after sending a test alert.
Save the Alert. Now we are ready for the final test. I put a ESXi host in maintenance mode and we should receive within 5 minutes a MS Teams notification. It works!
I hope this blog post will help you configure vRLI to send notifications to MS Teams. Please remember that MS Teams is not a monitoring tool. So be selective with the alerts you forward. I have chosen to only forward alerts that I know need to be acted on as soon as possible.
Recently I was asked if it is possible to receive an email notification if a vSAN storage policy with Force Provisioning enabled is applied to a vm. In this blogpost I want to show that this is possible.
Use Case – An administrator wants apply an vSAN storage policy with Force Provisioning enabled to virtual machines beacause of a possible shortage of vSAN storage capacity. In my opinion not a very good idea in a production environment!
Goal – Get an email notification when a vSAN storage policy with Force Provisioning enabled is applied to an vm. There is also the wish to make this visable in a dashboard.
Solution – With a bit of reverse engineering and vRealize Log Insight (vRLI) it’s possible to achieve this.
This is the information we need to created a new filter in vRLI Interactive Analytics.
Add the associated storage policy id “98df0443-5244-49af-9069-ad9fdbfedb52″ to the text field and add an extra filter (+ ADD FILTER). Choose from the pull down menu “vc_event_type” contains com.vmware.pbm.profile.associate. Choose a time window. In our example I choose “Latest 24 hours of data”. You can also choose here the last hour or last 5 minutes of data. It depends on the time you applied the policy and when you search in vRLI.
In de results above you don’t see the name of the vm with the applied storage policy. In the last image above here you see on the right of Events the Field Table section. Select Field Table. Search for the row with name vc_vm_name. Below here is the vm friendly name displayed of the vm with new applied storage policy.
Finally you want a email notification and a dashboard. I am not going to explain here how to create an email notification and a dashboard. This can be done in vRLI at the same way you normally create notifications and dashboards. Press the icon (1) to create a email notification and press the icon (2) to create a dashboard.
If you want receive an email notification if the vm get another storage policy applied. Then you should create another filter including the following two details.
In this blog post I wanted to demonstrate that it is possible with vRLI to receive an email notification if a vm has a storage policy applied where Force Provisioning is enabled. A disadvantage is that if the vm gets a different storage policy with different settings, this email notification is no longer valid, because the notifications are based on this specific storage policy id.
I have shown that it is works but as far as I believe it is not a solution for a production environment.
If there is a hardware issue that could cause problems within a vSAN cluster, you want to know as early as possible. Once you know this, you may have time to resolve the issue before business is compromised.
I have seen the following error several times in the results of a VxRail VxVerify check, which is performed to identify issues in a VxRail clusterbefore an update.
2021-10-08 15:01:00.012 esxi01.vrmware.nl vcenter-server: vSAN detected an unrecoverable medium or checksum error for component AB1234 on disk group DG5678
It could be possible that an underlying hardware device (physical disk) is causing this error. This is why you want to be informed as early as possible if there is an error that can cause an vSAN issue in the near future. This allows you to proactively carry out repair work, without any downtime to business operations.
How do you find out on which physical disk the component resides on? You need to identify the following information (first 3 bullets). The 4th bullet is about the vm which can be possible affected by the issue.
Virtual Machine where the component belongs to
Let’s start to identify the disk where the component resides:
Write down the component and diskgroup from the error
Ssh to an arbitrary ESXI server in the vSAN cluster. It doesn’t matter what server you choose.Type the following command: esxcli vsan debug object list –all > /tmp/objectlist.txt
Transfer /tmp/objectlist.txt to local pc
Open objectlist.txt and search for component AB1234.
Snippet from objectlist.txt: ++++++++++++++++++++++
Component State: ACTIVE, Address Space(B): 39369834496 (36.67GB), Disk UUID: 52ec6170-5298-7f14-1069-d0d3872b742a, Disk Name: naa.PD9012:1