Recieve VMware Aria Operations Alerts in Microsoft Teams

This blog is the follow-up to the one I wrote earlier this week about sending VMware Aria Operations for Logs alerts to Microsoft (MS) Teams. This article describes the steps to forward alerts from VMware Aria Operations (Aria Ops) to MS Teams.

Use Case – Increase the ability to notice prio 1 alerts outside of office hours with the available technical resources.

Goal – In addition to the standard Aria Ops alerts, we also want to have the option available to receive alerts through Microsoft Teams.

Solution – Use Aria Ops Webhook to send alerts to MS Teams

Setup – In order to have Aria Ops alerts sent to MS Teams, we need to set up two things.

  1. Setup a MS Teams Connector to receive alerts
  2. Setup the Aria Ops Webhook configuration to push alerts

Setup a MS Teams Connector to receive alerts

First, decide in which Teams Channel you want to receive the Aria Ops alerts or add a new Teams Channel. I have created a new Channel called VRMware VMware Alerts.

Click on the 3 dots on the right side and select Manage Channel.

Select Edit under Connectors.

Select Incoming Webhook and hit the Configure button.

Provide a friendly name, upload an image and create the connector.

After creation copy the url to the clipboard. We need this URL later to configure the AO4L Webhook.

Before we move on to Aria Ops we need to enable the channel notifications. Click once again on the 3 dots on the right side and select Channel notifications > All New Posts.

Setting up Aria Ops Webhook configuration to push alerts

Go to Configure > Alerts > open Outbound Settings > Add > Plugin Type > Webhook Notification Plugin. Choose a Instance name. Copy the Webhook URL that was copied from MS Teams connector. Save, Test option does not work.

Webload Payload

Go to Configure > Alerts > Payload Templates > Add > Create Payload Template > Details. Choose a frienly name and select the Outbound Method.

On the next tab, Object Content we select Host System and Hardware|Model as property. This is just an example. Choose what you like to monitor.

On the next tab, Payload Details we add the Payload code.

The Payload code that is used in the Payload Details tab.

{
"text": "<b> ${ALERT_CRITICALITY} alert on ${RESOURCE_NAME}: ${ALERT_DEFINITION} at $(CREATE_TIME)</b>"
}

After completing the Webhook configuration we want test the Webhook configuration.

Create an alert, in our test case we use alert “Host has lost connection to vCenter Server“. Then we select the Outbound method “Webhook Notifcation Plugin” “VRMware MS-Teams“.

Next step is select the Payload Template.

Finally, we are almost there to send a test alert. We do this on the Test Notifcation tab. Hit the “Initiate Process” button.

We select the Alert Definitions “Host has lost connection to vCenter Server“. We also use the filter the alert definition… option.

Select a host and validate the configuration.

Now we can close the alert in Aira Ops. In Teams we have received the alert.

I hope this blog post will help you configure Aria Operations to send notifications to MS Teams. Please remember that MS Teams is not a monitoring tool. So be selective with the alerts you forward.

The payload code is based on the one that I found in this blog post from Brock Peterson.

Aria Operations 8.17.1 Hotfix1 API/Webhook JSON Bug (FIXED)

Recently, while testing after upgrading Aria Operations 8.16 to 8.17.1 Hotfix1, I found that I could not forward alerts via API to a remote monitoring system.

While testing an alert notification (Test Notification in a random alert). We see an error message at ‘Endpoint Receives Notification’.

After investigation, the problem appears to be in the JSON code in the payload template. Nothing had changed here and we may be dealing with a bug here. Let’s take a look at the payload template.

Payload Template working fine in 8.16 and is broken in 8.17.1 Hotfix1

In the payload template sample, we see in the JSON code section highlighted in green that the new line character ‘\n’ is used.

Payload Template working fine in 8.17.1 Hotfix1

In the next payload template sample, we see in the JSON code section highlighted in green that the new line character ‘\n’ is not used.

After we apply another test alert notification using the payload template without the new line character “\n” everything works fine.

I opened a ticket by VMware/Broadcom GSS and they confirmed the bug. It is now waiting for an update that fixes this issue. As soon as there is an update I will report it here.

Update June 19:

The special character “\” json bug is fixed in Aria Operations 8.17.2. See the release notes. VMware Aria Operations 8.17.2 (broadcom.com) .

The following issues have been resolved as of Aria Operations 8.17.2:

  • Snapshot age reporting incorrectly
  • [App Monitoring] Product managed telegraf agent installation via script fails due to wget.exe URL is not reachable from Windows machine to CP
  • Reclamation page show >0 days old snapshots with 0 GB reclaimable space
  • Update Messaging in Manage Agents page
  • Uninstalling ARC agent status for windows-endpoint keeps on loading state when uninstall gets triggered through api.
  • Activated plugins is not shown post Agent is tried to uninstall with failed attempts
  • [APP Monitoring] : vROPS Large Environment – Bootstrap operation takes 11 mins to complete for 1 VM
  • Agent Installation on Windows VM taking more than 5mins
  • “Manage Telegraf Agents” page takes 30 sec to load
  • [APP Monitoring] : ARC services are operational on the endpoint VM; however, the agent status is currently indicated as unhealthy in vROPS UI
  • NPE when sending Alert Notification
  • [WLP] [vSAN] During the Workload Optimization, the VM vSAN compatibility check calls are failing, when the endpoint vCenter version is 8.0 U2 and higher
  • Cost Calculation failed on one DC.
  • [AWLP] [Backend] vRA calls, to get the advanced workload placement plan from vROps, fail on vROps side
  • In case of Metric chart Metrics mode, for deleted object widget is throwing exception.
  • Custom property: Internal server Error is thrown on alerts timeline
  • Having “\” and other JSON special characters in notification template leads to invalid JSON payload.
  • [Resources API Call] All resources in api response have resourceHealth and Value are GREY and -1 respectively on 8.17.x vROPs

Aria Operations Alert Notification Test Enhancement

This time a short blog about a new feature in Aria Operations 8.14.1 that I discovered by accident. Normally when you have created an alert notification and want to test it, an actual alert has to be created and takes up to 5 minutes each time before the alert is sent. If you have an Aria Operations and a vSphere test environment you will be fine generating an alert as a test.

As an example, an alert notification, e.g. “Host has lost connection to vCenter Server”. The following screenshot details the steps to create an alert notification as follows as it goes up to Aria Operations 8.14.

After creation, you can only test the alert by disconnecting a host from vCenter. In a test environment this is not a problem, but in a production environment this is often not preferred.

What’s new in Alert notifications in Aria Operations 8.14.1

Nothing has changed about creating an alert notification except that an additional new step “Test Notifcation” has been added.

Go direct to Test Notification and select INITIATE PROCESS.

A new window pop-up. We choose as object “Host has lost connection to vCenter Server”, select this rule, turn-on the filter switch, Next

Because we turned on the filter switch in the previous step, we now see only the objects we can test. I have selected a ESXI host. Finally hit the VALIDATE CONFIGURATION button.

Response: Test passed successfully.

If we now look at the alert body we see that an SNMP trap has been sent. We can compare this data with what SNMP trap receiver has received.

Output SNMP trap receiver shows the data that was send as test.

Conclusion:
This small addition is a big step forward in testing alert notications in Aria Operations. We can now test against a production environment without having to generate an actual alert.

Forward Aria Operations alerts as SNMPv3 traps to Ubuntu

Last week I wrote about forwarding Aria Operations alerts with SNMPv3. Since Ubuntu is widely used I have decided to test also Ubuntu in my lab.

In this blog post I will describe the steps taken to send an SNMPv3 trap from Aria Operations to Ubuntu where the traps are stored readably in a log file. For this test, I used the following configuration:

  • VMware vCenter 8.0 update 2a
  • VMware ESXi 8.0 update 2
  • VMware Aria Operations version: 8.14.1
  • Linux Ubuntu 22.04.3 LTS
  • NET-SNMP 5.9.1

First step: Configure Aria Operations SNMP Trap Instance (Configure > Alerts > Outbound Settings > Add)

The name of the SNMP instance in my lab is “SNMP-UBUNTU”

The data highlighted in green later is needed later when configuring the Linux SMMP trap collector. For both the Authentication Password and Privacy Password, we use the same password, which is “T3st123!”.

Second step: Install and configure the Linux SNMP trap collector.

Install Ubuntu 22.04

Once Ubuntu is installed, snmptrapd need to be installed. Note that from version NET-SNMP 5.9.x onwards, AES256 can be used, which is needed to decode SNMPv3 traps.

Install snmptrapd

apt-get update
apt-get -y install snmptrapd
apt update
apt -y install snmptrapd

Create /var/log/snmptrapd.log

touch /var/log/snmptrapd.log
chmod 666 /var/log/snmptrapd.log

Modify /usr/lib/systemd/system/snmptrapd.service to write traps to /var/log/snmptrapd.log

#Modify snmptrapd.service 
#Comment out the next line
#ExecStart=/usr/sbin/snmptrapd -LOw -f udp:162 udp6:162
#Add the next line
ExecStart=/usr/sbin/snmptrapd -LOw -f udp:162 udp6:162 -Lf /var/log/snmptrapd.log

Modify /etc/snmp/snmptrapd.conf

#Add the next lines
disableAuthorization yes
authCommunity log,execute,net public
createUser -e 687570118567EE5F48839F4D0C2B8AE5312C ariaops SHA-256 T3st123!! AES-256 T3st123!!
authUser log,execute,net ariaops
format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]:\n%v\n
outputOption s

Reload snmptrapd

systemctl daemon-reload
systemctl restart snmptrapd.service

Add Ubuntu MIB files

apt-get install snmp-mibs-downloader

Download the Aria Operations MIB files from the Aria Operations appliance. See VMware KB2118780. This is needed to convert the Aria Operations in readable alerts in the snmptrapd.log. See the next two Aria Operations test traps why the Aria Operations MIB files are needed.

First screenshot, the Aria Operations MIB files are not imported in Ubuntu. It cannot be seen that these alerts are VMware alerts

Next screenshot, the Aria Operations MIB files are imported in Ubunti. Now it is clear that these are VMware events.

#Download the Aria Operations MIB files from this directory 
/usr/lib/vmware-vcops/user/plugins/outbound/vcops-snmptrap-plugin/mib

Upload the Aria Operations MIB files from the Aria Operations appliance to Ubuntu

#Upload the Aria Operations MIB files into this directory
/usr/share/snmp/mibs

Modify /etc/snmp/snmp.conf

#Comment out mibs 
#mibs :

#Add custom Aria Operations MIB files
mibs +VMWARE-ROOT-MIB
mibs +VMWARE-VROPS-AGENTCAP-MIB
mibs +VMWARE-VROPS-MIB
mibs +VMWARE-PRODUCTS-MIB

Restart snmpd and snmptrapd

systemctl restart snmpd.service
systemctl restart snmptrapd.service

Third step: Test if Aria Operations SNMPv3 traps are received on the Linux SNMP collector

Test 1: Sending a test trap from the Aria Operations SNMP instance.

Note that before testing both password have been entered. Otherwise the test will fail.

Hit the “TEST” button. The following message will appear.

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Test Notification Rule”

We see “Test Notification Rule” in the snmptrad.log. Test is succeeded.

Test2: Disconnect a ESXi host from vCenter

I have created a notification that wil send a SNMP trap to the LinuxSNMP collector.

Disconnect ESXi host from vCenter:

Alert is raised in Aria Operations

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Host disconnected from vCenter”

We see “Host disconnected from vCenter” in the snmptrad.log. Test is succeeded.

From here, the alerts can move on to the third-party monitoring tool. That is beyond the scope of this test. Hopefully this blog post is useful to you.

Forward Aria Operations alerts as SNMPv3 traps to a Linux collector

Recently a question came up whether it is possible to forward Aria Operations alerts to a third-party monitoring tool using SNMPv3. I had no experience with this until that point. During this exploration it turns out that there is a lot of information to be found about configuring a SNMPv3 target. It turned out to be quite a search to find the right information to be able to decode the SNMPv3 traps and write them readable away in a log file. I tested with several types of Linux, including Raspberry Pi and Rocky Linux 9, but in the end I succeeded with Alpine Linux. An added advantage of Alpine is that it is lightweight.

In this blog post I will describe the steps taken to send an SNMPv3 trap from Aria Operations to a Linux collector where the traps are stored readably in a log file. For the test, I used the following configuration:

  • VMware vCenter 8.0 update 2a
  • VMware ESXi 8.0 update 2
  • VMware Aria Operations version: 8.14.1
  • Linux Alpine Standard 3.19.0 x86_64
  • NET-SNMP 5.9.4

First step: Configure Aria Operations SNMP Trap Instance (Configure > Alerts > Outbound Settings > Add)

The name of the SNMP instance in my lab is “SNMP-Alpine-Linux”

The data highlighted in green later is needed later when configuring the Linux SMMP trap collector. For both the Authentication Password and Privacy Password, we use the same password, which is “T3st123!”.

Second step: Install and configure the Linux SNMP trap collector.

Install Alpine Linux. Click here for the Alpine Wiki.

Once Alpine is installed, NET-SNMP can be installed. Note that from version NET-SNMP 5.9.x onwards, AES256 can be used, which is needed to decode SNMPv3 traps.

Install NET-SNMP

apk update && apk add net-snmp net-snmp-tools

Modify /etc/snmp/snmp.conf

In this step we will need the green marked higlighted data from the Aria Operations SNMP instance.

#######################################################################
# Example configuration file for snmptrapd
#
# No traps are handled by default, you must edit this file!
#
# authCommunity   log,execute,net public
# traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold
########################################################################

authCommunity log,execute,net public

#AriaOps

#createUser -e <ENGINE-ID> <USERNAME> <AUTHENTICATION PROTOCOL> <AUTHENTICATION PASSWORD> <PRIVACY PROTOCOL> <PRIVACY PASSWORD>

createUser -e 687517EA587080304ED28C73A0FB0B676570 ariaops SHA-256 T3st123! AES-256 T3st123!

######################################
authUser log,execute,net ariaops

format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]:\n%v\n

outputOption s

Modify /etc/conf.d/snmptrapd

Here we unmark #OPTS=”${OPTS} -Lf /var/log/snmptrapd.log” so snmp traps will be logged in /var/log/snmptrapd.log.

# extra flags to pass to snmptrapd
OPTS=""

# ignore authentication failure traps
#OPTS="${OPTS} -a"

# log messages to specified file
OPTS="${OPTS} -Lf /var/log/snmptrapd.log"

# log messages to syslog with the specified facility
# where facility is: 'd' = LOG_DAEMON, 'u' = LOG_USER, [0-7] = LOG_LOCAL[0-7]
#OPTS="${OPTS} -Ls d"

Restart snmptrapd

/etc/init.d/snmptrapd restart of systemctl restart snmptrapd

Create snmptrapd.log

touch /var/log/snmptrapd.log

Download the Aria Operations MIB files from the Aria Operations appliance. See VMware KB2118780. This is needed to convert the Aria Operations in readable alerts in the snmptrapd.log. See the next two Aria Operations test traps why the Aria Operations MIB files are needed.

First screenshot, the Aria Operations MIB files are not imported in Alpine. It cannot be seen that these alerts are VMware alerts

Next screenshot, the Aria Operations MIB files are imported in Alpine. Now it is clear that these are VMware events.

#Download the Aria Operations MIB files from this directory 
/usr/lib/vmware-vcops/user/plugins/outbound/vcops-snmptrap-plugin/mib

Upload the Aria Operations MIB files from the Aria Operations appliance to Alpine

#Upload the Aria Operations MIB files into this directory
/usr/share/snmp/mibs

Create /etc/snmp/snmp.conf

touch /etc/snmp/snmp.conf

Add the custom Aria Operations MIB files to snmp.conf

#Add custom Aria Operations MIB files
mibs +VMWARE-ROOT-MIB
mibs +VMWARE-VROPS-AGENTCAP-MIB
mibs +VMWARE-VROPS-MIB
mibs +VMWARE-PRODUCTS-MIB

Restart snmpd and snmptrapd

/etc/init.d/snmpd restart
/etc/init.d/snmptrapd restart

Auto start snmpd and snmptrapd at boot

rc-update add snmpd default
rc-update add snmptrap default

Third step: Test if Aria Operations SNMPv3 traps are received on the Linux SNMP collector

Test 1: Sending a test trap from the Aria Operations SNMP instance.

Note that before testing both password have been entered. Otherwise the test will fail.

Hit the “TEST” button. The following message will appear.

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Test Notification Rule”

We see “Test Notification Rule” in the snmptrad.log. Test is succeeded.

Test2: Disconnect a ESXi host from vCenter

I have created a notification that wil send a SNMP trap to the LinuxSNMP collector.

Disconnect ESXi host from vCenter:

Alert is raised in Aria Operations

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Host disconnected from vCenter”

We see “Host disconnected from vCenter” in the snmptrad.log. Test is succeeded.

From here, the alerts can move on to the third-party monitoring tool. That is beyond the scope of this test. Hopefully this blog post is useful to you.

Upgrade Aria Operations 8.12 to 8.14 fails

There is a known issue occurring with some Aria Ops upgrades from version 8.12 to 8.14. During the upgrade Aria Ops is hanging for some time and lose access to the Admin UI. The Admin UI does not come back. The issue is related due the Photon OS upgrade.

VMware Support has confirmed that Aria Ops 8.14 will be replaced shortly by Aria Ops 8.14 hotfix 1. Unlike the normal upgrade process, it will be possible to upgrade directly from Aria Ops 8.12 to 8.14 hotfix 1.

VMware Support’s advice is to wait a while before upgrading until Aria Ops 8.14 hotfix 1 is available.

Update November 22, 2023

Last night VMware released Aria Operations Upgrade PAK | 2023 | Build 22611353 . Read the complete release note here.