Category: vRealize Operations

Posted on

Forward Aria Operations alerts as SNMPv3 traps to Ubuntu

Last week I wrote about forwarding Aria Operations alerts with SNMPv3. Since Ubuntu is widely used I have decided to test also Ubuntu in my lab.

In this blog post I will describe the steps taken to send an SNMPv3 trap from Aria Operations to Ubuntu where the traps are stored readably in a log file. For this test, I used the following configuration:

  • VMware vCenter 8.0 update 2a
  • VMware ESXi 8.0 update 2
  • VMware Aria Operations version: 8.14.1
  • Linux Ubuntu 22.04.3 LTS
  • NET-SNMP 5.9.1

First step: Configure Aria Operations SNMP Trap Instance (Configure > Alerts > Outbound Settings > Add)

The name of the SNMP instance in my lab is “SNMP-UBUNTU”

The data highlighted in green later is needed later when configuring the Linux SMMP trap collector. For both the Authentication Password and Privacy Password, we use the same password, which is “T3st123!”.

Second step: Install and configure the Linux SNMP trap collector.

Install Ubuntu 22.04

Once Ubuntu is installed, snmptrapd need to be installed. Note that from version NET-SNMP 5.9.x onwards, AES256 can be used, which is needed to decode SNMPv3 traps.

Install snmptrapd 

apt-get update
apt-get -y install snmptrapd
apt update
apt -y install snmptrapd

Create /var/log/snmptrapd.log

touch /var/log/snmptrapd.log
chmod 666 /var/log/snmptrapd.log

Modify /usr/lib/systemd/system/snmptrapd.service to write traps to /var/log/snmptrapd.log

#Modify snmptrapd.service 
#Comment out the next line
#ExecStart=/usr/sbin/snmptrapd -LOw -f udp:162 udp6:162
#Add the next line
ExecStart=/usr/sbin/snmptrapd -LOw -f udp:162 udp6:162 -Lf /var/log/snmptrapd.log

Modify /etc/snmp/snmptrapd.conf

#Add the next lines
disableAuthorization yes
authCommunity log,execute,net public
createUser -e 687570118567EE5F48839F4D0C2B8AE5312C ariaops SHA-256 T3st123!! AES-256 T3st123!!
authUser log,execute,net ariaops
format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]:\n%v\n
outputOption s

Reload snmptrapd

systemctl daemon-reload
systemctl restart snmptrapd.service

Add Ubuntu MIB files

apt-get install snmp-mibs-downloader

Download the Aria Operations MIB files from the Aria Operations appliance. See VMware KB2118780. This is needed to convert the Aria Operations in readable alerts in the snmptrapd.log. See the next two Aria Operations test traps why the Aria Operations MIB files are needed.

First screenshot, the Aria Operations MIB files are not imported in Ubuntu. It cannot be seen that these alerts are VMware alerts

Next screenshot, the Aria Operations MIB files are imported in Ubunti. Now it is clear that these are VMware events.

#Download the Aria Operations MIB files from this directory 
/usr/lib/vmware-vcops/user/plugins/outbound/vcops-snmptrap-plugin/mib

Upload the Aria Operations MIB files from the Aria Operations appliance to Ubuntu

#Upload the Aria Operations MIB files into this directory
/usr/share/snmp/mibs

Modify /etc/snmp/snmp.conf

#Comment out mibs 
#mibs :

#Add custom Aria Operations MIB files
mibs +VMWARE-ROOT-MIB
mibs +VMWARE-VROPS-AGENTCAP-MIB
mibs +VMWARE-VROPS-MIB
mibs +VMWARE-PRODUCTS-MIB

Restart snmpd and snmptrapd

systemctl restart snmpd.service
systemctl restart snmptrapd.service

Third step: Test if Aria Operations SNMPv3 traps are received on the Linux SNMP collector

Test 1: Sending a test trap from the Aria Operations SNMP instance.

Note that before testing both password have been entered. Otherwise the test will fail.

Hit the “TEST” button. The following message will appear.

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Test Notification Rule”

We see “Test Notification Rule” in the snmptrad.log. Test is succeeded.

Test2: Disconnect a ESXi host from vCenter

I have created a notification that wil send a SNMP trap to the LinuxSNMP collector.

Disconnect ESXi host from vCenter:

Alert is raised in Aria Operations

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Host disconnected from vCenter”

We see “Host disconnected from vCenter” in the snmptrad.log. Test is succeeded.

From here, the alerts can move on to the third-party monitoring tool. That is beyond the scope of this test. Hopefully this blog post is useful to you.

Posted on

Forward Aria Operations alerts as SNMPv3 traps to a Linux collector

Recently a question came up whether it is possible to forward Aria Operations alerts to a third-party monitoring tool using SNMPv3. I had no experience with this until that point. During this exploration it turns out that there is a lot of information to be found about configuring a SNMPv3 target. It turned out to be quite a search to find the right information to be able to decode the SNMPv3 traps and write them readable away in a log file. I tested with several types of Linux, including Raspberry Pi and Rocky Linux 9, but in the end I succeeded with Alpine Linux. An added advantage of Alpine is that it is lightweight.

In this blog post I will describe the steps taken to send an SNMPv3 trap from Aria Operations to a Linux collector where the traps are stored readably in a log file. For the test, I used the following configuration:

  • VMware vCenter 8.0 update 2a
  • VMware ESXi 8.0 update 2
  • VMware Aria Operations version: 8.14.1
  • Linux Alpine Standard 3.19.0 x86_64
  • NET-SNMP 5.9.4

First step: Configure Aria Operations SNMP Trap Instance (Configure > Alerts > Outbound Settings > Add)

The name of the SNMP instance in my lab is “SNMP-Alpine-Linux”

The data highlighted in green later is needed later when configuring the Linux SMMP trap collector. For both the Authentication Password and Privacy Password, we use the same password, which is “T3st123!”.

Second step: Install and configure the Linux SNMP trap collector.

Install Alpine Linux. Click here for the Alpine Wiki.

Once Alpine is installed, NET-SNMP can be installed. Note that from version NET-SNMP 5.9.x onwards, AES256 can be used, which is needed to decode SNMPv3 traps.

Install NET-SNMP 

apk update && apk add net-snmp net-snmp-tools

Modify /etc/snmp/snmp.conf

In this step we will need the green marked higlighted data from the Aria Operations SNMP instance.

#######################################################################
# Example configuration file for snmptrapd
#
# No traps are handled by default, you must edit this file!
#
# authCommunity   log,execute,net public
# traphandle SNMPv2-MIB::coldStart    /usr/bin/bin/my_great_script cold
########################################################################

authCommunity log,execute,net public

#AriaOps

#createUser -e <ENGINE-ID> <USERNAME> <AUTHENTICATION PROTOCOL> <AUTHENTICATION PASSWORD> <PRIVACY PROTOCOL> <PRIVACY PASSWORD>

createUser -e 687517EA587080304ED28C73A0FB0B676570 ariaops SHA-256 T3st123! AES-256 T3st123!

######################################
authUser log,execute,net ariaops

format2 %.4y-%.2m-%.2l %.2h:%.2j:%.2k %B [%b]:\n%v\n

outputOption s

Modify /etc/conf.d/snmptrapd

Here we unmark #OPTS=”${OPTS} -Lf /var/log/snmptrapd.log” so snmp traps will be logged in /var/log/snmptrapd.log.

# extra flags to pass to snmptrapd
OPTS=""

# ignore authentication failure traps
#OPTS="${OPTS} -a"

# log messages to specified file
OPTS="${OPTS} -Lf /var/log/snmptrapd.log"

# log messages to syslog with the specified facility
# where facility is: 'd' = LOG_DAEMON, 'u' = LOG_USER, [0-7] = LOG_LOCAL[0-7]
#OPTS="${OPTS} -Ls d"

Restart snmptrapd

/etc/init.d/snmptrapd restart of systemctl restart snmptrapd

Create snmptrapd.log

touch /var/log/snmptrapd.log

Download the Aria Operations MIB files from the Aria Operations appliance. See VMware KB2118780. This is needed to convert the Aria Operations in readable alerts in the snmptrapd.log. See the next two Aria Operations test traps why the Aria Operations MIB files are needed.

First screenshot, the Aria Operations MIB files are not imported in Alpine. It cannot be seen that these alerts are VMware alerts

Next screenshot, the Aria Operations MIB files are imported in Alpine. Now it is clear that these are VMware events.

#Download the Aria Operations MIB files from this directory 
/usr/lib/vmware-vcops/user/plugins/outbound/vcops-snmptrap-plugin/mib

Upload the Aria Operations MIB files from the Aria Operations appliance to Alpine

#Upload the Aria Operations MIB files into this directory
/usr/share/snmp/mibs

Create /etc/snmp/snmp.conf

touch /etc/snmp/snmp.conf

Add the custom Aria Operations MIB files to snmp.conf

#Add custom Aria Operations MIB files
mibs +VMWARE-ROOT-MIB
mibs +VMWARE-VROPS-AGENTCAP-MIB
mibs +VMWARE-VROPS-MIB
mibs +VMWARE-PRODUCTS-MIB

Restart snmpd and snmptrapd

/etc/init.d/snmpd restart
/etc/init.d/snmptrapd restart

Auto start snmpd and snmptrapd at boot

rc-update add snmpd default
rc-update add snmptrap default

Third step: Test if Aria Operations SNMPv3 traps are received on the Linux SNMP collector

Test 1: Sending a test trap from the Aria Operations SNMP instance.

Note that before testing both password have been entered. Otherwise the test will fail.

Hit the “TEST” button. The following message will appear.

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Test Notification Rule”

We see “Test Notification Rule” in the snmptrad.log. Test is succeeded.

Test2: Disconnect a ESXi host from vCenter

I have created a notification that wil send a SNMP trap to the LinuxSNMP collector.

Disconnect ESXi host from vCenter:

Alert is raised in Aria Operations

Check the snmptrapd.log (tail -f /var/log/snmptrapd.log) and search for “Host disconnected from vCenter”

We see “Host disconnected from vCenter” in the snmptrad.log. Test is succeeded.

From here, the alerts can move on to the third-party monitoring tool. That is beyond the scope of this test. Hopefully this blog post is useful to you.

Posted on

Upgrade Aria Operations 8.12 to 8.14 fails

There is a known issue occurring with some Aria Ops upgrades from version 8.12 to 8.14. During the upgrade Aria Ops is hanging for some time and lose access to the Admin UI. The Admin UI does not come back. The issue is related due the Photon OS upgrade.

VMware Support has confirmed that Aria Ops 8.14 will be replaced shortly by Aria Ops 8.14 hotfix 1. Unlike the normal upgrade process, it will be possible to upgrade directly from Aria Ops 8.12 to 8.14 hotfix 1.

VMware Support’s advice is to wait a while before upgrading until Aria Ops 8.14 hotfix 1 is available.

Update November 22, 2023

Last night VMware released Aria Operations Upgrade PAK | 2023 | Build 22611353 . Read the complete release note here.

Posted on

Disable ESXi host alerts in vROPS when a host is in maintenance mode

For many years I have used Veeam Management Pack for VMware to monitor the VMware environments. After the switch to vROPS I never really missed it. However, whenever an ESXi host was in maintenance, there was an option that I could not find in vROPS. It’s such a small thing that makes you think “I still have to do something with that”. What it is? If an ESXi host is in maintenance, I don’t want any alerts from this host.

Recently, a colleague pointed me to an article that offers a solution to this problem. It is actually a very simple solution that only needs to be configured once. Because the article was already several years old, I have rewritten it based on the most recent version vROPS 8.2.x.

Use Case – An administrator wants to disable alerts on a ESXi host which has been put into maintenance mode in vCenter. This to avoid any alerts from this ESXi host inside of vROPS, while the admin wants to continue to collect metrics from this ESXi host.
Goal – Do this automatically without any manual changes in vROPS. As soon as a host is in maintenance mode in vCenter, vROPS should be aware of this and should stop alerting on the host in vROPS.
Solution – This can be achieved by a one time configuration using Custom Groups and Policy.

1- Create a new policy in vROPS named “Policy ESXi Hosts in maintenance mode”. This policy can be created under the default policy.
Go to Administration -> Policies -> Add
2- Select the default policy and click on the Add symbol to add a new policy.
3- Give it a name and description as shown below.

4- Click on Alerts and Symptom Definitions and filter the list of alerts with only host system alerts. We want a filtered list so that we can disable these in one go.

5- Now press CTRL + A on the keyboard to select all of them, you can also click on Actions -> Select All.
6- Click on Actions – > State -> Disable 

7- Click on Save and now you can see he new policy under your default policy.

8- Create a new custom group named “Group ESXi hosts in maintenance mode”. Use the following criteria to dynamically add members to this custom group based on ESXi host property which vROPS collects every few minutes.
Click on Environment -> Custom Groups -> Click on Add to add a new custom group.
Make sure to select the policy “Policy ESXi hosts in maintenance mode” which we created earlier.

9- Click on Preview to see if you are getting results. If there is any host in maintenance mode it will be displayed in the preview.

10 – Finally go into Administration -> Policies -> Active Policies and set the newly created policy at priority 1.

Now, as soon as you will put an ESXi host into maintenance mode in vCenter, within a few minutes it will be discovered as a ESXi host in maintenance in vROPS. It will be added to the new created custom group “Group ESXi hosts in maintenance mode”. Now all the alerts from this ESXi hosts are disabled. You will not see any alerts as long it is in maintenance mode.

Once the ESXi host is out of maintenance mode it will be moved out of the custom group.
Do note, that if you add any new alerts in the future, related to hosts, you would need to make sure that they are disabled in this policy.

Reference: Vxpresss blogspot