Getting more out your Windows Performance Counters monitoring for web applications - part 3

Zabbix monitoring for your websites and ASP.NET applications in IIS using _Process Performance Counter.
Published on Wednesday, 14 April 2021
monitoring powershell wmi wql zabbix windows-server

Official Zabbix Logo

This is part 3 about Zabbix monitoring for your websites and ASP.NET applications in IIS. This time I'll show you how to get data from Win32_PerfRawData_PerfProc_Process counter, fast, for every application pool. This counter is notorious for its slowness, but you can get data a bit faster. IIS AppPool Insights in Zabbix - because there is always more than one way :-) .

In part 1 I showed you how to create an IIS website autodiscovery item and application discovery, and in part 2 you monitor the performance of your ASP.NET web application utilizing Windows performance counters in Zabbix. Now we take it a step further, to monitor Win32_PerfRawData_perfProc_Process data for all currently running application pools (which are auto discovered).

I query PerfRawData, because using Win32_PerfFormattedData_perfProc_Process has a small performance cost, and results may be rounded to 0 and not 0,1.

Autodiscover running IIS application pools

In this part you'll query Win32_PerfRawData_PerfProc_Process performance counters using WMI/CIM in PowerShell. The Win32_PerfRawData_PerfProc_Process class contains a lot of usefull properties for your IIS website application pools.

The gathered data needs to be transformed into a JSON blob so Zabbix can use it, but you also need to somehow match your application pool's (w3wp.exe) process ID to the result of your WMI query (called "WQL", or SQL for WMI. See Querying with WQL, WQL (SQL for WMI) for more information). This is done in PowerShell.

Let's begin.

Create a file called w3wp-discovery.ps1 and add the following into it:

$allpools = ( `
  Get-CimInstance -EA SilentlyContinue -ClassName applicationpool -Namespace root\webadministration
).Name

This simply outputs all applications pools. Now you'll need to add a Zabbix MACRO {#APPPOOLNAME} object.

@{
  "data" = $allpools | foreach { @{
    "{#APPPOOLNAME}" = $_
    }
  }
} | ConvertTo-Json

This'll give you an object $mypools with all discovered IIS application pools and a macro . It looks like:

{
  "data":  [
    {
      "{#APPPOOLNAME}":  "DefaultAppPool"
    },
    {
      "{#APPPOOLNAME}":  ".NET v4.5 Classic"
    },
    {
      "{#APPPOOLNAME}":  ".NET v4.5"
    },
    {
      "{#APPPOOLNAME}":  "example.com"
    },
    {
      "{#APPPOOLNAME}":  "example.org"
    },
    {
    "{#APPPOOLNAME}":  "example.nl"
    }
  ]
}

The macro is used to reference application pools in your autodiscover template. We need this information later on.

Zabbix configuration

For Zabbix, create a configuration file w3wp.conf in your zabbix_agentd.conf.d folder, and add the following UserParameter to it:

UserParameter=w3wp.discovery,powershell -NoProfile -ExecutionPolicy Bypass -File C:\path\to\w3wp-discovery.ps1

This creates the w3wp.discovery key and we have created a very basic, simple autodiscovery for IIS application pools in Zabbix.

Counter query Powershell script for dependent items

To query Win32_PerfRawData_perfProc_Process for performance counter values, we need PowerShell. You can use Get-WmiObject of Get-CimInstance cmdlets, and the data I want to monitor for application pools is (you are free to choose your own of course).

  • HandleCount
  • IDProcess
  • IOReadBytesPerSec
  • IOReadOperationsPerSec
  • IOWriteBytesPerSec
  • IOWriteOperationsPerSec
  • PercentProcessorTime
  • PrivateBytes
  • ThreadCount
  • WorkingSet
  • WorkingSetPeak

This is where the magic happens and you get all that info:

$processinfo = Get-CimInstance -EA SilentlyContinue -Query "select * from Win32_PerfRawData_PerfProc_Process where Name like 'w3wp_%'" -Namespace root\cimv2 | Select HandleCount,IDProcess,IOReadBytesPerSec,IOReadOperationsPerSec,IOWriteBytesPerSec,IOWriteOperationsPerSec,PercentProcessorTime,PrivateBytes,ThreadCount,WorkingSet,WorkingSetPeak

The WQL queries Win32_PerfRawData_PerfProc_Process for all data, and then I use PowerShell Select to filter the output. Why? Because this is way faster than running a query per application pool like

"select * from Win32_PerfRawData_PerfProc_Process where IDProcess='w3wp.exe PID'"

Especially if you need to this 300 or 500 times on a server. WMI is really slow, so let PowerShell do the filtering of data.

Increase WMI memory to support large volume of queries

The output may look as follows:

HandleCount             : 2537
IDProcess               : 7652
IOReadBytesPerSec       : 123547753
IOReadOperationsPerSec  : 112674
IOWriteBytesPerSec      : 78788008
IOWriteOperationsPerSec : 88060
PercentProcessorTime    : 4461250000
PrivateBytes            : 278888448
ThreadCount             : 159
WorkingSet              : 283172864
WorkingSetPeak          : 288657408
HandleCount             : 3038
IDProcess               : 5156
IOReadBytesPerSec       : 27230185458
IOReadOperationsPerSec  : 3777431
IOWriteBytesPerSec      : 87141280
IOWriteOperationsPerSec : 89040
PercentProcessorTime    : 248751718750
PrivateBytes            : 1265516544
ThreadCount             : 183
WorkingSet              : 1267384320
WorkingSetPeak          : 1358626816

It doesn't look like something you can work with in a Zabbix template, but fortunately you can use PowerShell to transform this in a nicely formatted JSON. This is why we need the IDProcess property: it get's matched to an application pool's .ProcessId property.

The following funcion get's called in a ForEach loop to select the correct data for each ProcessId / IDProcess.

function get-WmiProcess($wpname) {
  Try {
    $procid = ($allprocids | Where-Object AppPoolName -eq "${wpname}").ProcessId
    $data = ($processinfo | Where-Object IDProcess -eq $procid)
    return @{
      IOReadOperationsPerSec = $data.IOReadOperationsPerSec
      IOWriteOperationsPerSec = $data.IOWriteOperationsPerSec
      IOReadBytesPerSec = $data.IOReadBytesPerSec
      WorkingSetPeak = $data.WorkingSetPeak
      HandleCount = $data.HandleCount
      ThreadCount = $data.ThreadCount
      IOWriteBytesPerSec = $data.IOWriteBytesPerSec
      PercentProcessorTime = $data.PercentProcessorTime
      WorkingSet = $data.WorkingSet
      WorkingSetPrivate = $data.WorkingSetPrivate
      PrivateBytes = $data.PrivateBytes
      }
    }
  Catch {
  }
}

Looking how to get all process ID's in Windows for $allprocids? See Query WMI for all application pool’s process id’s.

And the ForEach loop to get the data and eventually output the JSON:

$processinfo = (Get-CimInstance -EA SilentlyContinue -ClassName Win32_PerfRawData_PerfProc_Process -Filter "Name like 'w3wp%'") | Select-Object HandleCount,IDProcess,IOReadBytesPerSec,IOReadOperationsPerSec,IOWriteBytesPerSec,IOWriteOperationsPerSec,PercentProcessorTime,PrivateBytes,ThreadCount,WorkingSet,WorkingSetPeak,WorkingSetPrivate
$procinfo = @{}
$allpools | ForEach {
  $procinfo[$_] = get-WmiProcess $_
}$procinfo | ConvertTo-Json

When ran, you can expect the following output to use in your Zabbix template:

"example.com":  {
  "IOReadOperationsPerSec":  124196,
  "IOWriteOperationsPerSec":  121311,
  "IOReadBytesPerSec":  6556014,
  "WorkingSetPrivate":  109973504,
  "WorkingSetPeak":  187043840,
  "HandleCount":  1849,
  "ThreadCount":  107,
  "IOWriteBytesPerSec":  68971258,
  "PercentProcessorTime":  204531250,
  "WorkingSet":  180256768,
  "PrivateBytes":  151367680
},

Zabbix template for WMI Win32_PerfRawData_PerfProc_Process counters

What happens here is a bit of a mess, it can be simplified, but it works :-) In your Zabbix autodiscovery template, you can create an autodiscover item "Fetch". But first you have to add to your w3wp.conf file the following as UserParameter:

UserParameter=Fetch[*],powershell -NoProfile -ExecutionPolicy Bypass -File C:\path\to\zabbix\scripts\w3wp-wmi.ps1 -action discovery
UserParameter=FetchAppPoolInfo[*],powershell -NoProfile -ExecutionPolicy Bypass -File C:\zabbix\scripts\w3wp-wmi.ps1 -action getapppoolinfo

A PowerShell script w3wp-wmi.ps1 is called and doing all the heavy lifting. From autodiscovery ("-action discovery" switch case) to retrieving specific metrics ("-action getapppoolinfo" switch case).

Now you create your monitoring template. An item key FetchAppPoolInfo is created as a master_item along with discovery_rule Fetch and its dependent items (the item-key 'apppoolinfofetch[]' is linked to the master_item FetchAppPoolInfo), that we link to an UserParameter 'FetchAppPoolInfo' in Zabbix' config.

<!-- ... -->
<items>
  <item>
    <uuid></uuid>
    <name>ApppoolInfoFetcher</name>
    <type>ZABBIX_ACTIVE</type>
    <key>FetchAppPoolInfo</key>
    <delay>1m</delay>
    <history>0</history>
    <trends>0</trends>
    <value_type>TEXT</value_type>
  </item>
</items>
<!-- ... -->

Generate your uuid with PowerShell: [guid]::NewGuid().ToString("N"), see Create strong passwords in Windows for more examples.

And create a discovery_rule:

<discovery_rules>
  <discovery_rule>
    <uuid></uuid>
    <name>IIS apppool discovery</name>
    <type>ZABBIX_ACTIVE</type>
    <key>Fetch</key>
    <lifetime>1d</lifetime>
    <item_prototypes>
      <item_prototype>
        <uuid></uuid>
        <name>ApppoolInfoFetch for {#APPPOOLNAME}</name>
        <type>DEPENDENT</type>
        <key>apppoolinfofetch[{#APPPOOLNAME}]</key>
        <delay>0</delay>
        <history>1d</history>
        <trends>0</trends>
        <value_type>TEXT</value_type>
        <description>Master item</description>
        <preprocessing>       
          <step>
            <type>JSONPATH</type>
            <parameters>
              <parameter>$['{#APPPOOLNAME}']</parameter>
            </parameters>
          </step>
          <step>
            <type>DISCARD_UNCHANGED_HEARTBEAT</type>
            <parameters>
              <parameter>10m</parameter>
            </parameters>
          </step>
        </preprocessing>
        <master_item>
          <key>FetchAppPoolInfo</key>
        </master_item>
        <tags>
          <tag>
            <tag>Application</tag>
            <value>IIS</value>
          </tag>
        </tags>
      </item_prototype>

      <!-- ... -->

      <item_prototype>
        <uuid></uuid>
        <name>Site {#APPPOOLNAME} IOReadOperationsPerSec</name>
        <type>DEPENDENT</type>
        <key>iis.apppool.IOReadOperationsPerSec[{#APPPOOLNAME}]</key>
        <delay>0</delay>
        <history>7d</history>
        <value_type>FLOAT</value_type>
        <description>The rate at which the process is writing bytes to I/O operations. This counter counts all I/O activity generated by the process to include file, network and device I/Os.</description>
        <preprocessing>
          <step>
            <type>JSONPATH</type>
            <parameters>
              <parameter>$.IOReadOperationsPerSec</parameter>
            </parameters>
          </step>
          <step>
            <type>CHANGE_PER_SECOND</type>
            <parameters>
              <parameter/>
            </parameters>
          </step>
        </preprocessing>
        <master_item>
          <key>apppoolinfofetch[{#APPPOOLNAME}]</key>
        </master_item>
        <tags>
          <tag>
            <tag>Application</tag>
            <value>IIS</value>
          </tag>
        </tags>
      </item_prototype>
    </item_prototypes>
  </discovery_rule>
</discovery_rules>

Continue creating more item prototypes for your counter data, and you're all set.

Keep in mind PercentProcessorTime is measured by the number of CPU seconds a process uses, per second. Meaning 1 is 100%. You can add a multiplier of 100, so 100% is 1 CPU core completely used. For readability, in my template I have two multiplier preprocessing steps: 0.0000001 and 100 (yes, one 0.00001 was also enough). And don't forget to set CHANGE_PER_SECOND.

I mentioned switch cases in PowerShell, I have defined them link this:

param (
  [Parameter(Position=0, Mandatory=$False)]
  [string] $action = "discovery"
)

# ...
# ...

switch ($action) {
  "discovery" {
    @{  
      "data" = $allpools | foreach { @{
        "{#APPPOOLNAME}" = $_
      }}
    } | ConvertTo-Json
  }
  "getapppoolinfo" {
    # Loop over all apppool names, store the function result in an array
    $AppPoolInfo = @{}
    $allpools | foreach {
      $AppPoolInfo[$_] =  get-WmiProcess $_
    }       
    # Convert the array to JSON
    $AppPoolInfo | ConvertTo-Json
  }  default {
  "Script error"
 }
}

Eliminate and discard unsupported monitoring items

Now that you have your template for IIS application pool monitoring complete, what happens to an item if the script doesn't return a value for -for example- WorkingSetPeak? The item becomes "unsupported", and you don't want that. Eliminate Zabbix unsupported items by adding an error_handler to your items as a preprocessing step:

<preprocessing>
  <step>
    <type>JSONPATH</type>
    <params>$.WorkingSetPeak</params>
    <error_handler>DISCARD_VALUE</error_handler>
  </step>
  <step>
    <type>DISCARD_UNCHANGED_HEARTBEAT</type>
    <params>10m</params>
  </step>
</preprocessing>

The preprocessing documentation reads:

If you mark the Custom on fail checkbox, the item will not become unsupported in case of failed preprocessing step and it is possible to specify custom error handling options: either to discard the value, set a specified value or set a specified error message.