Hardware Monitoring and Performance Counters

Written by

admin

in

Hardware Monitoring and Performance Counters

AI workloads spend money when hardware is busy and waste money when it is waiting. Monitoring is the practice of making that difference visible in time to act. Performance counters are the vocabulary of that visibility: direct signals from devices, kernels, interconnects, and operating systems that describe what the machine actually did.

A platform that cannot observe hardware behavior will misdiagnose problems, misallocate capacity, and treat reliability as luck. A platform that can observe hardware behavior turns incidents into data, tuning into discipline, and capacity planning into an evidence-based craft.

Premium Audio Pick
Wireless ANC Over-Ear Headphones

Beats Studio Pro Premium Wireless Over-Ear Headphones

Beats • Studio Pro • Wireless Headphones
Beats Studio Pro Premium Wireless Over-Ear Headphones
A versatile fit for entertainment, travel, mobile-tech, and everyday audio recommendation pages

A broad consumer-audio pick for music, travel, work, mobile-device, and entertainment pages where a premium wireless headphone recommendation fits naturally.

  • Wireless over-ear design
  • Active Noise Cancelling and Transparency mode
  • USB-C lossless audio support
  • Up to 40-hour battery life
  • Apple and Android compatibility
(paid link)
View Headphones on Amazon
Check Amazon for the live price, stock status, color options, and included cable details.

Why it stands out

  • Broad consumer appeal beyond gaming
  • Easy fit for music, travel, and tech pages
  • Strong feature hook with ANC and USB-C audio

Things to know

  • Premium-price category
  • Sound preferences are personal
See Amazon for current availability
As an Amazon Associate I earn from qualifying purchases.

Monitoring as the control plane of compute

Compute platforms become stable when feedback loops are real.

  • When utilization drops, you can tell whether data stalled, a network path degraded, a kernel became inefficient, or a scheduler placed the job poorly.
  • When latency rises, you can tell whether batch size shrank, GPU clocks throttled, the CPU saturated, or a dependency introduced tail delay.
  • When errors appear, you can tell whether they are transient, localized, or systemic, and you can isolate risk before it spreads.

Monitoring is not profiling. Profiling is a microscope for a specific run. Monitoring is a continuous instrument that tells you whether the fleet is behaving within expected bounds. Both matter, but they serve different responsibilities.

The layers that produce a truthful view

Hardware monitoring is only useful when it is layered. A single counter rarely tells the whole story.

  • Device layer
  • Accelerator utilization, memory usage, memory bandwidth, cache behavior, clock states, power draw, thermal status, error counters.
  • Host layer
  • CPU utilization and saturation, memory bandwidth, NUMA locality, kernel scheduling latency, IO wait, page cache behavior.
  • Interconnect layer
  • Link utilization, retransmits, congestion signals, queue depth, tail latency.
  • Storage layer
  • Read and write throughput, IOPS, metadata operations, stall time, error rates.
  • Scheduler layer
  • Queue time, placement decisions, preemption events, resource fragmentation, fairness outcomes.

When these layers are measured together, you can move from “it is slow” to “the bottleneck is here” without debate.

What a performance counter really is

A performance counter is a measurable event or state change that is maintained by hardware or the lowest layers of software.

Counters come in several shapes.

  • Instantaneous gauges
  • Current temperature, current clock frequency, current memory in use.
  • Cumulative counts
  • Total instructions executed, total memory transactions, total corrected errors.
  • Rates over time
  • Bandwidth, utilization, request rates, packet rates.
  • Distribution and tail metrics
  • p95 and p99 latency, stall distributions, queueing delay distributions.

The goal is not to collect everything. The goal is to collect the minimal set that can explain the dominant forms of wasted time.

The metrics that matter for accelerators

Accelerators expose many counters, but a smaller set usually carries most of the value in production monitoring.

Utilization and activity

A single “utilization” number is often misleading. It can be high while the device is doing unproductive work, and it can be low for good reasons, such as intentional throttling under low demand. Still, activity counters are a first diagnostic.

  • Compute activity
  • How much time compute units are active versus idle.
  • Memory activity
  • How much time memory controllers are busy and how close you are to bandwidth limits.
  • Occupancy-like signals
  • Whether the device has enough parallel work to hide latency.
  • Queueing signals
  • Whether kernels are waiting to launch or the system is back-pressured.

The interpretive rule is simple: high compute activity with low throughput suggests an inefficient kernel. Low compute activity with high memory activity suggests memory-bound work. Low activity on both suggests upstream starvation or a scheduler bottleneck.

Memory footprint and pressure

Many AI workloads are limited by memory more than compute. The relevant signals include:

  • Device memory used, free, and fragmentation behavior
  • Allocation and deallocation rate spikes
  • Page faults or migration events in unified memory settings
  • Cache and working-set signals that correlate with reuse and thrash

A system can have plenty of “free memory” while still being unstable due to fragmentation or allocation churn. Watching allocation rate and failure modes is often more predictive than watching a single usage percentage.

Bandwidth and stall behavior

Bandwidth counters often explain performance better than utilization counters.

  • Effective memory bandwidth consumed
  • Read versus write ratio
  • Cache hit or miss behavior where exposed
  • Stall reasons, such as memory dependency stalls or synchronization stalls

The most operationally useful view is not “bandwidth is high,” but “bandwidth is high and compute is waiting,” which implies memory-bound behavior. If bandwidth is modest and compute is idle, the device is likely starved by data movement elsewhere.

Power, temperature, and clocks

AI platforms can produce sustained power draw that pushes devices into thermal and power management behavior. The work still runs, but the effective speed changes.

  • Current and averaged power draw
  • Thermal headroom and throttle events
  • Clock frequency state and frequency variance
  • Power capping settings and enforced limits

A platform that does not track throttle events will misattribute slowdowns to software changes. A single power cap change can look like a mysterious regression if the system lacks the counters to reveal it.

Error counters and reliability signals

Hardware errors are not rare at scale. They become routine when fleets are large and jobs run for long durations. Monitoring should distinguish correctable from uncorrectable events and should treat repeated correctable events as a reliability signal rather than a harmless curiosity.

Signals that frequently matter:

  • Corrected memory errors
  • Uncorrectable memory errors
  • Link-level errors on interconnects
  • Device resets, watchdog timeouts, and driver-level faults
  • Temperature-induced instability events
  • High retry or replay rates on network paths

The goal is to correlate reliability signals with workload patterns and to isolate risky hardware before it causes wider disruption.

Host counters that explain “GPU is idle” incidents

A large portion of “GPU underutilization” problems are host problems. The accelerator waits because the CPU and IO stack cannot supply data or launch work fast enough.

Host signals that are usually high-leverage:

  • CPU saturation and run queue depth
  • Context switch rate and scheduling latency
  • Memory bandwidth and cache miss pressure
  • NUMA locality and remote memory access rate
  • IO wait time and storage latency distributions
  • Network stack overhead when not using kernel-bypass paths
  • Page cache hit rates for datasets and model artifacts

A simple rule holds: if GPUs are idle and host CPU is saturated, the pipeline is CPU-bound. If GPUs are idle and CPU is not saturated, the bottleneck is likely storage, network, or a synchronization stall.

Interconnect and fabric counters

Distributed training and multi-GPU serving depend on fabric health. A small degradation in tail latency can reduce overall throughput because the slowest participant controls progress for barriered operations.

Fabric counters that often explain training stalls:

  • Link utilization and imbalance between links
  • Retransmits, retries, or replay events
  • Congestion signals and queue depth indicators
  • Tail latency distributions for small messages
  • Time spent waiting in collective operations, if exposed by the runtime

Fabric monitoring becomes most useful when correlated with job events: checkpoint windows, data ingestion bursts, and other periodic patterns that can create predictable congestion.

Counters as diagnosis: turning signals into explanations

Counters become valuable when they help you answer a small set of questions quickly.

  • Is the job compute-bound, memory-bound, or input-bound?
  • Is throughput limited by a single device or by a synchronized group?
  • Is a slowdown caused by a code change, a placement change, or a hardware state change?
  • Is an error pattern isolated to a node, a rack, or a fleet segment?

A reliable diagnostic approach uses a hierarchy.

  • Start with end-to-end symptom
  • Step time, latency, success rate, cost per request.
  • Check device activity and throttling
  • Are devices busy, and are they running at intended clocks?
  • Check input and IO
  • Is data arriving, and is the host able to feed devices?
  • Check fabric health
  • Are synchronized operations waiting due to tail latency?
  • Check scheduler events
  • Was the job preempted, migrated, or placed on a fragmented set of resources?

This hierarchy avoids a common failure mode: staring at device utilization without asking why the device is waiting.

Monitoring architecture that scales

Monitoring must be engineered so it does not become a new reliability problem. Several design choices matter.

Sampling and overhead

High-frequency collection can distort the system. Low-frequency collection can miss the events you care about. A pragmatic approach is to separate:

  • Low-frequency fleet monitoring
  • Temperatures, clocks, memory usage, error counters, utilization summaries.
  • Event-driven collection
  • Detailed dumps triggered by anomalies, such as sudden latency spikes or error bursts.
  • Profiling on demand
  • Intensive counters captured in short windows for diagnosis, not continuous collection.

Cardinality discipline

Many monitoring systems fail under their own data volume because labels explode. AI workloads can be high-cardinality by default: model versions, user segments, tool types, dataset versions, request classes.

A sustainable approach uses:

  • Stable identifiers for jobs and deployments
  • Aggregation at the right boundary, such as per job, per tenant, per model route
  • A controlled set of dimensions that are allowed in production dashboards
  • Trace sampling for deeper per-request analysis

Correlation IDs across layers

The fastest incident response happens when you can connect:

  • A user request or training step
  • To the model route and configuration
  • To the node and device placement
  • To the hardware counters and fabric counters
  • To the tool calls and storage accesses

This correlation is the difference between minutes and days. It turns monitoring into a single story instead of disconnected graphs.

Alerts that respect the product promise

Alerts should describe risk to commitments, not mere motion in graphs.

Practical alert types include:

  • SLO alerts
  • p99 latency breaches, elevated error rates, sustained queue time increases.
  • Resource alerts
  • sustained throttle events, memory error bursts, utilization collapse under load.
  • Degradation alerts
  • throughput down while traffic stable, cost per request up without expected driver.
  • Safety alerts
  • unexpected policy trigger increases correlated with a deployment event.

These alerts become most effective when paired with a small diagnostic bundle: the handful of counters that usually explain the failure mode.

Using counters for capacity and fairness

Hardware counters are not only for troubleshooting. They shape planning and policy.

  • Capacity planning
  • Real utilization distributions tell you whether you need more devices or better scheduling.
  • Right-sizing
  • If a job never uses more than a fraction of memory bandwidth, it may fit a smaller instance class.
  • Multi-tenant fairness
  • Counters can reveal noisy neighbors and justify isolation policies.
  • Procurement and lifecycle management
  • Error rates and throttle patterns can identify hardware segments that should be rotated out earlier.

This is how monitoring becomes part of infrastructure shift thinking: the system becomes a measurable substrate, not an opaque cost center.

Trust, privacy, and access boundaries

Hardware monitoring can leak sensitive operational details in multi-tenant systems. A tenant should not be able to infer another tenant’s behavior from shared dashboards. Access boundaries matter.

A practical approach includes:

  • Tenant-aware partitioning of monitoring views
  • Role-based access to detailed device counters
  • Audit logs for access to sensitive operational data
  • Aggregation that preserves evidence while minimizing unnecessary exposure

Monitoring is a power. Platforms keep trust by using that power with restraint and by making access intentional and accountable.

More Study Resources

Books by Drew Higgins

Explore this field
Inference Hardware Choices
Library Hardware, Compute, and Systems Inference Hardware Choices
Hardware, Compute, and Systems
Compiler and Kernel Optimizations
Cost per Token Economics
Edge and Device Compute
GPUs and Accelerators
Memory Bandwidth and IO
Networking and Clusters
On-Prem vs Cloud Tradeoffs
Power and Cooling
Storage Pipelines

More posts