Windows Performance Analyzer shows negative DPC numbers on dxgkrnl.sys - Possibly an error of AMD drivers?

Alchemy Tweaks · Aug 6, 2024

All necessary files and info to understand my issue

The beginning

Let's start from the beginning. I use the Windows Performance Toolkit for performance analysis that I conduct systematically. Additionally, I use it for comparisons (before and after changes that I test). Recently, I decided to start taking measurements for this exact reason—to determine if a particular setting provides a detectable performance boost in the system.

During the measurements, I noticed an anomaly in a metric (DPC/ISR Enter Time min). When I navigated to Graph Explorer > Computation > DPC/ISR > DPC/ISR Duration by Module, Function, I observed negative values in dxgkrnl.sys, such as -1,449,211,376.700 (µs). The other metrics, although extremely high, did not have negative values. Additionally, there were no issues with other drivers besides dxgkrnl.sys and occasionally Wdf01000.sys.

Since we are talking about core Windows drivers, it is expected to see high values, but always within reasonable limits.

Measurements procedure

The measurements were conducted using xperf and through the script available in the post. The main command that invoked the DPC/ISR providers was "xperf -on base+interrupt+dpc". By default, the recording was done using Memory Logging mode and not File Logging mode. I emphasize this because, as I will mention later, Jeff Stokes (ETL,WPA Expert) noted the following:

"So WPR records in memory mode, it's common to get traces where it's missing a lot of the CPU graph due to the buffer being set too conservatively. It doesn't report as dropped events, but you'll see all your CPU time is hyper-compressed in one small area on the right side of the graph. WPR -filemode fixes this."

Based on this, it could be possible that an issue arises due to recording in memory mode. On the other hand, Pavel Yosifovich mentioned the following:

"Maybe the sum overflows (32-bit) and that's why it becomes negative."

Based on this, we might be dealing with a bug in WPA.

The way I conducted the tests was as follows: I used an xperf script with the command mentioned above and made sequential recordings, typically around 50, with each lasting 10 seconds. I conducted measurements both in idle mode and during games to capture the system's behavior under heavy workload. In both cases, I observed these contaminated results with negative values and a high number of DPCs/ISRs, with the main culprit always being dxgkrnl.sys.

I should note that with xperf, I exported the data in two ways: I generated a TXT report and the classic ETL file. Almost always, the following was true: if I opened individual samples in WPA, there were no strange or negative numbers. When I opened them together, the anomalies appeared. Secondly, the TXT reports exhibited the problem, whereas the corresponding ETL file, when opening the counters, did not show anything unusual at first glance. This led to a deadlock.

Below, I present a sample TXT report to illustrate what I mean. Additionally, the frequency at which I observed the problem was such that, out of 50 samples, at least 5 would be problematic based on the TXT file or if I opened all ETLs together as a single trace in WPA.

--------------------------
DPC Info

--------------------------

Total = 46039
Elapsed Time, > 0 usecs AND <= 1 usecs, 21153, or 45.95%
Elapsed Time, > 1 usecs AND <= 2 usecs, 2805, or 6.09%
Elapsed Time, > 2 usecs AND <= 4 usecs, 1357, or 2.95%
Elapsed Time, > 4 usecs AND <= 8 usecs, 4590, or 9.97%
Elapsed Time, > 8 usecs AND <= 16 usecs, 9954, or 21.62%
Elapsed Time, > 16 usecs AND <= 32 usecs, 2064, or 4.48%
Elapsed Time, > 32 usecs AND <= 64 usecs, 45, or 0.10%
Elapsed Time, > 64 usecs AND <= 128 usecs, 3313, or 7.20%
Elapsed Time, > 128 usecs AND <= 256 usecs, 756, or 1.64%
Elapsed Time, > 256 usecs AND <= 512 usecs, 1, or 0.00%
Elapsed Time, > 512 usecs AND <= 1024 usecs, 0, or 0.00%
Elapsed Time, > 1024 usecs AND <= 2048 usecs, 0, or 0.00%
Elapsed Time, > 2048 usecs AND <= 4096 usecs, 0, or 0.00%
Elapsed Time, > 4096 usecs AND <= 8192 usecs, 0, or 0.00%
Elapsed Time, > 8192 usecs AND <= 16384 usecs, 0, or 0.00%
Elapsed Time, > 16384 usecs AND <= 32768 usecs, 0, or 0.00%
Elapsed Time, > 32768 usecs AND <= 65536 usecs, 0, or 0.00%
Elapsed Time, > 65536 usecs AND <= 131072 usecs, 0, or 0.00%
Elapsed Time, > 131072 usecs AND <= 262144 usecs, 0, or 0.00%
Elapsed Time, > 262144 usecs AND <= 524288 usecs, 0, or 0.00%
Elapsed Time, > 524288 usecs AND <= 1048576 usecs, 0, or 0.00%
Elapsed Time, > 1048576 usecs AND <= 2097152 usecs, 0, or 0.00%
Elapsed Time, > 2097152 usecs , 1, or 0.00%
Total, 46039

Total = 35182 for module dxgkrnl.sys
Elapsed Time, > 0 usecs AND <= 1 usecs, 15734, or 44.72%
Elapsed Time, > 1 usecs AND <= 2 usecs, 1784, or 5.07%
Elapsed Time, > 2 usecs AND <= 4 usecs, 53, or 0.15%
Elapsed Time, > 4 usecs AND <= 8 usecs, 2932, or 8.33%
Elapsed Time, > 8 usecs AND <= 16 usecs, 8658, or 24.61%
Elapsed Time, > 16 usecs AND <= 32 usecs, 1928, or 5.48%
Elapsed Time, > 32 usecs AND <= 64 usecs, 28, or 0.08%
Elapsed Time, > 64 usecs AND <= 128 usecs, 3308, or 9.40%
Elapsed Time, > 128 usecs AND <= 256 usecs, 756, or 2.15%
Elapsed Time, > 256 usecs AND <= 512 usecs, 0, or 0.00%
Elapsed Time, > 512 usecs AND <= 1024 usecs, 0, or 0.00%
Elapsed Time, > 1024 usecs AND <= 2048 usecs, 0, or 0.00%
Elapsed Time, > 2048 usecs AND <= 4096 usecs, 0, or 0.00%
Elapsed Time, > 4096 usecs AND <= 8192 usecs, 0, or 0.00%
Elapsed Time, > 8192 usecs AND <= 16384 usecs, 0, or 0.00%
Elapsed Time, > 16384 usecs AND <= 32768 usecs, 0, or 0.00%
Elapsed Time, > 32768 usecs AND <= 65536 usecs, 0, or 0.00%
Elapsed Time, > 65536 usecs AND <= 131072 usecs, 0, or 0.00%
Elapsed Time, > 131072 usecs AND <= 262144 usecs, 0, or 0.00%
Elapsed Time, > 262144 usecs AND <= 524288 usecs, 0, or 0.00%
Elapsed Time, > 524288 usecs AND <= 1048576 usecs, 0, or 0.00%
Elapsed Time, > 1048576 usecs AND <= 2097152 usecs, 0, or 0.00%
Elapsed Time, > 2097152 usecs , 1, or 0.00%
Total, 35182

Here are the initial steps I took to identify the cause:

Reduced the number of recordings to 40, then to 30, and so on.
Decreased the duration of each recording or increased it in some cases.
Tried different versions of the Windows Performance Toolkit (WPT).
Conducted recordings with Windows Performance Recorder (WPR).
Modified the xperf commands to see if it would resolve the issue.
Opened the recordings with WPA Preview.
Reinstalled Windows 10 with all updates and the latest drivers.
Tried Windows 11.
Ran the DirectX Diagnostic Tool without detecting any issues and ensured all DirectX updates were installed.
Restored BIOS settings to default.

All these tests were conducted on the same machine, and essentially, I tried everything on this system.

Then, as anyone would, I tried the tests on another machine, specifically a laptop with an Intel processor and integrated graphics. I conducted simple recordings within the same range, and this time I didn't observe anything unusual. However, I didn't have the luxury of performing measurements under heavy system load as I did previously. Therefore, I concluded that the issue might be with my machine.

Jeff Stokes

While reading the book "Windows Performance Analysis Field Guide 1st Edition.pdf," I came across Jeff Stokes. Therefore, I took the liberty to explain the problem to him and get his opinion. Indeed, he found it peculiar and mentioned a bug in AMD drivers related to DPC Watchdogs. Specifically, he noted:

"It's more likely a DPC GUID is reused from one trace to another, and it's piecing together the start and end times that are inverted. I know AMD GPUs still seem to have DPC watchdog events (last tried a 7900xtx and it was unreliable to me). This could be a legitimate reading from the AMD GPU, the real source of the TDR/DPC watchdog events AMD seems plagued with. Maybe it sends something with a negative timing, like the old AMD Opterons used to do."

LAST THOUGHTS

I should mention that otherwise, the system is functional, without issues like FPS drops, stutters, high latency, high ping, or screen tearing when playing games or during regular computer use. The recordings I make with FrameView and CapFrameX have never shown anything unusual; everything was normal.

So, what is really happening here? Am I wasting my time chasing ghosts, or have I truly encountered something interesting? I hope someone can shed light on this matter so we can proceed further.

I plan to conduct additional measurements under different conditions and on a system with an NVIDIA GPU to see if the results are consistent or different under similar conditions. Thank you very much to anyone who read all of this, and I apologize if I'm not fully clear. For any clarifications, please feel free to contact me!

CONTACT ME

Discord: Alchemy Tweaks#3412
X: x.com
Email: vassilis04asim@gmail.com

garlin · Aug 6, 2024

It's always been observed in the Windows community, that graphics drivers for advanced GPU's are the leading cause of random DPC latency. Some GPU's and/or combinations of their drivers have it worse than others. The more expensive and faster a system, doesn't always translate to lower DPC latency. Other than praying for a driver fix, the ad hoc workaround is switching your GPU into a power-saving mode to minimize its priority over other processes.

Alchemy Tweaks · Aug 6, 2024

garlin said:
It's always been observed in the Windows community, that graphics drivers for advanced GPU's are the leading cause of random DPC latency. Some GPU's and/or combinations of their drivers have it worse than others. The more expensive and faster a system, doesn't always translate to lower DPC latency. Other than praying for a driver fix, the ad hoc workaround is switching your GPU into a power-saving mode to minimize its priority over other processes.

Most people I mentioned my issue in msft they said "Probably is bug issue" and likely it's because of something in my captures? And not an issue of WPA?

garlin · Aug 6, 2024

Bugs in another vendor's HW drivers. Your Z590M UD has Intel onboard graphics. For the sake of argument, remove the RX 5700 and re-run your tests only using the Intel GPU. Probably a big difference because the Intel GPU isn't in the same class as the AMD GPU. Or run your AMD at the lowest visual performance level from the control panel.

WPA is flagging DX, because it's waiting for the driver.

Alchemy Tweaks · Aug 6, 2024

garlin said:
Bugs in another vendor's HW drivers. Your Z590M UD has Intel onboard graphics. For the sake of argument, remove the RX 5700 and re-run your tests only using the Intel GPU. Probably a big difference because the Intel GPU isn't in the same class as the AMD GPU. Or run your AMD at the lowest visual performance level from the control panel.

WPA is flagging DX, because it's waiting for the driver.

Doing a similar test on my Intel-based laptop under as similar conditions as possible, and indeed the problem did not exist. It makes sense that I have nothing to lose by trying what you suggested on the main system to ensure 100% that the AMD driver is at fault. Nevertheless, since this problem exists and I obviously need to use the GPU, should I ignore the samples that come with the bug?

P.S I submitted my issue in WPT tool team and I wait for their response (I tried the same with AMD but I don't expect something huge)

garlin · Aug 6, 2024

This is a well known and hotly debated issue in the NVIDIA community. It doesn't take long to find the threads, though unfortunately most posters are nowhere near as technical as yourself. I wouldn't imagine AMD is immune to the same problems. But the impact greatly varies across the GPU families, and it's believed some (older) versions of Windows are less affected. MS is always polite because NVIDIA, AMD and Intel are key partners; but essentially they don't think it's a Windows problem but won't say that out loud.

The bigger the NVIDIA GPU you can find, the worse your DPC will get.

Alchemy Tweaks · Aug 6, 2024

I did more tests on the main machine (AMD Based), I tried with ultra low res on CS2.....

On this case I hadn't the issue (that was the problem? Likely I cannot be sure, but when I revert to high res the problem came up.

I saw also negative numbers in ISR at this time with dxgkrnl.sys

Batch:

--------------------------
Interrupt Info

--------------------------
CPU Usage Summing By Module For the Whole Trace


Total = 38882
Elapsed Time, >        0 usecs AND <=        1 usecs,  16855, or  43.35%
Elapsed Time, >        1 usecs AND <=        2 usecs,   2058, or   5.29%
Elapsed Time, >        2 usecs AND <=        4 usecs,   3223, or   8.29%
Elapsed Time, >        4 usecs AND <=        8 usecs,   6155, or  15.83%
Elapsed Time, >        8 usecs AND <=       16 usecs,   7461, or  19.19%
Elapsed Time, >       16 usecs AND <=       32 usecs,   2748, or   7.07%
Elapsed Time, >       32 usecs AND <=       64 usecs,    378, or   0.97%
Elapsed Time, >       64 usecs AND <=      128 usecs,      3, or   0.01%
Elapsed Time, >      128 usecs AND <=      256 usecs,      0, or   0.00%
Elapsed Time, >      256 usecs AND <=      512 usecs,      0, or   0.00%
Elapsed Time, >      512 usecs AND <=     1024 usecs,      0, or   0.00%
Elapsed Time, >     1024 usecs AND <=     2048 usecs,      0, or   0.00%
Elapsed Time, >     2048 usecs AND <=     4096 usecs,      0, or   0.00%
Elapsed Time, >     4096 usecs AND <=     8192 usecs,      0, or   0.00%
Elapsed Time, >     8192 usecs AND <=    16384 usecs,      0, or   0.00%
Elapsed Time, >    16384 usecs AND <=    32768 usecs,      0, or   0.00%
Elapsed Time, >    32768 usecs AND <=    65536 usecs,      0, or   0.00%
Elapsed Time, >    65536 usecs AND <=   131072 usecs,      0, or   0.00%
Elapsed Time, >   131072 usecs AND <=   262144 usecs,      0, or   0.00%
Elapsed Time, >   262144 usecs AND <=   524288 usecs,      0, or   0.00%
Elapsed Time, >   524288 usecs AND <=  1048576 usecs,      0, or   0.00%
Elapsed Time, >  1048576 usecs AND <=  2097152 usecs,      0, or   0.00%
Elapsed Time, >  2097152 usecs                      ,      1, or   0.00%
Total,                                                 38882



                    

Total = 37050 for module dxgkrnl.sys
Elapsed Time, >        0 usecs AND <=        1 usecs,  16442, or  44.38%
Elapsed Time, >        1 usecs AND <=        2 usecs,   2013, or   5.43%
Elapsed Time, >        2 usecs AND <=        4 usecs,   3016, or   8.14%
Elapsed Time, >        4 usecs AND <=        8 usecs,   5308, or  14.33%
Elapsed Time, >        8 usecs AND <=       16 usecs,   7185, or  19.39%
Elapsed Time, >       16 usecs AND <=       32 usecs,   2707, or   7.31%
Elapsed Time, >       32 usecs AND <=       64 usecs,    375, or   1.01%
Elapsed Time, >       64 usecs AND <=      128 usecs,      3, or   0.01%
Elapsed Time, >      128 usecs AND <=      256 usecs,      0, or   0.00%
Elapsed Time, >      256 usecs AND <=      512 usecs,      0, or   0.00%
Elapsed Time, >      512 usecs AND <=     1024 usecs,      0, or   0.00%
Elapsed Time, >     1024 usecs AND <=     2048 usecs,      0, or   0.00%
Elapsed Time, >     2048 usecs AND <=     4096 usecs,      0, or   0.00%
Elapsed Time, >     4096 usecs AND <=     8192 usecs,      0, or   0.00%
Elapsed Time, >     8192 usecs AND <=    16384 usecs,      0, or   0.00%
Elapsed Time, >    16384 usecs AND <=    32768 usecs,      0, or   0.00%
Elapsed Time, >    32768 usecs AND <=    65536 usecs,      0, or   0.00%
Elapsed Time, >    65536 usecs AND <=   131072 usecs,      0, or   0.00%
Elapsed Time, >   131072 usecs AND <=   262144 usecs,      0, or   0.00%
Elapsed Time, >   262144 usecs AND <=   524288 usecs,      0, or   0.00%
Elapsed Time, >   524288 usecs AND <=  1048576 usecs,      0, or   0.00%
Elapsed Time, >  1048576 usecs AND <=  2097152 usecs,      0, or   0.00%
Elapsed Time, >  2097152 usecs                      ,      1, or   0.00%
Total,                                                 37050




==============

1080p in CS2




[ATTACH type="full"]103327[/ATTACH]


Tests even with the latest Pro Driver of AMD..........


[ATTACH type="full"]103328[/ATTACH]

[CODE]--------------------------
DPC Info


Total = 56104
Elapsed Time, >        0 usecs AND <=        1 usecs,  29075, or  51.82%
Elapsed Time, >        1 usecs AND <=        2 usecs,   1747, or   3.11%
Elapsed Time, >        2 usecs AND <=        4 usecs,   3821, or   6.81%
Elapsed Time, >        4 usecs AND <=        8 usecs,  14236, or  25.37%
Elapsed Time, >        8 usecs AND <=       16 usecs,   3139, or   5.59%
Elapsed Time, >       16 usecs AND <=       32 usecs,    164, or   0.29%
Elapsed Time, >       32 usecs AND <=       64 usecs,     30, or   0.05%
Elapsed Time, >       64 usecs AND <=      128 usecs,   3881, or   6.92%
Elapsed Time, >      128 usecs AND <=      256 usecs,     10, or   0.02%
Elapsed Time, >      256 usecs AND <=      512 usecs,      0, or   0.00%
Elapsed Time, >      512 usecs AND <=     1024 usecs,      0, or   0.00%
Elapsed Time, >     1024 usecs AND <=     2048 usecs,      0, or   0.00%
Elapsed Time, >     2048 usecs AND <=     4096 usecs,      0, or   0.00%
Elapsed Time, >     4096 usecs AND <=     8192 usecs,      0, or   0.00%
Elapsed Time, >     8192 usecs AND <=    16384 usecs,      0, or   0.00%
Elapsed Time, >    16384 usecs AND <=    32768 usecs,      0, or   0.00%
Elapsed Time, >    32768 usecs AND <=    65536 usecs,      0, or   0.00%
Elapsed Time, >    65536 usecs AND <=   131072 usecs,      0, or   0.00%
Elapsed Time, >   131072 usecs AND <=   262144 usecs,      0, or   0.00%
Elapsed Time, >   262144 usecs AND <=   524288 usecs,      0, or   0.00%
Elapsed Time, >   524288 usecs AND <=  1048576 usecs,      0, or   0.00%
Elapsed Time, >  1048576 usecs AND <=  2097152 usecs,      0, or   0.00%
Elapsed Time, >  2097152 usecs                      ,      1, or   0.00%
Total,                                                 56104



Total = 41573 for module dxgkrnl.sys
Elapsed Time, >        0 usecs AND <=        1 usecs,  20150, or  48.47%
Elapsed Time, >        1 usecs AND <=        2 usecs,    583, or   1.40%
Elapsed Time, >        2 usecs AND <=        4 usecs,   2529, or   6.08%
Elapsed Time, >        4 usecs AND <=        8 usecs,  12588, or  30.28%
Elapsed Time, >        8 usecs AND <=       16 usecs,   1802, or   4.33%
Elapsed Time, >       16 usecs AND <=       32 usecs,     14, or   0.03%
Elapsed Time, >       32 usecs AND <=       64 usecs,     18, or   0.04%
Elapsed Time, >       64 usecs AND <=      128 usecs,   3879, or   9.33%
Elapsed Time, >      128 usecs AND <=      256 usecs,     10, or   0.02%
Total,                                                 41573



Total = 4030 for module ntoskrnl.exe
Elapsed Time, >        0 usecs AND <=        1 usecs,     11, or   0.27%
Elapsed Time, >        1 usecs AND <=        2 usecs,     73, or   1.81%
Elapsed Time, >        2 usecs AND <=        4 usecs,   1098, or  27.25%
Elapsed Time, >        4 usecs AND <=        8 usecs,   1436, or  35.63%
Elapsed Time, >        8 usecs AND <=       16 usecs,   1276, or  31.66%
Elapsed Time, >       16 usecs AND <=       32 usecs,    129, or   3.20%
Elapsed Time, >       32 usecs AND <=       64 usecs,      6, or   0.15%
Elapsed Time, >       64 usecs AND <=      128 usecs,      0, or   0.00%
Elapsed Time, >      128 usecs AND <=      256 usecs,      0, or   0.00%
Elapsed Time, >      256 usecs AND <=      512 usecs,      0, or   0.00%
Elapsed Time, >      512 usecs AND <=     1024 usecs,      0, or   0.00%
Elapsed Time, >     1024 usecs AND <=     2048 usecs,      0, or   0.00%
Elapsed Time, >     2048 usecs AND <=     4096 usecs,      0, or   0.00%
Elapsed Time, >     4096 usecs AND <=     8192 usecs,      0, or   0.00%
Elapsed Time, >     8192 usecs AND <=    16384 usecs,      0, or   0.00%
Elapsed Time, >    16384 usecs AND <=    32768 usecs,      0, or   0.00%
Elapsed Time, >    32768 usecs AND <=    65536 usecs,      0, or   0.00%
Elapsed Time, >    65536 usecs AND <=   131072 usecs,      0, or   0.00%
Elapsed Time, >   131072 usecs AND <=   262144 usecs,      0, or   0.00%
Elapsed Time, >   262144 usecs AND <=   524288 usecs,      0, or   0.00%
Elapsed Time, >   524288 usecs AND <=  1048576 usecs,      0, or   0.00%
Elapsed Time, >  1048576 usecs AND <=  2097152 usecs,      0, or   0.00%
Elapsed Time, >  2097152 usecs                      ,      1, or   0.02%
Total,                                                  4030

Tests with WPR - Logging mode (File)

If I expand the collums I can also see this (dxgkrnl.sys) some lines that reporting normal values...

Windows Performance Analyzer shows negative DPC numbers on dxgkrnl.sys - Possibly an error of AMD drivers?

Alchemy Tweaks

Member

All necessary files and info to understand my issue

The beginning

Measurements procedure

Jeff Stokes

LAST THOUGHTS

CONTACT ME

My Computer

System One

garlin

Well-known member

My Computer

System One

Alchemy Tweaks

Member

My Computer

System One

garlin

Well-known member

My Computer

System One

Alchemy Tweaks

Member

My Computer

System One

garlin

Well-known member

My Computer

System One

Alchemy Tweaks

Member

Attachments

My Computer

System One

Latest Support Threads

Latest Tutorials

Windows Performance Analyzer shows negative DPC numbers on dxgkrnl.sys - Possibly an error of AMD drivers?

Member

All necessary files and info to understand my issue​

​

The beginning​

Measurements procedure​

Jeff Stokes​

LAST THOUGHTS​

CONTACT ME​

My Computer

Well-known member

My Computer

Member

My Computer

Well-known member

My Computer

Member

My Computer

Well-known member

My Computer

Member

Attachments

My Computer

All necessary files and info to understand my issue

The beginning

Measurements procedure

Jeff Stokes

LAST THOUGHTS

CONTACT ME