Sudden Severe Memory Leak

I have a messaging/communication system that I've built, currently running on a Windows 11 machine, for a BOINC server farm. It provides a central console function. It runs many conventional DOS-command batch (.cmd) files that I've written, debugged and tweaked over the years. They are complex command files, retrieving system and BOINC manager statuses from other Windows 11 machines, parsing directory contents for message files and performing complex text parsing operations. One command file collects system status data and sends data out through a serial port to an Arduino which drives a series of lights.

The thing is, this system has worked nearly flawlessly for 8 years, up to a few days ago. It has worked on numerous Windows 10 systems on many different hardware builds. It literally runs for months without any intervention. I migrated it to an AMD NUC Windows 11 system last year and it continued to run without any issues.

Then suddenly a few days ago, without me making any changes anywhere, including not installing any Windows Updates, I noticed that the system was stressed. The 8GB memory in the system was exhausted. The memory utilization was 99% and the system was thrashing. Looking at Task Manager and Process Explorer, the list of items in memory was highly fragmented, but the total memory used by these programs and processes did NOT add up to the system's 8GB memory - NOWHERE CLOSE. 85% of the memory usage was not in the list of programs and processes - it was lost.

I shut down all the command files. The memory usage did not recover. It sat at 99%.

Ok, so I've suddenly developed a severe memory leak that I can't trace. I have no idea what is suddenly eating all the memory since it is not showing up in Task Manager or Process Explorer.

I've checked the memory with the built-in Windows memory tester (I know it isn't the best but it was a starting point). I've scanned for malware, I've run SFC /SCANNOW and it found two bluetooth drivers that were corrupt and it fixed them. I've checked the integrity of the only drive (C:) and it is fine. I don't use browsers on the machine, no email.

I tested the machine for days without anything running and the memory utilization stayed around 50%. As soon as I launch any of the batch .cmd files, the memory utilization climbs until all memory is exhausted.

The system is a stock Intel NUC (NUC11ATKC4), the OS is Windows 11 Pro version 10.0.26100 Build 26100, the processor is an Intel Celeron N5105, 2GHz, 4-core, which has not been tinkered with in any way (i.e., no overclocking; BIOS is at factory defaults).

Unless I can find out what has happened I'm going to have to wipe the system and start over from scratch to see if that solves the issue. This is painful, and I'd like to avoid that if I can.

Any thoughts? Thank you.

In that case I would download Superantispyware and do a full system scan, then remove all results. Better to do that in Safe Mode. If you find too many results (over 500), I would use a second application such as Malwarebytes Antimalware to find anything the first application missed. In very severe cases I would consider doing a backup and then format and reinstall Windows.

Do you not have an image made when all was working well? If so restore it.

Hi Spapakons,

Thanks for your suggestion. I used Malwarebytes since I'm already familiar with the product and have used it for years. I scanned the entire system with all the options available from Malwarebytes and it found nothing. That, together with the security suite already installed (F-Secure) which found nothing also, I think the system has not been compromised by malware.

Hi Glasskuter,

Thanks for suggesting this. I do have an image that was taken prior to this problem arising. I will restore it and see if the problem goes away. I will make a new image first, though, to preserve what I have in case I need to undo the restoration. It will take some time to go through all of this but I'll report back on the results.