Out of Memory, Find Leaked RAM.
Recently, I have always encountered an out of memory exception. I have installed 32GB of memory for my Windows system, but the system always says it is not enough. Increase virtual memory multiple times, even using another RAID M.2 SSD as swap memory and changing the settings of virtual memory (pagefile.sys, hiberfil.sys, swapfile.sys) are also useless
What happens to the computer when out of memory?
Out of memory is in an unstable state. Don’t assume that every program has out of memory testing or protection. In addition to the various crashes of the program, the most serious is that the display driver cannot work at all and then you will see nothing of your screen. If you cannot open the background remote DEBUG, you can only restart the computer by pressing the RESET button. One of the most troubling things is playing games, inexplicable delays and even crashes. This problem has been particularly noticeable since I install my new 4K screen.
The problem is such a nightmare. Typically, the computer will restart or crash in less than a week. I often want to know if my computer is growing a Trojan horse or a virus. It took a lot of time to update the software, scan for viruses, remove Trojan horses and delete them, but you still can’t solve the problem.
Is It a Microsoft Problem?
I decided to solve this problem. Finally, it took a whole day to find it and there is only one imperfect solution.
Task Manager is the primary tool often used to view system health. Obviously, task management shows that my system has nearly 50% of the available physical memory, but you still get an out of memory warning and ask you to free up more memory. A bit strange. How is it possible? We use Microsoft Poolmon to check for huge memory leaks in kernel memory, but it looks good and there are no kernel level leaks.
For my case, you see a huge amount of cache memory in the task manager, and it gets higher and higher. Cache memory is a mechanism for buffering data with free memory, so you won’t load data from slow disks again. When there is not enough memory, the cached memory will return available memory so that you can use the memory for your application.
Now, we open system resource manager to observe the memory usage. You see the huge amount of Standby memory. The standby memory will grow to 12-15GB.
The available memory = Free Physical Memory + Standby Memory
Since we found Standby to be a problem. We can take a closer look at the more detailed questions.
We use RAMMap (the great tool of SysInternals) to view more realistic memory usage. Finally, we found the root cause. The mechanism of standby memory turning into the usable memory mechanism does not work properly on Windows 10. Eventually all of the memory is in standby, even though the system tells you that there is still a lot of available memory, but the system can’t actually find the free memory page for use.
This is really bad. A major problem in memory management. But this problem does not seem to happen on every computer. The real reason is not clear. This issue still exists with the latest version of Windows 10 1809. My observation seems to be related to the massive use of the file IO.
Solution!
The system was unable to successfully switch back to available memory. Then we have to force the system to do this conversion manually.
Use this gadget to force the system to empty the standby memory. This is equivalent to manually turning the standby memory into available.
So when should I use it? There is no way to find this effectively, so it doesn’t seem to be a clever way to write tools to monitor problems. Perhaps the easiest way is to determine when to do this on a regular basis.
Performance analysis:
Executed per 60 minutes, “Standby” will accumulate to about 5GB.
Executed per 30 minutes, “standby” will accumulate to about 2GB.
Honestly, this is really not a good solution. Standby memory plays an important role in improving system IO performance. Emptying standby memory may interfere with this mechanism. Unfortunately, we can’t let Out of Memory happen. This might be my best solution before we or Microsoft solve the problem.
How to Schedule It:
Like Linux / Unix / Mac Cron, Windows has its own schedule. The previous AT command is no longer used. Use system task schedule to assign the a task to run “Empty Standby Tool”.
Open Windows Task Scheduler:
%windir%\system32\taskschd.msc /s
Additional information:
After updating to 1903, this problem seems to have improved. The standby memory problem is solved (need more tests).
We also find a tool named PoolmonX, a GUI version Poolmon. It’s a standalone tool and you don’t need to install the Windows SDK.
Another Memory Leak And Hard to Track:
My new computer has 64GB of RAM installed, but it still suffers out of memory for some weird case. This problem is not the one mentioned above that Standby cannot be converted into usable memory. This issue may be related to driver or resource leak. Once again, we have to use the RamMap tool to observe the problem.
As you can see, you can observe a large number of Shareable memory and are in the Modified state but the Active is still keeping low. This problem is sometimes difficult to detect. Because these resource consumption cannot be represented on the process’s Private Bytes. It’s hard to see which application or driver is causing the problem but your available RAM is getting less and less.
Due to committed limit, once you run out of committed memory pages, you can not find enough free memory pages for your free physical memory.
Process Hacker (ProcessHacker is renamed System Informer)
Process Hacker is a tool similar to Process Explorer. But it provides some good observation tools.
In order to trace the Shareable Memory Leak we just said, we have to go back and use the system built-in Task Manager.
In the detailed field, sorted with the Handles. The unreasonable uses clearly tell us that the application itself is very problematic. So we try to close it and then observe the changes in RamMap.
Before，
After，
We can notice that a large number of Modified has disappeared when the problem application is closed. About 3GB of RAM is released to Standby or Unused memory.
Update 2020.04.20,
Alternative way to find out the process consumes lots of memory.
Open Task Manager -> Details -> Right Click on List Title -> Select Column -> Commit Size
Now, you could sort the process in commit size.
Commit Size = Physical Memory + Virtual Memory
You can observe the commit size to understand how the process uses memory. It is very useful to find out the processes with leak memory.
Tools
Poolmon: A Microsoft tool form Windows SDK.
PoolmonX: GUI version Poolmon. A standalone tool and you don’t need SDK.
RAMMap: The tool you must have for memory usage monitor!
ProcessExplorer: The best task management tool.
ProcessHacker: Another Process Explorer.
System Informer: ProcessHacker is renamed System Informer
Empty Standby List: Turn Standby memory to free.