"The run-time environment has detected an inconsistency in its internal state." Over time, on a number of web servers (Windows 2003, IIS 6.0), with many running application pools, I noticed many - at first inexplicable - COM+ error messages. This may be caused by something known as desktop heap exhaustion.
The COM+ error messages I found were:
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in d:\nt\com\complus\src\comsvcs\threads\stathread.cpp(285), hr = 80070008: CSTAThread: CoGetApartmentID failed
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in d:\nt\com\complus\src\comsvcs\threads\staactivity.cpp(805), hr = 8000ffff: CSTAActivity: Failed to enqueue work.
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in d:\nt\com\complus\src\comsvcs\threads\activityworkqueue.cpp(364), hr = 8000ffff: CActivityWorkQueue: Failed to bind queue.
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in d:\nt\com\complus\src\comsvcs\threads\stathreadpool.cpp(1081), hr = 8000ffff: CSTAThreadPool: Unable to get bind thread.
The run-time environment has detected an inconsistency in its internal state. This indicates a potential instability in the process that could be caused by the custom components running in the COM+ application, the components they make use of, or other factors. Error in d:\nt\com\complus\src\comsvcs\threads\stathread.cpp(272), hr = 80070057: CSTAThread: CoInitializeEx failed
Desktop Heap Exhaustion
A quick Google search brought me to "Vapor Blog", and MVP blog, where he described the same issue. It turns out that the desktop heap size is the cause. By default, it is set to 512 KB and is used for all non-interactive users, including W3WP.exe worker processes. This value is set in the registry:
HKEY_LOCAL_MACHINE\System\CurrentControlSet\Control\Session Manager\SubSystems\Windows -> SharedSection = 1024,3072,512
You can read more in the Desktop Heap Overview on the Ntdebugging Blog.
Desktop heap size
The fix is to increase the desktop heap size for non-interactive users.
As you can see, 3072 KB is available for the interactive user and 512 KB for non-interactive users, in which thread handles and the like are stored. When many application pools are running and therefore many non-interactive users are logged in, the 512 KB heap is full and no more threads can be started.
After doubling the heap for non-interactive users (512 -> 1024) the problem was solved.
This registry change only becomes active after a reboot.