IOCP Thread Pooling in C# - Part I - The Article (Page 2 of 3 )
A basic understanding of C# is required to follow through the examples and the classes. Basic concepts of type, properties, threading, synchronization, and delegates are required.
Defining the Problem
IOCP thread support has not been made available to C# developers through the “System.Threading” namespace. We need to access the Win32 API calls from the Kernel32.dll. This requires us to write unsafe code. This is really not a problem, but something that needs to be discussed. Let’s take a look at the Win32 API calls we need to implement an IOCP thread pool.
This Win32 API call is used to create an IOCP thread pool. The first argument will always be set to INVALID_HANDLE_VALUE, which is 0xFFFFFFFF. This tells the operating system this IOCP thread pool is not linked to a device. The second argument will always be set to 0. There is no existing IOCP thread pool because we are creating this for the first time. The third argument will always be null.
We do not require a key because we have not associated this IOCP thread pool with a device. The last argument is the important argument. Here we define the concurrency level of the thread pool. If we pass a 0 for this argument the operating system will set the concurrency level to match the number of CPU’s in the machine.
This option gives us our best chance to be scalable and take advantage of the number of CPU’s present in the machine. This API call will return a handle to the newly created IOCP thread pool. If the API call fails, it will return null.
This Win32 API call is used to post work in the IOCP thread pool queue. Other threads in our application will make this Win32 API call. The first argument is the handle to the IOCP thread pool. The second argument is the size of the data we are posting to the queue. The third argument is a value or a reference to an object or data structure we are posting to the queue. The last argument will always be null. The following diagram shows how the data is associated with the posted work.
In diagram E, we have two threads actively processing posted work and one piece of work on the queue waiting for its data to be processed. The thing to note here is that each piece of work was given a reference to its specific data. I am calling this variable pData to help describe what is happening in the IOCP thread pool. The actual name or structure of this variable is undocumented.
When we make this API call in a C++ application, we can pass the address of any object in memory we wish, as in diagram E. In C#, we don’t have the same luxury because of the managed heap. The managed heap is a contiguous region of address space that contains all of the memory allocated for reference variables. The heap maintains a pointer that indicates where the next object is to be allocated, and all allocations are contiguous from that point. This is much different from the C-runtime heap.
The C-runtime heap uses a link list of data structures to reference available memory blocks. For the C-runtime heap to allocate memory, it must walk through the link list until a large enough block of free memory is found. Then the free block of memory must be resized, and the link list adjusted.
If objects are allocated consecutively in a C++ application, those objects could be allocated anywhere on the heap. This can never happen with the managed heap. Objects that are allocated consecutively in a C# application will always be allocated consecutively on the managed heap. The catch is that the managed heap must be compacted to guarantee the heap does not run out of memory. That is the job of garbage collection.
For more information on garbage collection, try these links:
In diagram F, we have allocated four objects on the managed heap. Imagine that the managed heap has allocated memory for these objects at address FDEO, FDDO, FDCO, and FDBO. This would mean the value of pClass1 is FDEO, the value of pClass2 is FDDO, the value of pClass3 is FDCO, and the value of pClass4 is FDBO.
MyClass pClass1 = new MyClass(); MyClass pClass2 = new MyClass(); MyClass pClass3 = new MyClass(); MyClass pClass4 = new MyClass();
Now we write the following code.
pClass2 = null;
Diagram G shows what happens to the managed heap after garbage collection takes place and the managed heap is compacted.
The Class 2 object has been removed from the managed heap and the Class3 and Class 4 objects have been moved. Now the value of pClass3 is FDDO and the value of pClass4 is FDCO. The value that the pointer points to has changed. The garbage collection process changes the values of all reference variables to make sure they are pointing to the correct objects after the managed heap is compacted.
So what does this mean for our IOCP thread pool implementation? If we pass the reference of a managed object as the data for the work, there is a chance the reference is no longer valid when a thread in the pool is chosen to work on the data.
In diagram H, we have passed a reference to the Class 3 object as the data for the work posted to the IOCP thread pool. This object is at address FDCO. Before the work is given to thread 1, the Class 2 object is marked for deletion. Then the garbage collection process runs, and the managed heap is compacted. Now in diagram I, the work has been given to thread 1 for processing. The value of pData is still FDCO, but Class 3 is no longer at address FDCO, it is at address FDDO. The thread will perform the work, but using Class 4 instead of Class 3.
The garbage collection process can not change the value of pData, as it does with other variables, because this variable is not a managed variable. It is a variable owned by the IOCP thread pool and exists outside the scope of the CLR. The garbage collector has no knowledge of this variable or access to this variable. The variable is set during the unsafe call to PostQueuedCompletionStatus.
Unfortunately, pinning the objects we want to pass as the data for the work posted to the IOCP thread pool is not a possible solution. Pinning provides the ability to prevent an object from being moved on the manage heap during the garbage collection process. We can not pin these objects because there is no way to pin an object in one thread and unpin the object in a different thread. To pin an object, we need to use the fixed keyword. This keyword can only be used in the context of a single method. Here is a quick example of pinning.
The safest thing we can do is pass a value to the IOCP thread pool. This value could be the index from a managed array, containing a reference to an object on the managed heap. If the garbage collection process does compact the heap, the index values of the array will not change. In diagram J and K, we can see one way to properly pass data for the work posted to the IOCP thread pool. After the garbage collection process compacts the heap, the values of pData change, but the index positions to the pData variables do not change.
The final Win32 API call is used to add threads to the IOCP thread pool. Any thread that makes this Win32 API call will become part of the IOCP thread pool. This is a blocking call and the method will return when the IOCP thread pool chooses the thread to perform work.
The first argument is the handle to the IOCP thread pool. The second argument is the size of the data associated with the work. This value was provided when the work was posted. The third argument is the data value or data reference associated with the work. This value was provided when the work was posted. The forth argument is the address to a pointer of type OVERLAPPED.
This address is returned after the call. The last argument is the time in milliseconds the thread should wait to be activated to perform work. We will always pass INFINITE or 0xFFFFFFFF.
These are the Win32 API calls we need to add IOCP thread pool support to our C# server based applications. We need to encapsulate these Win32 API calls using .NET threads and minimize the sections of unsafe code. We need to prevent the application developer from passing a reference variable into the IOCP thread pool, by restricting them to passing only integer values.