DLL Conventions: Issues and Solutions - Part II

In the previous part of this series we saw the basic issues encountered when we try to use DLLs in an environment different from the one in which they were developed. We discussed the problems with plain C style functions exported from the DLLs, and the cause of these problems.

In this part we will elaborate on the calling conventions and how to resolve these differences, after we learn how to identify which calling conventions are in use for a specific exported function. Near the end we'll also take a look at the different tools and utilities available to help accomplish these tasks easily.

We'll also write some dummy DLLs and analyze them to get a firm grasp and clear understanding of the problems and the way we solve them.

This is something that happens at a very low level, say at the time your C or C++ code is compiled to native CPU instructions, and comes into the picture then. A good understanding of calling conventions can help you write better code, debug the code better, and finally enable you to use a source level debugger like Softice. But this is not something we'd like to get into. So we'll limit the discussion to our purpose -- how calling conventions affect functions exported from a DLL, and what benefit we get if we can find the calling convention used for a specific export.

When a function is invoked, whether it resides in a DLL or in your own source files, it receives its arguments/parameters through the stack. These arguments/parameters are set up for the called function by the calling function. We can use these parameters, but how does this frame of stack values get cleaned up? Of course it'll be done by either the caller or the callee. And this, finally, is specified by the specific calling convention used. You may have seen a few in action, but never took the time to understand what they actually are. Never mind, below is a list of keywords that will refresh your memory.

__pascal, __fortran, and __syscall,  __cdecl, __stdcall, __fastcall

Out of these PASCAL, FORTRAN and SYSCALL are obsolete and their use, even if supported, is discouraged. When we use the CDECL calling convention, the calling function is responsible for cleaning up the stack that it set up before the call was made, whereas in STDCALL and FASTCALL, the callee or the called function is responsible for the cleanup of stack.

Apart from the listed ones THISCALL ( thiscall ) is another convention which is the default calling convention used by C++ member functions that do not use variable arguments. Under thiscall, the callee cleans the stack, and as this is not a keyword, unlike other calling conventions, you can't explicitly specify it. To put it simply, you can assume it to be used whenever a class member function is called which uses the this pointer (that's right, static members don't use thiscall). Essentially, the thiscall convention applies to C++ only.

So now that you understand what these __cdecl and __stdcall mean, let's move on to identifying the convention used in functions exported to a DLL. For the purpose of demonstration you may use the DemoDLL project (as well as the compiled binaries) provided as a download with the article.