Q:

Confused about calling convention in a Windows kernel mode driver

Reverse engineering a kernel mode driver (in its 32-bit x86 incarnation) I stumbled over what seems to be an odd calling convention. For a driver I'd expect to see __cdecl, __fastcall and __stdcall in the Microsoft flavor. And since this driver obviously uses its own C++ runtime I'll expect to see __thiscall as well.
However, in this driver I see functions that are passed their first argument in eax. This is completely unexpected, so I am wondering if anyone here has an idea what could be going on?
sub_40XXXX      proc near
    push    ebx
    push    esi
    mov     esi, eax
    push    edi
    xor     edi, edi

Frame pointer omission doesn't seem like a credible cause for what I am seeing. Is this a bad case of LTCG?

The driver in question is a file system mini filter driver and from the looks of it I'd guess it was linked by the linker in the Windows 7 WDK (although I cannot really say which exact version, e.g. 7600.16385.1 or another). Linker version states 9.00 in the PE optional header. Subsystem version is 5.00 which would indicate that it was built to target Windows 2000. It also suggests that the subsystem was passed explicitly, since the Vista WDKs were the last ones to support targeting Windows 2000. Of course this could also have been built using VS 2008, hard to tell (the standalone WDKs used to include the same optimizing compilers as VS). For all I know it could also be a mix of compilers and linkers from different toolchains. But given that the linker needs to know about calling conventions, I'm still expecting to see some standard calling convention.
Here are the clues for the driver in question (trimmed down version of dumpbin /headers ... output):
        9.00 linker version
        1000 section alignment
         200 file alignment
        5.00 operating system version
        0.00 image version
        5.00 subsystem version

The standard calling convention for drivers is __stdcall. But to verify that __fastcall doesn't indeed use eax as I see in the other driver, I decided to create a little driver. In order to prevent the compiler or linker to optimize out my function calls, I messed up some variables subsequently passed to IoCreateSymbolicLink.
The C++ code for the respective two functions is:
UINT_PTR __fastcall fastcall_test(UINT_PTR arg1, UINT_PTR arg2)
{
    UINT_PTR ret = arg1 + arg2;
    DbgPrint("%u, %u -> %u", arg1, arg2, ret);
    return ret;
}

UINT_PTR __stdcall stdcall_test(UINT_PTR arg1, UINT_PTR arg2)
{
    UINT_PTR ret = arg1 + arg2;
    DbgPrint("%u, %u -> %u", arg1, arg2, ret);
    return ret;
}

and they're called as:
UINT_PTR fct = fastcall_test((UINT_PTR)status, RegistryPath->MaximumLength);
UINT_PTR sct = stdcall_test((UINT_PTR)status, RegistryPath->MaximumLength);

usSymlinkName.Buffer += fct;
usSymlinkName.Length += (USHORT)fct;
usSymlinkName.Buffer += sct;
usSymlinkName.MaximumLength += (USHORT)(sct + fct);

from DriverEntry between IoCreateDevice and IoCreateSymbolicLink in a default project generated using DDKWizard.
When compiling this targeting Windows XP, the outcome is as follows:
.text:00010512 unsigned int __fastcall fastcall_test(unsigned int, unsigned int) proc near
.text:00010512                                         ; CODE XREF: DriverEntry(x,x)+41p
.text:00010512 arg1 = ecx
.text:00010512 arg2 = edx
.text:00010512                 mov     edi, edi
.text:00010514                 push    esi
.text:00010515                 lea     esi, [arg1+arg2]
.text:00010518                 push    esi
.text:00010519                 push    arg2
.text:0001051A                 push    arg1
.text:0001051B                 push    offset Format   ; "%u, %u -> %u"
.text:00010520                 call    _DbgPrint
.text:00010525                 add     esp, 10h
.text:00010528                 mov     eax, esi
.text:0001052A                 pop     esi
.text:0001052B                 retn
.text:0001052B unsigned int __fastcall fastcall_test(unsigned int, unsigned int) endp
.text:00010532 unsigned int __stdcall stdcall_test(unsigned int, unsigned int) proc near
.text:00010532                                         ; CODE XREF: DriverEntry(x,x)+50p
.text:00010532
.text:00010532 arg1            = dword ptr  8
.text:00010532 arg2            = dword ptr  0Ch
.text:00010532
.text:00010532                 mov     edi, edi
.text:00010534                 push    ebp
.text:00010535                 mov     ebp, esp
.text:00010537                 mov     eax, [ebp+arg1]
.text:0001053A                 mov     ecx, [ebp+arg2]
.text:0001053D                 push    esi
.text:0001053E                 lea     esi, [eax+ecx]
.text:00010541                 push    esi
.text:00010542                 push    ecx
.text:00010543                 push    eax
.text:00010544                 push    offset Format   ; "%u, %u -> %u"
.text:00010549                 call    _DbgPrint
.text:0001054E                 add     esp, 10h
.text:00010551                 mov     eax, esi
.text:00010553                 pop     esi
.text:00010554                 pop     ebp
.text:00010555                 retn    8
.text:00010555 unsigned int __stdcall stdcall_test(unsigned int, unsigned int) endp

As expected __fastcall ends up passing arguments via ecx and edx. So what's going on with my other driver?

Meanwhile I found out how the PE header values could be achieved using a vanilla Windows 7 WDK. This will then yield a binary which is compatible with Windows 2000, assuming you build for x86, and contain these exact values (and of course assuming you don't do stupid things like statically importing DDIs only available after Windows 2000).
In sources specify ...
USE_MAKEFILE_INC=1
SUBSYSTEM_VERSION=$(SUBSYSTEM_500)
# Alternatively:
#SUBSYSTEM_VERSION=5.00

which will force nmake to include makefile.inc from the same location as the sources file and set the SUBSYSTEM_VERSION correctly (5.00 for x86 and 5.02 for amd64).
Then in makefile.inc override
LINKER_APP_VERSION=0.00
LINKER_OS_VERSION=$(SUBSYSTEM_VERSION)

which works because makefile.inc gets included quite late from makefile.new and therefore we can use it to override the defaults specified by the default build environment.

A:

Most likely it's just LTCG/LTO, especially if the function in question is only never called externally. Using it can result in the following:

Cross-module inlining
Interprocedural register allocation (64-bit operating systems only)
Custom calling convention (x86 only)
Small TLS displacement (x86 only)
Stack double alignment (x86 only)
Improved memory disambiguation (better interference information for
global variables and input parameters)

