redstack

This is the first article of a (I hope) long series of articles about ‘The Basics: What everyone should know about’

The calling convention defines the way a function or a piece of code should arrange data before calling a function, and what to do after. It responds to questions like “In which order should I pass the arguments ?”, “Should I clean something ?”, “Where is the result ?”, …

There is a lot of different calling conventions. Here are the 3 I see the most of the time:

• cdecl
• stdcall
• fastcall

cdecl convention

The cdecl convention is the default one used when working with a C compiler like GCC or MSVC. To use the cdecl scheme for a function, you can use this syntax (GCC):

__attribute__((cdecl)) int function(int arg1, int arg2, ...);

GCC will produce the following code when calling a cdecl function with 4 arguments :

push   0x4 ; arg4
push   0x3 ; arg3
push   0x2 ; arg2
push   0x1 ; arg1
call   _cdecl_fct
add    esp,0x10
mov    DWORD PTR [ebp-0x4],eax

As you can see, arguments are pushed into the stack in right to left order, and it’s up to the caller to remove the arguments from the stack (Here this is done by add esp, 0x10). The result of the function is stored in the EAX register.

stdcall convention

The stdcall convention is the one used by Win32 APIs. It’s also the easyest to use when writing ASM code, in my opinion. A function can be declared as a stdcall function in C with this syntax (GCC):

__attribute__((stdcall)) int function(int arg1, int arg2, ...);

GCC will produce the following code when calling a stdcall function with 4 arguments :

push   0x4 ; arg4
push   0x3 ; arg3
push   0x2 ; arg2
push   0x1 ; arg1
call   _stdcall_fct@16
mov    DWORD PTR [ebp-0x4],eax

As for the cdecl calling style, arguments are pushed from right to left, but in stdcall mode, the caller doesn’t have to clean the arguments from the stack after calling the function. A stdcall function removes arguments from the stack before returning. This is done by using the ret n instruction most of the time.
Like for cdecl, result is in EAX.

fastcall convention

The fastcall convention is not standardized, but we will watch the way GCC and MSVC handle it. A function can be declared as a fastcall function in C with this syntax (GCC):

__attribute__((fastcall)) int function(int arg1, int arg2, ...);

GCC will produce the following code when calling a stdcall function with 4 arguments :

push   0x4 ; arg4
push   0x3 ; arg3
mov    edx,0x2 ; arg2
mov    ecx,0x1 ; arg1
call   @fastcall_fct@16
mov    DWORD PTR [ebp-0x4],eax

As you can see, not all the arguments are pushed into the stack. The first two arguments are passed via the ECX, for the first argument, and EDX, for the second argument. The remaining arguments are pushed into the stack from right to left. The called function has to pop the arguments from the stack before returning, like for stdcall.
The result is, as usual, in EAX

This entry was posted on Wednesday, January 16th, 2008 at 05:45 and is filed under The Basics. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.