This is the first article of a (I hope) long series of articles about 'The Basics: What everyone should know about' :)
The calling convention defines the way a function or a piece of code should arrange data before calling a function, and what to do after. It responds to questions like "In which order should I pass the arguments ?", "Should I clean something ?", "Where is the result ?", ...
There is a lot of different calling conventions. Here are the 3 I see the most of the time:
- cdecl
- stdcall
- fastcall
cdecl convention
The cdecl convention is the default one used when working with a C compiler like GCC or MSVC. To use the cdecl scheme for a function, you can use this syntax (GCC):
__attribute__((cdecl)) int function(int arg1, int arg2, ...);
GCC will produce the following code when calling a cdecl function with 4 arguments :
push 0x4 ; arg4
push 0x3 ; arg3
push 0x2 ; arg2
push 0x1 ; arg1
call _cdecl_fct
add esp,0x10
mov DWORD PTR [ebp-0x4],eax
As you can see, arguments are pushed into the stack in right to left order, and it's up to the caller to remove the arguments from the stack (Here this is done by add esp, 0x10). The result of the function is stored in the EAX register.
stdcall convention
The stdcall convention is the one used by Win32 APIs. It's also the easyest to use when writing ASM code, in my opinion. A function can be declared as a stdcall function in C with this syntax (GCC):
__attribute__((stdcall)) int function(int arg1, int arg2, ...);
GCC will produce the following code when calling a stdcall function with 4 arguments :
push 0x4 ; arg4
push 0x3 ; arg3
push 0x2 ; arg2
push 0x1 ; arg1
call _stdcall_fct@16
mov DWORD PTR [ebp-0x4],eax
As for the cdecl calling style, arguments are pushed from right to left, but in stdcall mode, the caller doesn't have to clean the arguments from the stack after calling the function. A stdcall function removes arguments from the stack before returning. This is done by using the ret n instruction most of the time.
Like for cdecl, result is in EAX.
fastcall convention
The fastcall convention is not standardized, but we will watch the way GCC and MSVC handle it. A function can be declared as a fastcall function in C with this syntax (GCC):
__attribute__((fastcall)) int function(int arg1, int arg2, ...);
GCC will produce the following code when calling a stdcall function with 4 arguments :
push 0x4 ; arg4
push 0x3 ; arg3
mov edx,0x2 ; arg2
mov ecx,0x1 ; arg1
call @fastcall_fct@16
mov DWORD PTR [ebp-0x4],eax
As you can see, not all the arguments are pushed into the stack. The first two arguments are passed via the ECX, for the first argument, and EDX, for the second argument. The remaining arguments are pushed into the stack from right to left. The called function has to pop the arguments from the stack before returning, like for stdcall.
The result is, as usual, in EAX :)
Comments !