Machine Code

With great appreciation of the original MCode Tutorial by nnnik

Machine code is the lowest level of binary code that your computer can run on. Programming languages like C, C++, Rust, and Go all compile to machine code in order for your computer to understand them. Machine code can run several hundred times faster than equivalent AutoHotkey code, which does not compile to machine code.

In the AutoHotkey community the term "MCode" refers to tools and methods used for putting machine code into scripts. These MCode tools normally take code written in a language like C, use a compiler to turn that code into machine code, then turns that machine code either directly into AutoHotkey code or into text that can be loaded using a custom AutoHotkey library and then called by DllCall.

MCode is important for optimization when writing scripts that need to process a relatively large amount of data quickly. Here are some common situations where MCode can be helpful:

• Encoding or decoding data more than a few kilobytes in size, in formats like Json
• Hashing files or large amounts of text
• Manipulating images like GDI+ bitmaps (especially for custom ImageSearch algorithms)
• Performing real-time calculations, such as for a physics engine

MCode is not the only way to achieve these performance goals. It is possible, and sometimes more flexible, to use the normal tooling of those compiled languages to produce a standard machine code DLL that can be used from AutoHotkey. However, a script that comes with a custom DLL is harder to share because it takes multiple files, it can take up more disk space than a normal script, and is more likely to be blocked by antivirus and corporate application filters.

Before you can get started with MCode, you will need:

• Some understanding of how to read and write C or C++ code
• A basic understanding of how to use DllCall to interact with C APIs (like the Windows API)
• An MCode library which can create and load MCode
• A compiler compatible with your chosen MCode library

If you are not very familiar with programming in C or C++, a great place to start is on Codecademy which offers a variety of free interactive programming courses.

There are not very many MCode libraries, but the most comprehensive is MCL [v1][v2]. Other options are listed on the Libraries page.

If using MCL, the easiest compiler to install is likely TDM-GCC if you do not already have a compiler installed. MCL will support most GCC compatible compilers.

One of the simplest MCode projects you can do is to have a bit of MCode that can return a number.

For example, the C code:

int alpha() { return 42; }

With a function this trivial, it is possible to manually convert the C code to MCode using a compiler like godbolt. From godbolt the C code can be entered, a compatible compiler can be chosen like "MinGW gcc", and the output settings can be changed (enabling "compile to binary object") in order to reveal the machine code generated from the function. By default it outputs 64-bit machine code, so if you are using 32-bit AutoHotkey you may need to add -m32 to the godbolt compiler options in order to get compatible machine code.

The hex code on every other line of the output can be entered into a script to form MCode. The simplest method is using a Buffer and NumPut to write the code into that buffer. When writing large amounts of data it is most efficient to write the data in groups, using the Int64 type rather than trying to write each individual byte with the Char type.

To do this, construct a buffer large enough to hold the bytes of machine code, rounded up to the nearest multiple of 8. Note that we can safely ignore any 90 NOP bytes at the end of the code. In this case, 11 bytes were output by the compiler so we will make a buffer of 16 bytes.

Next, split the bytes into groups of at most 8: 55 48 89 e5 b8 2a 00 00 / 00 5d c3

Now reverse the bytes in each group: 00 00 2a b8 e5 89 48 55 / c3 5d 00

Join the bytes together and add the 0x prefix to create your constants: 0x00002ab8e5894855 / 0xc35d00

Now you can assemble your mcode:

By calling the buffer with DllCall, execution will jump to the first byte of the machine code and execute whatever function is at that position. For code compiling only one function, it will normally be the start of that one function.

A function that just returns one number is not very useful. A more useful function could be one to find the length of a string:

int StringLen(short *str)
{
	int i = 0;
	for (; str[i] != 0; i++) {}
	return i;
}

This function loops over the characters of a unicode string until it reaches the end, then returns the count of characters.

When fed into godbolt, we get an output like 55 48 89 e5 48 83 ec 10 48 89 4d 10 c7 45 fc 00 00 00 00 eb 04 83 45 fc 01 8b 45 fc 48 98 48 8d 14 00 48 8b 45 10 48 01 d0 0f b7 00 66 85 c0 75 e4 8b 45 fc 48 83 c4 10 5d c3 which is very quickly getting out of hand.

Performing the same manipulations as before, we can write the MCode as follows:

As the functions get more complicated, the need for a dedicated MCode compiler becomes more apparent. An MCode compiler will take your C / C++ input and turn it into something you can put directly into your AHK script.

When provided to joedf's MCode4GCC, the provided function will output a string like this 2,x86:i1QkBDHAZoM6AHQUjXQmAIPAAWaDPEIAdfbDjXQmAJDD,2,x64:ZoM5AHQiuAEAAAAPH0QAAEGJwEiDwAFmg3xB/gB18USJwMMPH0QAAEUxwESJwMM= which can be provided to a pre-written mcode loader function in order to load your machine code. Noting that it compiles for both 32 and 64 bit.

Similarly, when provided to the MCL compiler it will output the loader itself, with the 32 and 64 bit MCode already baked in:

With the MCL library on a system which has a compatible compiler installed, you can skip the need to separately compile and then paste a loader into your script to test. Instead, compiling and loading can be done as a single step inside your script:

It is only once you are done developing your MCode, you would use the MCL.StandaloneAHKFromC method to generate the standalone loader that does not require a compiler, to be pasted into a script.

When working with C code that defines multiple functions, MCode becomes much trickier to generate. Consider the code as follows:

int alpha() { return 42; }
int beta() { return alpha(); }

When compiled using godbolt, we see two problems arise:

1. When the machine code is generated, only alpha is at the start of the machine code, so when your MCode is called only alpha can be executed. This can be fixed by counting the byte offset of beta from the start of the code and calling DllCall(mcode.Ptr + betaOffset, …
2. If beta were to be called, it doesn't know where alpha is to call it. If you look in the code, it defines the call to alpha as c8 00 00 00 00 which means to enter the function at memory location 0, not memory location of alpha.

Traditional MCode tools like MCode4GCC do nothing to address these problems. Instead, you have to compile and load each function separately, managing and passing around function pointers manually to everywhere they are used.

This kind of workaround is not necessary when using MCL, which uses its built-in linker and loader to automatically identify function offsets and adjust the MCode at run-time so that references to other functions resolve correctly instead of remaining 0.

For code that defines multiple functions to be called from AHK, those functions must be exported so that MCL can determine what should be included in the output MCode. Functions that are not exported, or used by exported code, will be trimmed from the output automatically.

To export a function, you must include the MCL header file and then use the MCL_EXPORT macro.

Optionally, the export macro can accept DllCall types as parameters. When these are specified, the exports will be exported as pre-made wrappers that can be called without having to pull out DllCall and specify types in your AutoHotkey code.

Although traditional MCode compilers like MCode4GCC did not have any tools for constructing and managing global variables inside your MCode, MCL supports this readily.

Like exporting functions when working with multiple functions, you can export global variables with the MCL_EXPORT_GLOBAL macro. This provides the user with a pointer to where that global lives in RAM, so that it can be modified with NumPut and NumGet.

Like with exporting functions, exporting a global can also export with typing information to form a wrapper.

Although there are many things you can do with MCode, sometimes it is helpful or even necessary to callback to AutoHotkey for some actions. For example, it can be extremely valuable to call back to AutoHotkey for OutputDebug, MsgBox, or other debugging-oriented logging tools.

AutoHotkey provides a tool to export its own functions to be consumed as a callback from C/C++ APIs, CallbackCreate, which can be used very easily with your own MCode functions as well. The simplest way to do this is to create your callback, then pass it as a parameter to the function.

When working with MCL, you can do a little better by passing the functions as global variables instead of parameters. This lets you more easily add and remove debug functions to be used all throughout the MCode without having to update all your function calls each time.

It may be tempting to try to use CallbackCreate in order to wrap mathematical functions like Sqrt, however this should be avoided normally because it will absolutely destroy any performance gains that you were aiming to achieve by using MCode in the first place. Instead, native machine code implementations of sqrt and other mathematical functions should be embedded or imported.

Writing C or C++ code in an MCode environment, without any access to the standard libraries, can be extremely limiting. These limits can be eased significantly by importing key functions from DLLs that ship with Windows, or from third-party DLLs that you have on hand.

The basic strategy is this:

1. Load the DLL using AutoHotkey
2. Fetch the function pointer using AutoHotkey
3. Pass that function pointer using the same strategy we did for CallbackCreate
4. Call that function from your MCode

For example, we can import sqrt from msvcrt.dll in order to do some mathematical calculations:

When working with MCL, importing functions like this can be handled automatically by the MCL_IMPORT macro. Any standalone loader generated by MCL that has an import like this will automatically include any AHK-side import loading code that would otherwise have to be written manually. Imports done this way are automatically added to the global scope, so they do not have to be passed as a parameter to DllCall.

MCL can also be used to import functions from third-party DLLs, such as lua54.dll. For more information about that, please refer to the library page for MCL.

MCode libraries generally support generating both 32 and 64 bit machine code, but they may not generate both at once.

Although it is generally recommended to use AutoHotkey U64, when 32 bit is required there are some things to keep in mind. C functions in 32 bit machine code default to the "Cdecl" calling convention, which will cause memory leaks if not accounted for. To account for that, you must either adjust any DllCall lines to specify "Cdecl" (which you can read more about on the DllCall documentation page) or you can adjust your C function declaration to use the standard calling convention by adding _stdcall to its signature:

int _stdcall SomeFunc(int arg1, int arg2) {}

When exporting functions using the MCL_EXPORT macro, you can specify Cdecl_ as a prefix to any type in order to ensure the Cdecl calling convention is used. For example, MCL_EXPORT(alpha, Int, a, Int, b, Cdecl_Int).