Solutions: The Hex Factor v2009 (Level C300)

The binary foo level 300 reverse engineering challenge was designed to be difficult to solve, but not too hard. Designing challenges nobody can solve isn't hard, the difficult part is finding the right balance so that people can still solve the challenge in the allotted time.

The challenge is completely written in assembler, and no libraries were used. This gives me full control over every byte emitted by the assembler and linker to produce the executable. But it also implies that I had to write my own I/O routines, instead of using printf and scanf.

When you open re-300.exe with IDA Pro, you'll get this warning:



This is a good indication that the executable is packed. The name UPX1 of the code segment is another good indication:



Trying to decompress re-300.exe with UPX succeeds:

copy re-300.exe re-300-unpacked.exe
upx -d re-300-unpacked.exe

So you might be tempted to believe that the executable was just packed to obfuscate the code, and that you can solve the challenge by analyzing the unpacked executable. This is not completely true. Take a look at the UPX decompression stub of re-300.exe:



This stub is slightly different from the standard decompression stub used by UPX. I inserted the instruction dec ecx as a first anti-reversing trick.
When a normal executable is launched, register ECX has value 0x00000000 when the entry-point of the executable is called. When a UPX packed executable is launched, register ECX also has value 0x00000000. But in our challenge executable, the stub decreases the value ECX with 1 just before jumping to the entry-point. So ECX has value 0xFFFFFFFF when the packed challenge passes execution to the entry-point. If you unpack the challenge executable and run it, ECX will have value 0x00000000. Needless to say, I use this value ECX later on in the challenge, and if it's not equal to 0xFFFFFFFF, you'll never find the right answer.

So you can continue your analysis of re-300-unpacked.exe (the unpacked challenge), but remember to initialize ECX to 0xFFFFFFFF when you perform dynamic analysis (like debugging).

When you analyze the unpacked challenge, you'll notice that it uses Dave Aitel's shellcode method to locate 4 win32 API functions (GetStdHandle, WriteConsoleA, lstrlenA and ReadConsoleA), mainly used to read from and write to the console.

But here also, I've added an anti-reversing trick:



The value pointed to by [eax+2] in the PEB is equal to 0x00000001 when you execute the challenge under the control of a debugger, and it is equal to 0x00000000 when it runs without debugger (this value is set by Windows when a process is created/debugged). So when you run this code under the control of a debugger, the win32 API locating code will fail.

When you analyze the input and output routines, you'll discover they are designed to write text and input and output 8-digit uppercase hex values, like this: 89ABCDEF.

Further analysis shows that the password and the code (the answer) are not hardcoded, but that they are calculated. This is done to prevent you from just changing the logic of the challenge so that any password would be OK. If you do this, you'll get a congratulatory message, but the provided code will be wrong.

Here is the code that calculates the password and the code:



Calculating the password is done by XOR-ing and left-shifting the program code, a rather simple exercise. The calculated password is in ECX, the password you provided is in EAX, and both registers are compared to decide if you provided the correct password. So you might be tempted to debug this program, and set a software breakpoint just before the passwords are compared. But this will not give you the right answer, because of a third anti-reversing trick I implemented. When you set a software breakpoint on a given instruction, the debugger will actually replace this instruction in memory with an INT3 instruction (opcode 0xCC). This means that setting a software breakpoint actually changes the program code, and because I calculate the password by XOR-ing and left-shifting the complete program code, the calculated password will not be the correct one!

How do you solve this?

There are several solutions. You could decide to write a program that calculates the correct password (for example by translating my assembly code to Python). Or you could use hardware breakpoints (these don't modify the program code).

I'm going to show you another solution using a debugger (ODBG).

Open re-300-unpacked.exe in the debugger:



Before you start the program, notice the value of ECX:



Remember that this value should be 0xFFFFFFFF, so change the ECX register:



And don't forget the second anti-reversing trick, so set a breakpoint here:



Now start the program. When the debugger stops at your first breakpoint, modify register ECX to replace 0x00000001 with 0x00000000.



Remove the breakpoint (otherwise the password calculation routine will provide the wrong password), and set a new breakpoint just before we enter the password calculation routine:



Continue running the program, and provide a password (any password will do for now, as long as it's a valid 8-digit hex number):



When the debugger breaks at the second breakpoint, remove this too, and then single step through the password calculation routine until you hit the SUB instruction. Remember, you can't set a breakpoint on the SUB instruction, because then the password calculation would give a wrong result. But single stepping doesn't change the code.



Now, register ECX contains the calculated password: 1FF29D9B (and register EAX contains the password you provided).



You've found the correct password, now you need to find the correct code.
You could change register EAX with the correct password, and then continue to debug the program until the code is calculated. But there are 2 more anti-reversing tricks you'll need to defeat. One trick calculates the execution time of the routine (by subtracting 2 timestamps provided by instruction RDTSC), and another trick calculates an XOR value of the program code. Both tricks are blended together so you can't use breakpoints or single step (single stepping would make the timestamp delta too long).

How do you solve this? Take a step back. You've the correct password, but you don't have the correct code. Just use the original challenge (re-300.exe), run it with the correct password, and it will give you the correct code! You don't have to debug anymore, you already have the correct password:



Enjoying the masochistic assembly tricks of Didier? Get your tickets now for BruCON (September, Brussels) or at SANS London (December, London)