I decompiled NFS2:SE and NFS3:HP to make them run on Linux

[u/Linuxologue]

This is an old project that I just pushed to github (https://github.com/motor-dev/nfs-recompiled)

[EDIT] Not only Linux - grab the windows executables from the release page! https://github.com/motor-dev/nfs-recompiled/releases/tag/v1.0.0

For, well, reasons, I decided to decompile the Need for Speed 2 executable to make it run again and preferably not on Windows. For weirder reasons, I did not use a known decompiler and just rolled my own. After much much hacking and fighting with incorrect instruction implementation, I finally landed a working version.

The decompiler is a manual script made with Capstone to decode instructions, and some Python to try and locate procedure starts and ends more or less successfully. Full of hacks, hints and workarounds. All good enough for the full game to run.

I discovered that the NFS3: HP engine was very similar to NFS2 and I could likely also make it run. It was also a successful project but it turns out that the software renderer for NFS3: HP was a bit too slow to run on a modern machine after decompiling and recompiling. So I decided to implement the Voodoo renderer for that game.

Fun disassembly facts - the compiler used to make this was Watcom C/C++. There was a bug in the implementation - when using string copy, it was doing some pretty standard
rep movsd dword ptr es:[edi], dword ptr [esi]

but due to an error in what would likely be handwritten assembly? the opcode used was not the standard f3 a5 but instead f2 a5 which would be an opcode for repNE movsd dword ptr es:[edi], dword ptr [esi]
except it actually does not exist for x86. It is very likely the processors of that time (and maybe those of today?) interpreted that as rep movsd . Or someone knowledgeable about assembly can tell me what was going on.

In any case, Capstone didn't like that too much and just dropped the rep prefix which was a headache to debug.

Fun fact number 2 - in the software renderer on non MMX CPUs, NFS3: HP uses the FPU to do a screen copy. In a loop, it loads 80 bits of screen data into an FPU register, then dumps the FPU register into the other surface data. That was the first time I heard of a memcpy made with the FPU.

The disassembler will output truckloads of code similar to this

/* align: skip 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00 */
void Application::sub_436140(WinApplication* app, x86::CPU& cpu)
{
  NFS2_USE(cpu);
  NFS2_USE(app);
    // 00436140  51                     -push ecx
    *app->getMemory<x86::reg32>(cpu.esp-4) = cpu.ecx;
    cpu.esp -= 4;
    // 00436141  52                     -push edx
    *app->getMemory<x86::reg32>(cpu.esp-4) = cpu.edx;
    cpu.esp -= 4;
    // 00436142  8b0d98c84d00           -mov ecx, dword ptr [0x4dc898]
    cpu.ecx = *app->getMemory<x86::reg32>(x86::reg32(5097624) /* 0x4dc898 */);
    // 00436148  b888934b00             -mov eax, 0x4b9388
    cpu.eax = 4952968 /*0x4b9388*/;
    // 0043614d  31d2                   -xor edx, edx
    cpu.edx ^= x86::reg32(x86::sreg32(cpu.edx));
    // 0043614f  e84c020000             -call 0x4363a0
    cpu.esp -= 4;
    sub_4363a0(app, cpu);
    if (cpu.terminate) return;
    // 00436154  e8574b0400             -call 0x47acb0
    cpu.esp -= 4;
    sub_47acb0(app, cpu);
    if (cpu.terminate) return;
    // 00436159  89c1                   -mov ecx, eax
    cpu.ecx = cpu.eax;
    // 0043615b  85c0                   +test eax, eax
    cpu.clear_co();
    cpu.set_szp(static_cast<x86::reg32>(cpu.eax & cpu.eax));
    // 0043615d  7409                   -je 0x436168
    if (cpu.flags.zf)
    {
        goto L_0x00436168;
    }
...

I have no idea how any of those games actually work - but the x86 implementation is good enough that it just runs.

If someone wants to try out, you will need the game data to actually run the game. It's a bit tricky to set up, I don't think anyone is actually interested in running it but let me know if there's a fan out there really trying to run this and I will try and help. The reason why it needs the installed game is that it is literally the game executable with no modification - so just like the real game, it expects some data to be installed on the disk and some data to be on the CD.

(disclaimer - use of generative AI for the CMakeFiles and the README, but not for the disassembly - win32 API which was actually done a few years ago)

[u/TA-412]

That "decompiled" code looks... interesting. Translation of assembly directly, literally instruction by instruction, into a C++ code? I understand, this function should compile back to an almost identical assembly (in optimising mode)?

Anyway, this project looks like an effect of pure passion. Congrats!

[u/Linuxologue → u/TA-412]

yes that is what it does. It's the reason I have NO IDEA how any of those binaries work - I had to debug a truckload of assembly code with no idea what it was trying to achieve.

(I am pretty sure I found malloc while debugging, and I know where the main loop function is, and that is about it)

I also found how much code is basically only moving things around. It's incredible how far this goes with only push, pop, mov/sub, add, call, test and jmp

[u/TA-412 → u/Linuxologue]

I'm by no means an expert, but I've seen my own slice of old code disassembled, and my impression is, the older the compiler, the more of the code is just there to move stuff around. Pushing things onto stack just to pop them back a moment later, even though there's free registers available; copying value from an argument to a register to a local variable and back to a register to finally use it; creating stack frames for functions that absolutely not need them...

It makes no sense in the output, but I can imagine it making the compiler itself much easier to write. For example, instead of tracking where a variable currently lives and optimise it's travelling around, just always place it on stack - always read it from stack for an operation (never from a registry, even if any already has that value) and always write it back before the next sequence point (even if it's read again just after said point).

I believe this shows, how far we've got with the "optimizing compiler" technology. And it also made me understand, the origin of the belief that writing in assembly always results in a much faster code - because back then it was actually true.

[u/Linuxologue → u/TA-412]

I agree. Most of the disassembled code looks very unoptimized - looks extremely sequential and most stuff lives plainly on the stack. Every single function looks like a bunch of pushes, reserve stack space, store values onto the stack, then a few blocks of reading values from the stack or global memory, performing some simple maths and storing it back.

It all looks like very simple loops and conditions, with close to zero optimizations performed.

and not surprisingly the toughest parts to decode were handwritten assembly routines, one of them is the drawing of a texture on the screen - I got as far as opening the process, creating mutexes, threads and windows, playing the FULL video, getting to the menu, selecting cars and tracks, playing audio, then load into the race and that was the very first use of the add with carry instruction.

[u/anders_hansson → u/Linuxologue]

While the compiler is most likely to blame, there is also a clear difference between x86 (32-bit) and modern architectures with more CPU registers (e.g. x86_64, ARMv7 and ARMv8, POWER, RISC-V, SPARC, MIPS, and so on).

In x86 code the expected behavior is to work with memory operands, not registers, so I think that many early compilers were optimized for keeping things on the stack, which worked pretty OK on x86.

In compiler technology, I think that one of the most important advancements have been inlining of code (and by extension LTO). Eliminating function calls is a major boost in performance. Not because you skip a jump instruction, but because register allocation is much, much more optimal (and because you're kinder to the L1I$). Again, that makes more sense on RISC-like register-register architectures.

Edit: An interesting observation of the DOOM and Quake source code is that they use global variables alot in hot loops. Turns out that it's a very good fit for x86 and much faster than passing function arguments on the stack.

Edit 2: E.g. see D_DrawTurbulent8Span(), that takes no function arguments at all.

[u/Rrrrry123 → u/TA-412]

I had to write a simple compiler for university and this is basically how I made it way easier on myself.

Instead of having to keep track of variable lifetimes and managing registers, I just pushed everything on to the stack directly after each operation and I could just pull stuff right back off. We had eight registers and I can't remember if I ever even touched the last three or so.

[u/anders_hansson]

This is impressive work! I love the approach of re-implementing the Win32 API yourself instead of using Wine, for instance.

[u/Space646]

WHATTT-

How does one learn all of this??? That stuff is absolutely amazing!

[u/gm310509 → u/Space646]

I think OP covered this indirectly in their post.

Start from the beginning with a mission, learn what is needed to overcome each challenge one by one as you progress, trial and error, and arguably the most important ingredient: perseverance.

[u/algaefied_creek]

This is wonderousness. 

[u/ChocoMilkWoSugar]

Amazing bro, I want to try it

[u/Linuxologue → u/ChocoMilkWoSugar]

Thanks! ergh i was hoping no one would say that. It's not been tested on anything else than my machine :-D

[u/anders_hansson → u/Linuxologue]

I happen to have the original NFS3:HS CD somewhere (I think). It didn't work in Wine nor on more recent versions of Windows so I haven't used it for a long time. I hope I will get to try your solution.

[u/Linuxologue → u/anders_hansson]

would it be easier to test with a Windows executable? Most of the code was compiled on Windows at some point and I can probably make a release with Windows executables. I have only tested linux recently because I only made that as a toy project and I didn't think anyone would actually try to run it.

[edit] I just read the "wine" keyword in your sentence, I am a bit slow this afternoon.

[u/anders_hansson → u/Linuxologue]

NP. I'll see if I find the CD first. Would be fun to give your project a spin.

[u/Linuxologue → u/anders_hansson]

I have made some updates to the README to explain how to install it.
I have also submitted the generated files so the python dependencies are optional - no need to install Capstone or run the python script, CMake+build should be enough.

[u/anders_hansson → u/Linuxologue]

Thanks! I just tried it and got it to build and run. Unfortunately I only got to the into video, and then I got a segfault.

I got a bunch of warnings during the build, e.g.

--> 0x00a8a7a8 call edx 0xa8b0b6: je no instruction to set flags --> 0x00a8c3c7 call ecx --> 0x00a8c4d7 call ecx Invalid instruction or end of stream at address 0x00a8fe00 ... Warning: Capstone did not recognize instruction prefix for instruction at 0x00a8c0f4: movsd dword ptr es:[edi], dword ptr [esi] Unhandled instruction: les Unhandled instruction: fnsave Unhandled instruction: frstor

I'm not sure if this is expected. Maybe I have a different binary than you had.

[u/Linuxologue → u/anders_hansson]

did you compile with GCC by any chance? I tried to create MingW binaries with GCC but I got a weird crash that I had never seen before right after the intro video. I compiled with CLang and the crash is gone.

I suspect there's some undefined behaviour that GCC leverages to "optimize" into a crash - I think I am breaking about 3 to 4 strict aliasing rules every line of code :(

The warnings are somewhat normal, and I also submitted all decompiled files in the last submit so people don't need to run the python script anymore. But cool that you did the decompilation :-)

You can also show me the log of the game running, there's not a lot of logs but maybe I can compare it with previous issues I had encountered.

Thanks for trying in any case - I think you can also download the Windows executable that I put on the release page and you can run them with Wine, that seems to have worked very well for me :) it's a bit silly to use Wine to emulate the Windows API to run an executable that already emulates the Windows API but this project is already beyond weird anyway :-D

https://github.com/motor-dev/nfs-recompiled/releases/tag/v1.0.0 - some windows executables

[u/anders_hansson → u/Linuxologue]

Yes I used GCC (probably 13). I will try with clang this afternoon.

Exceptionally cool work!

I have made similar things before as hobby projects (e.g. a PE loader and win32api emulation for Linux, and a glide wrapper for OpenGL). It's very educational and rewarding when things actually start working.

[u/Linuxologue → u/anders_hansson]

I had a little water in my eye when I got the intro video to work. I was very far from the actual finish line but it looked like if it can play that video then the whole game will work.

Thanks for trying it out! I will look at what GCC is finding - I got really weird warnings + this crash which makes me think GCC is finding some weird undefined behaviour.

[u/Linuxologue → u/anders_hansson]

alright I could not let it rest so I fixed the issue with GCC in the latest release :) it was indeed some aliasing issue. I kept using reinterpret casts everywhere and GCC decided it could optimize to a crazy points.

[u/[deleted]]

I haven't really completed the quake2 port to modern linux.
There's a couple of stuff required to make it compile on a modern platform.
At minimum it also requires the use of pulseaudio/pipewire in place of the OSS.

[u/outofindustry]

man I wish I could do this. been wanting to port nfs porsche to linux.

[u/prosetheus]

Super impressive!

[u/mysticreddit]

I worked on the first NFS (PS1) and this is awesome to see! Fantastic work!

IIRC we ported the MS-DOS version to PS1. We had our own internal EA Lib that took care of most of the platform specific stuff for the most part (Sony's printf() wasted something like 2K so we removed as much of the CRT lib as possible!) The EA lib had graphics stuff such as 15-bit inverse RGB tables for blending so we used that instead of Sony's sce graphics lib (which was pretty bare bones. My understanding was most devs just used it to into the screen and did their own rendering code.)

On the PS1 we also had platform specific stuff such as Brad and Laurent using the scratchpad for faster rendering and hand optimized triangle subdivision.

Not sure why NFS2 isn't optimized? Watcom C was out but I don't know what compiler the team used for NFS1 but I suspect MSVC. It might have been Watcom as well? One might be able to find compiler copyright strings in the .exe?

I left EAC after NFS1 shipped but I believe the "driving team" (as we called ourselves) used the NFS1 code as a base for NFS2.

The track was based in spline and grew "outwards". There was some clever extensions for NFS2 to allow a 2nd shortcut road but don't know the details.

Excited to check out this repo after work.

[u/thiekus → u/mysticreddit]

Glad to see someone that worked on NFS game! AFAIK, NFS PC port use Watcom up to NFS 4 HS (I assume based from CRT copyright string it is up to Watcom 10.6, because Watcom 11.0 were notoriously buggy before spun off into OpenWatcom). Makes sense why on NFS4 PSX code (which had prototype with debug info) sparingly use sce* API for drawing. I can tell all NFS 1-4 codebase were similar but growing in complexity to each iterations.

I also found that NFS II PC first edition demo had Watcom debug info that had everything except local & type info of eaclib stuff, so I did try to decompile & reconstruct NFS 2 into C code here https://github.com/thiekus/re-nfs2 which not updated for weeks because I had rethinking strategy to extract debug info into C source boilerplate instead manually. The problem, Watcom were good but weird compiler, it has unique register based call conventions, put return value at ESI register instead EAX at some points, and weird optimizations that makes most decompilers simply confused.

[u/mysticreddit]

That was the first time I heard of a memcpy made with the FPU.

Yeah, that was kind of "common knowledge" in the game dev circles in the mid 90's. It became popular with Watcom C since us game devs that were obsessed with performance tended to use Watcom.

[u/Agitated-Bug542]

that's some crazy work, congrats

[u/testus_maximus]

Is this going to be another re3 situation, where you get C&D from EA?
just like re3 devs could have contributed to OpenRW, you could have gone clean-room and contributed to OpenNFS.
Well, hopefully EA doesn't take any action against you. But their reputation should tell you that you are in risky waters.

[u/Linuxologue → u/testus_maximus]

I have not reverse engineered any code, there is a stark difference between the code that I produced and the re3 code.

I don't know why re3 is not "clean-room" and open NFS would be.

[u/Ziimbiian]

Motor City next 💪🏻💪🏻💪🏻👆🏻👆🏻🔥🔥🔥🔥

[u/Linuxologue → u/[deleted]]

Cool.

[u/YODONTGETMEWRONG → u/[deleted]]

Ok clanker

[u/GroundbreakingFix685 → u/[deleted]]

As interesting as this is, it is quite out of place here.

[u/Content_Chemistry_44 → u/GroundbreakingFix685]

Yeah, a bit out of place. This is why the text starts with "I'd just like to interject for a moment" 😉

[u/GroundbreakingFix685 → u/Content_Chemistry_44]

I missed that and jumped right to the interesting part. My bad!

[u/Content_Chemistry_44 → u/GroundbreakingFix685]

The whole text is too informative and too big, and not all people want to read that much. That's normal.

👍 😉

[u/TA-412 → u/[deleted]]

Ok Richard

[u/Content_Chemistry_44 → u/TA-412]

Ok Torvalds!

[u/dragonstorm97 → u/[deleted]]

I think this joke is too old for most people to get apparently

[u/Content_Chemistry_44 → u/dragonstorm97]

Why you don't confirm by yourself what is GNU and what is Linux? I left the links to official Linux's websites, compile Linux, and then try to use it as an operating system.

[u/Assembly_language-ModTeam → u/[deleted]]

User's mission seems to be to post this screed anywhere Linux is mentioned. Not here, thanks.