Make no mistake about it. I LOVE cmake. CMake, git and post-2011 C++ reinvigorated my love for what I do and confirmed for me that whatever development I do, I want it to be in C++, extra marks for cross-platform, and can we be test-driven please? I'm doing all that on my own stuff, which has slowed to a crawl for a variety of reasons these days.
But this week, I bumped into a wrinkle. I needed to deploy an extension .DLL (it's called a .so in Linux, .dylib on macOS) with an installer, and it was working just fine... until I tried to test it on Windows 11, in which case my extension DLL just plain failed to load.
I speculated about what new thing on Windows 11 might be causing my problem. Was it some "property" -- no fiddling with icacls that I could do seemed to fix it. Overnight that night, I woke up thinking, "no, it's not that, it's something about the code underneath." What? I though? Like C++ and depending on the STL? So I implemented my way around that (which was fun), ran it as a test DLL and everything seemed to work. I put it the result into the build system and the DLL from there wouldn't load any more than my first attempt. Poring over the compile commands, between the Visual Studio-produced build properties and those generated from CMake exposed some odd differences but nothing that looked dispositive.
The link switches, though, that was another question entirely. The CMake-generated version was showing /SUBSYSTEM:CONSOLE in the link parameters; the Visual Studio created one was showing /SUBSYSTEM:WINDOWS. Google did NOT lead me to where I wanted to go easily. Instead, I found stale articles from 2008 (with no answer) and conflicting advice on StackOverflow. A co-worker pointed to an article that said to specify something in the add_executable command inside CMake. Only... I was writing a DLL (add_library) and there wasn't the same option there. I did find one article that suggested a way to do it: the resulting DLL did indeed work on Windows 11, but it struck me as clunky, so I pushed a little harder on it and produced a one-line solution.
So, just to make sure I wasn't swallowing a horse unnecessarily (the STL free code that I wrote when I thought that was the problem?), I tried the same solution on my original code -- and it worked! So I had fallen prey to two sets of red herrings, not just one.
And so. I present here a formula, in one place, for forcing any EXE or DLL to be compiled for /SUBSYSTEM:WINDOWS, because sometimes that's just what you gotta have.
For an executable, add "WIN32" to the add_executable command where you create your target, as:
add_executable(${target_} WIN32 .... )
For a Dynamic Link Library, add the line
set_target_properties( ${target_} PROPERTIES LINK_FLAGS "/SUBSYSTEM:WINDOWS")
You may also find this useful -- to prevent your DLL from reaching out and including other DLLs on systems where they aren't already installed:
set_property(TARGET ${target_} MSVC_RUNTIME_LIBRARY "MultiThreaded$<$<CONFIG:Debug>:Debug>")
Simple solution and now, hopefully, it'll be easier for me (a) to remember it or if not (b) to find it when I need it again... okay google?
