When running your test executable, and hitting the "play" button, I get a dialog box with the following text:
extension = mkv
command line = "C:\Program Files\MPC HomeCinema (x64)\mpc-hc64.exe" ""
So, the associated executable it's getting from wxWidgets is definitely for MPC-HC.
After digging around in the wxWidgets source for a while, it looks like the root of the problem is that wxWidgets looks at hard-coded registry locations to get its Windows file associations rather than using APIs to abstract the registry access.
Windows stores its file associations in multiple registry locations, which may or may not vary by Windows version (I'm not sure, since I'm only looking at Win 7). Some locations (such as under HKEY_CLASSES_ROOT, which is what wxWidgets is looking at) are generalized for the entire system, while others are user-specific. Poking around in my registry, I see that indeed under HKEY_CLASSES_ROOT, it's got .mkv associated with MPC-HC. In a different location, however (under HKEY_CURRENT_USER), it's got .mkv associated with SMPlayer, which is the association that's used when opening an .mkv file in Explorer (and the one I see if I look at filetype associations in the Windows UI).
What's needed, it seems, is a generalized API that will get the right association without the need to hard-code registry locations. One solution is an API called AssocQueryString. For instance, I've compiled a simple test that will return the SMPlayer executable path given the .mkv extension as input:
const unsigned long bufSize = 255;
DWORD numChars = bufSize;
wchar_t regStr[ bufSize + 1 ];
HRESULT r = AssocQueryString( (ASSOCF)NULL, ASSOCSTR_EXECUTABLE, (LPCTSTR)L".mkv", (LPCTSTR)L"open",
(LPTSTR)®Str, &numChars );
When the above is finished executing, I have the following stored in regStr:
"C:\Program Files (x86)\SMPlayer\smplayer.exe"
Using AssocQueryString, you should be able to get the right executable for a given extension (much the same way you're using wxFileType::GetOpenCommand now), and then pass that to ShellExecute along with whatever command line you want. Given that different players use different command line switches, it might be a good idea to set up some kind of interface that would allow a Coollector user to specify his/her own command line for passing to the associated executable ("-f" probably isn't going to work with anything but VLC). Or at least some generalized checkbox sort of options to turn on/off different switches for different players, if you think it wouldn't be a good idea to allow direct editing of the command line.
AssocQueryString requires linking with shlwapi.lib, and the inclusion of shlwapi.h (at least that's what I had to do to get my test to compile). The documentation also states that AssocQueryString is a wrapper for the IQueryAssociations COM interface, which doesn't appear to require the extra library to be linked in (though I didn't test it). So theoretically, you could accomplish the same thing using IQueryAssociations, it'd just be more complicated.
Incidentally, I'll be quite interested to try out your upcoming versions with interface improvements. Particularly the upcoming skin system. I always like to be able to customize things.