Visual Debugger 2

Two not-so-brief moments of pain. They're fresh wounds and I'm sure the sting will dull with the passage of time, but hopefully if someone else encounters them in the future this will prove useful.

The first: In Visual Studio, I was able to open my tool window once, but if I closed it I couldn't open it a second time: all I got was "This operation could not be completed". As I dug in, I determined that IVsWindowFrame::Show() was failing. I went round and round trying to figure out how to detect that the window was closed, going as far as to copy-paste the contents of the ToolWindowBase class that ships with the VS SDK to gain access to its private variables. That wasn't a totally fruitless exercise because it made the error code more accessible, unfortunately I couldn't find the error code anywhere on the internet or in winerror.h.

COM interfaces, still a large mystery to me, made it difficult for me to try to determine the state of the object implementing IVsWindowFrame - had it actually been deleted, should I just try to create a new one or would I be leaking memory? Eventually I tried this - and here I'll pause to offer a tip that when your iteration times are slow, modifying data and instruction memory is a valuable accelerator - but creating the new window failed as well, this time with an E_POINTER error. Well at least that was something I could read more about, but it still wasn't incredibly illuminating.

It did encourage me to reevaluate the arguments I was supplying to create the window, which eventually led me to a creation flag that I recalled adding out of whimsical curiosity when I had first started exploring the tool window extension: CTW_fMultiInstance. I wasn't able to find tons of detail about why it would have caused my problem, other than a comment on a github submission:

// we don't pass __VSCREATETOOLWIN.CTW_fMultiInstance because multi instance panes are

// destroyed when closed.  We are really multi instance but we don't want to be closed.

So I removed the flag, was able to re-open my window, and moved on with my life.

The second: In Visual Studio, my tool window wasn't receiving arrow key input. I had previously determined that this was because it was sub-classed from the ATL dialog window, but didn't know much more. After some breakpoint tracing I figured out that my message was being turned into a dialog code message, WM_GETDLGCODE. I did some searching, and tried to follow the examples of returning DLGC_WANTARROWS, but it didn't work.

After trying different combinations of flags, it finally occurred to me that my code was being called by the ATL message map, not by Windows directly. The return value from the message map function is a BOOL indicating whether you handled the message, and the result (LRESULT) supplied to Windows is actually an out-arg. But because BOOL is actually just a typedef for a long integer and LRESULT is also a typedef for a long integer, the compiler provided no warning that I might not be doing what I thought. The fix:

lResult= DLGC_WANTARROWS;
return TRUE;