Question

A D R I A N on Mon, 28 Aug 2017 14:50:35


Our legacy MDI desktop application uses the /dde switch in the association. When opening a file associated with it, and the application has not yet started up, Explorer pops up the following error:

There was a problem sending the command to the program.

The registry looks something like this:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document]
@="App File"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\DefaultIcon]
@="d:\\Program Files (x86)\\MyApp\\version\\app.exe,1"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell]

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\open]

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\open\command]
@="\"C:\\Program Files\\App\\app.exe\" /dde"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\open\ddeexec]
@="[open(\"%1\")]"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\print]

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\print\command]
@="C:\\Program Files\\App\\app.exe /dde"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\print\ddeexec]
@="[print(\"%1\")]"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\printto]

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\printto\command]
@="C:\\Program Files\\App\\app.exe /dde"

[HKEY_LOCAL_MACHINE\SOFTWARE\Classes\App.Document\shell\printto\ddeexec]
@="[printto(\"%1\",\"%2\",\"%3\",\"%4\")]"

Just to be clear, I just took these entries from the registry. I'm not well versed in what they do, but I can hazard a guess that they link the verbs to actions by way of the DDE interface.

Note that if the application has already started up, the document opens up fine in that instance.  This is only an issue if the application hasn't started up and must execute a new instance of the application.

So, what is happening is that the associated file is opened through Explorer by double clicking on it, and the associated application is executed. Explorer would then pop up that message and our application would do nothing. Double clicking on the file a second time would then open the document.

We've had this issue previously, but we just decided to ignore it for a few years as no one really knew what it was and we had other priorities at the time. Our workaround was to tell the user to change the /dde to "%1". Yeah, lame, but it worked well enough. One issue with doing that though, was that it would execute a new instance of the application, regardless if the application was already running or not.

Anyways, this issue is now starting to become an actual problem and needs to be fixed. One of our developers is saying that the DDE system is antiquated and we should try writing a COM component that will redirect to our application like Visual Studio does as debugging this issue could take a while. I've not verified that yet, nor researched how much effort that would be. However, either may be resource intensive, either on the debugging or the research side, so I'm trying to do some preliminary research to see what I can dig up and determine which is the better approach.

Stepping in the code, I was able to determine that it gets to a ::SetWindowPlacement() call and stepping over that will cause the error message box to pop up (if Explorer hasn't timed out first). As it is a WINAPI, I cannot step into that function to see what it is doing.

The application is written mostly in VC/VC++ using MFC/API and other libraries.

So my question is, does anyone know why this is happening and how it can be fixed?

Some additional information:

I was able to intercept all of the SendMessage()/PostMessage()/DispatchMessage() function calls non-destructively, which will log all of the messages. This was achieved by using MS Detours 3.0.

What I am seeing is that there are 4 SendMessage calls with a WM_COPYDATA message which appears to be coming out of shell32.dll. However, it doesn't appear to be the messages that are at fault though.

Putting a __debugbreak() when it detects the WM_COPYDATA message results in no error until a few steps beyond. How far depends on if I step or if I put a breakpoint and run the code to somewhere beyond where I thought I was getting the error. Using DebugBreak() seems to slow down the debugger to the point where I can't step without the error showing up.

What I can't understand is that there doesn't appear to be any rhyme or reason as to what is triggering the error message to pop up. I doesn't appear to be a timeout as the timeout appears to be long until I start stepping in the code, and sometimes no messages are being Sent/Posted by the code. So there's no WM_DDE_ACK (or any message for that matter) being sent back to the Explorer window that has initiated this. This is very frustrating.

To further complicate things, if I use the intrinsic __debugbreak() call and I have a breakpoint somewhere else in the code, it sometimes can stop at that breakpoint rather than stopping at the __debugbreak(). And sometimes, when I run the code immediately when I get control of the debugger, it will sometimes result in a second break, as if it hit another __debugbreak(). What's that all about? Inconsistent debugging is certainly making this issue even harder to track down. >:(


I don't mind someone marking a post as "Proposed as answer", but DO NOT mark it as "Answered". If I am the OP, I will decide if a post actually answers my post or not. Thank you.





Sponsored



Replies

RLWA32 on Mon, 28 Aug 2017 15:50:16


Perhaps Once you go input-idle, your application is deemed ready to receive DDE messages will be of assistance.  Raymond Chen's blog includes several other posts about DDE that you might find informative and helpful.

Darran Rowe on Mon, 28 Aug 2017 16:33:51


For the _debugbreak issue, are you doing this investigation in the release build?

The compiler has some scope to reorder operations in the release build, so if you step through the application while in the release configuration it looks like the cursor is jumping all over the place.

If this problem also manifests itself in the debug version, try using that to debug the problem instead. If you are using the debug version then wow, is something eating SEH exceptions?


This is a signature. Any samples given are not meant to have error checking or show best practices. They are meant to just illustrate a point. I may also give inefficient code or introduce some problems to discourage copy/paste coding. This is because the major point of my posts is to aid in the learning process.

A D R I A N on Mon, 28 Aug 2017 17:22:26


Hi Darran,

No, in a debug build.  Yeah, I'm just not sure how to debug this as the Explorer's error message will pop up in a nondeterministic time (not any specific function is called). :(

A D R I A N on Mon, 28 Aug 2017 17:23:53


Thanks RLWA32,

Looked at that article, and I don't see any message pump running, but I cannot be sure.  I tried to pause all of the threads other than the main one to see if that would stop the issue if another thread was pumping a message somehow, but that didn't fix the issue. :(

A D R I A N on Mon, 28 Aug 2017 17:59:02


I hooked into GetMessageA() and GetMessageW() and have determined that they haven't been called until sometime after the WM_COPYDATA messages, so it would appear that no message pump is active.

RLWA32 on Mon, 28 Aug 2017 18:30:06


According to Are DDE and WM_COPYDATA related as IPC mechanisms? there is no connection between the use of DDE and WM_COPYDATA.



Darran Rowe on Mon, 28 Aug 2017 22:14:07


Hmm, then if explorer is on a strict time out then that could explain the non deterministic display of the error message. You probably know that if you run the same section of code multiple times, then they are most likely going to vary in the amount of time it takes to run slightly.

Anyway, the initial message by DDE (WM_DDE_INITIATE) is sent to the window, not posted. This message doesn't go in via the message queue, so you should be checking to see if the target window is receiving it directly.

A D R I A N on Wed, 30 Aug 2017 21:45:39


Reading this and other places on the web re WaitForInputIdle(), I have a question.  How do I know when there is a thread that has gone idle.  What function determines that a message loop has gone idle?  If I know what the function is, I can detour it and determine what loop has gone idle and figure out how to stop it.

Thanks,

A

Darran Rowe on Thu, 31 Aug 2017 04:09:41


I'm not completely sure, but I think the message queue handling for things like this is kernel mode (done in win32k.sys), and there are no public user mode functions to query this.

There are a couple of wrinkles in this too. The most you can check is whether a thread has a message queue or not by either trying to send a thread message using PostThreadMessage, it will basically fail if the target thread doesn't have a message queue. The IsGUIThread function can only check the thread that you are calling on. But even if you know that a thread has a message queue, you can't query if it has any messages in it, or if the thread itself is calling GetMessage/PeekMessage and thus making the thread input idle. None of the other message queue functions will work on a thread other than the one you are calling on, so the only time you can query the message queue state is when you are executing code on a thread.

RLWA32 on Thu, 31 Aug 2017 08:48:27


In the event that the DDE issue remains problematic and a COM alternative is considered there is a good writeup with sample code at How do I accept files to be opened via IDropTarget instead of on the command line?

I think it will help to reduce any research and debugging effort.