Perhaps I can create a version that updates the status bar with some kind of indicator as the connection is being torn down. Then, based on your snapshot of the crash, I can tell were the crash occurred. Let me work on this and I'll get back to you.

And another question... I think I asked this before but I didn't get an answer. Do you still believe that the crash is related to the duration of time the connection is active, or is it possible that it is related to total throughput? In other words, is it just as likely for an inactive instance of Absolute to crash after it's been open for a long time?

Have you been able to try these same test on a machine that doesn't have FortiGate?

Brian

HA!

I bet you thought I'd forgotten about this. Not a chance!! I've got a new approach to debugging this problem. I'll try to outline it here... What we're going to try to do is a post-mortem debug of the crash using a dump file generated by good ole Dr. Watson at crash time.

First, you'll need to install the latest and greatest beta, which has enough debug info in it to make a useful dump file when it crashes...

Beta Testing

Now, we have to configure Dr. Watson to generate a dump file when the program fails. At a command prompt, enter the following command:

drwtsn32

change the 'crash dump' to c:\\AbsoluteTelnet.dmp

make sure that the following check boxes are enabled:

dump all thread contexts

dump symbol table

create crash dump file

make sure that the 'crash dump type' is set to 'full'.

Now, wait for the crash to occur. When it does, you'll see the familiar 'something bad has happened' dialog. If there is a 'debug' button on it, hit it. Dr. Watson will now generate a dump file. Let me know when you have it and we'll arrange an ftp transfer or something. It will be quite big and probably won't go through the mail.

With the dump file, I should be able to get a full trace of everything that is happening at the time of the crash.

Let me know if you have any questions.

Brian

Hi Brian,
just configured Dear Dr. Watson and downloaded the latest version. Sadly, the RC14 has expired... Is it possible to obtain a new one?

Cheers,
/msa

P.S I know you don't forget things D.S

Thanks, Mattias.

The beta just expired today. Here's the latest, which gives us another month to figure this out.

Beta Testing

I'm excited to finally figure out this method of debugging. Known as 'crash dump debugging' or 'post-mortem debugging'

To date, my debugging options were either:

1. Reproduce the problem locally and do a live debug session.

2.  

   Reproduce the problem on another machine, and do a live debug session using the remote debugger over the network

3.  

   Guess None of these make it easy to debug problems that I can't reproduce or completely understand.

With the crash-dump debugging, I should be able to more easily address issues that before went unresolved.

Thanks for helping out!

Brian

Hi Brian,
none of the three methods are of any use when you have users installing crazy stuff on their computers which may clash with AT. I have downloaded RC15 and armed Dr. Watson and I hope I will be able to provide you with a dump soon. I am currently out of the office, but will return on Thursday but AT may crash even when I am not in the office 🙂

/msa

Thanks, Mattias!

Brian

AT is running as happily as ever... :confused: Still waiting for a crash to happen - VPN has not been updated, but Windows patched with latest and greatest.

[size=1][ September 03, 2005, 02:55 AM: Message edited by: Mattias Sandstrom ][/size]

Argh!!

Oh well, I guess it's better than nothing. There's just enough debug in there to generate a useable crash dump, but I guess it's enough to make the problem go away.

I'll probably end up leaving this additional code in there. It makes the executable slightly larger, but not unbearably so. If it doesn't affect performance too bad, I'll just leave it. It'll probably help me find problems in the future anyway.

Brian

Well, we both knows what will happen if that debug-code is removed 😀 ...
If it has too much impact on performance I can live with the crashes.

Can you tell a difference?

Brian

I don't suppose you're using the NOD32 Antivirus package, are you? There is a compatibility issue identified that has ties to NOD32.

This may help us identify the issue here.

Brian

I am using Symantec Corporate 9.0 Anti-Virus.

Thanks, Mattias.

I'm guessing now, but there may be a common thread here in that there are these dlls that antivirus software packages typically install that are supposed to scan each tcp/ip socket data for viruses. These services are called 'Layered Service Proveders' or 'LSP's and sit between the Operating System and the application. Apparently, these LSPs (unless they are very well written) can cause instability at the socket layer.

Looking now at the name of the dll in your case 'fortilsp.dll' validates some of my assumptions. Notice the 'lsp' (layered service provider) in the name.

I've had reported now three different situations where there are crashes closing an SSH2 connection. Yours, the fortigate LSP. Another involving NOD32 Antivirus LSP (imon.dll). And another that has yet to be identified.

Apparently, something in the handling of the SSH2 connection socket is unexpected to the LSP layer. I'm still researching it. I actually have a crash dump from my NOD32 Antivirus user, and I'll be investigating that. Hopefully, the benefits will translate over to you as well.

Brian