ToytownStLab: Debugging VisualWorks at the VM level

Occasionally it may be necessary to debug at the virtual machine level. If your virtual machine suddenly crashes finding out why either requires figuring it out or using a lower-level debugger. The reasons your VM might crash include Holes in the VM, bugs in external C code and VM bugs. This page exists to tell you

what the different VMs are
how to run the VM on a lower-level debugger on each platform
what commands you can use to debug at the engine level
a bit about the debug engine
why and how to create reproducible crashes

This page is *not* intended to be a guide to your platform's debugger. Use your platform's documentation for that. This page provides a minimum of incformation on driving the debugger, and hopes to provide a maximum of information on the debuggability of the VM.

What are the various engines?

As of 5i.2 each platform comes with more than one engine:

all Unix platforms come with three (5i.2 on up) or four (vw7 on up) distinct engines (1. stripped, 2. with symbols, 3. debug, and from vw7 4. assert),
for Mac OS 8/9 there are only two 5i engines (1. stripped, and 2. debug), but three vw7 engines (visual: production with debug symbols, vwpowermacast: assert, and dbgvwpowermac: debug)
and on Windows where there are 6 [more is better right ;-)].

On Windows there are two production engines, which are optimized engines with the highest compute performance available. One is "stripped", meaning it has no debug symbols for the low-level debugger. This is the visual or visual.exe engine. The other includes debug symbols, but is otherwise identical. This is the vw engine, e.g. vwlinux86 or vwnt.exe.

Why both? The stripped engine is significantly smaller and some customers think its neat to be able to fit a VM on to a floppy, so Cincom has to provide it. But in this age of cheap gigabyte drives the saving in space is insignificant. You need to use the production engine *with* debug symbols to be able to call debugging functions, or to get a meaningful stack trace. So Eliot recommends and urges you to use the unstripped production engine, as when crashes happen you'll be in a much better position to debug.

All platforms provide a "debug" engine. This is an unoptimized engine that also includes many assertion checks. This is the vwdbg engine, e.g. vwsun5dbg or vwntconsoledbg.exe. Depending on what the engine is doing, this engine is *much* slower than the production engine. The generated Smalltalk machine code is the same, but the C code is peppered with assertion checks, and is not at all optimized for full debuggability. So unless the application is doing pure Smalltalk it should be much slower. Further, the debug engine has two levels of asserts, one that are always on and another that are so time consuming they have to be turned on explicitly. To turn on these so-called dasserts use the -o11s argument. e.g.

vwlinuxdbg -o11s image.im

As of vw7 all platforms provide an "assert" engine. This is an optimized engine that also includes many assertion checks. This is the vwast engine, e.g. vwsun5ast or vwntconsoleast.exe. Depending on what the engine is doing, this engine is only moderately slower than the production engine, e.g. around 50%. The generated Smalltalk machine code is the same, but the C code is peppered with assertion checks, so unless the application is doing pure Smalltalk it should be slower, but not nearly as slow as the debug engine because the VM code is optimized. This doesn't allow full debuggab9ility, but does make it practicable to use in place of the proiduction engine for testing the engine using asserts. Like the debug engine the assert engine has two levels of asserts, one that are always on and another that are so time consuming they have to be turned on explicitly. To turn on these so-called dasserts use the -o11s argument. e.g.

vwlinuxdbg -o11s image.im

Oh yes, those other three Windows engines, vwntconsole.exe et al, are engines that stay connected to their command prompt, so they can read and write to standard input. This distinction isn't necessary on Unix because there are decent shells which allow you to start things in the background. But poor ickle bickle Windows doesn't yet grok this and requires one to duplicate the whole friggin' set of engines just to handle running with standard in/out or not. Hmph.

Because MacOS 8/9 doesn't have anything like a shell at all, there are no such engines for this platform. Furthermore, there are no command line parameters on the Mac, so setting special VM options has to be done in the debugger.

Debugging on Unixes

The main gotcher on Unix debuggers is the need to turn-off signal catching. By default a Unix debugger will halt execution when a signal is delivered. Given that e.g. mouse movements are communicated to the engine via signals this makes the system unusable unless one turns this off. One can do this with a command, typically the ignore command. But its much less tedious to put the commands in the debugger's initialization file. This file typically resides in a user's home directory, and can contain all sorts of useful stuff, like setting teh debugger's prompt to ugh>, which makes it seem so much cleverer.

- with dbx (e.g. Solaris)

the init file is called .dbxrc. Here's a sample

setenv PS1 "ugh> "
ignore io usr1 alrm 36
dalias l list
dalias n next
dalias ni nexti
dalias s step
dalias si stepi
dalias c cont
dalias b1 file assertFail.c\;stop at 54\;assign once=1
dalias sd 'call printStack(currentStack)'
dalias tf 'call printTopFrame(currentStack)'

To debug make sure VISUALWORKS is set and then launch dbx with the engine executable. Finally run the image using the run command, along with any other arguments. e.g.

$ VISUALWORKS=/usr/local/vw5i.3
dbx /usr/local/vw5i.3/bin/solaris/vwsun5
ugh> run /usr/local/vw5i.3/image/visual.im

You can get a C stack trace using the where command, e.g.

ugh> where
=>[1] updatePtrs() (optimized), at 0x5ce80 (line ~2853) in "/pps/eliot/hpsNext/build/sun5/fast/src/mman/mmInit.c"
  [2] initMmAfter() (optimized), at 0x5a1d8 (line ~548) in "/pps/eliot/hpsNext/build/sun5/fast/src/mman/mmInit.c"
  [3] startKernel() (optimized), at 0x59ef8 (line ~358) in "/pps/eliot/hpsNext/build/sun5/fast/src/mman/mmInit.c"
  [4] startSupervisor() (optimized), at 0x31f48 (line ~216) in "/pps/eliot/hpsNext/build/sun5/fast/src/ext/exInit.c"
  [5] oeMain(0x3, 0xeffffa44, 0xeffffa54, 0x0, 0x0, 0x0), at 0xb22ec
  [6] main(0x3, 0xeffffa44, 0xeffffa54, 0xf9000, 0x1, 0x0), at 0xde8fc

You can call functions in the VM using the call command, e.g.

ugh> call dumpAll(1)

- with gdb (e.g. linux)

the init file is called .gdbinit. Here's a sample

handle SIGUSR1 nostop noprint noignore
handle SIGUSR2 nostop noprint noignore
handle SIGALRM nostop noprint noignore
handle SIGPOLL nostop noprint noignore

To debug make sure VISUALWORKS is set and then launch dbx with the engine executable. Finally run the image using the run command, along with any other arguments. e.g.

bash$ VISUALWORKS=/usr/local/vw5i.3
gdb /usr/local/vw5i.3/bin/solaris/vwsun5
(gdb) run -=/usr/local/vw5i.3/image/visual.im some other arguments

Just like dbx you can get a C stack trace using the where command, and can call functions in the VM using the call command. See above for examples.

Debugging on Windows using Visual C++ Version 5 or higher

Here's one area where Windows kicks Unices' butts. MSVC++ is a great C debugger until Windows gets hoplessly confused and your system locks up with the blue screen of death telling you that "Windows is waiting to display the Close dialog box" at which point you can always reboot into Linux ;)

To install MSVC++ as your system's default debugger read the following vc-as-debugger.txt

To set-up MSVC++ to run VisualWorks you fill-out the Project->Settings... dialog. On older versions you can only do this by first invoking the debugger with the VM, e.g. evaluate ObjectMemory invokeDebugger and clock-on debug. With MSVC++ v5 and later you just need to open the VM via File->Open...

To call code in the VM fill-in the Shift-F9 evaluation dialog.

Debugging on MacOS 8/9

The MacOS 8/9 engines are done using MPW, a free development environment from Apple. This development environment makes use of the 'Power Mac Debugger' available for free at: developer.apple.com/tools/debuggers/PowerMacDebugger/. Download version 2.2d3, its the one used since 5i.0.

To debug VisualWorks on Mac, you have to have two files:

the debug engine, and
the corresponding .xcoff file.

Both files have to have identically build names, something like 'dbgvwpmac' and 'dbgvwpmac.xcoff'. After starting the debugger, 'Open...' the executable 'dbgvwpmac', which will make the debugger read the .xcoff file aswell, and opens up a window titeled like this: 'dbgvwpmac.xcoff Browser'. After this, the debugger halts with the dialog 'The process has been stopped due to a break on launch.', which can be closed pressing 'OK'. This break is not a real breakpoint, but an artifical one introduced by the debugger. Now you have the opportunity to:

set global variables in the VM before actually starting it up. For instance, open up the 'Global Variables' panel using the menu 'Window->Show Global Variables'. In the list, scroll to variable 'hpsOptions'. Set its value to 2048, which is the same as '-o11s' for commandline platforms (2048 has the 11th bit set).
set breakpoints in the .xcoff-Browser, if you have the source code of the VM.

To start up the VM from here, use the 'Control' menu or use the small right-arrow in the debugger control panel. VisualWorks will then ask you for an image to load.

Now, if everything runs fine until here (Or if you just use the debug VM without the debugger!), the debug VM will create two log files at the top level of your current volume you run VisualWorks on:

DebugLog.st80 - text is allways appended, all asserts go here,
ErrorLog.st80 - text is allways appended, all asserts go here, additional debug output is send here, e.g. from the socket primitives.

If you encounter a breakpoint, open the 'Stack Crawl' window from the debuggers 'View' menu to get the C call stack. A double-click on a stack element will open a detail window, where variables can be inspected and changed.

REMEMBER: This debugger doesn't support calling functions in a process under observation, so you cannot do all those nice things available in other debuggers.

Debugging Facilities in the VM

The VM is filled with a veritable hoard of debugging functions for

dumping the stacks of all Smalltalk processes (as of vw7.1a)
examining the Smalltalk stack
examining Smalltalk objects
examining native code

These functions sometimes get added to or modified as the VM implementors desire, so the folowing list is only guaranteed to apply to vw5i.3, but is likely to be the same thereafter. What follows is a subset of the full functions. If these don't fit the bill you'll probably need the VM source anyway, and you can get the functions by browsing the source.

miscellaneous but useful

vmExit() - called by the VM to exit; a good place to put a break-point to understand a VM abort
pdDebugLog(int) - returns the engine's notion of standard out, standard error or the debug file "debuglog.txt", depending on the argument being 0, 1 or 2 respectively.
pdCloseDebugLog() - closes "debuglog.txt" if it was opened via pdDebugLog(2)

dumping the stacks of all Smalltalk processes

For those with the VM sources these functions live in stack/checks.c. These are useful for findign out the state of teh system when it hangs or runs out of memory..

dumpAllProcesses(int brief) - prints the stacks of all Process instances in the image. Use brief to get a Smalltalk-debugger-style context list. i.e. dumpAllprocesses(1). Using brief as 0 will deluge you with information.
dumpAllProcessesToFile(int brief) - prints the stacks of all Process instances in the image to the file processes.txt.

examining the Smalltalk stack

For those with the VM sources these functions live in stack/checks.c

dumpAll(int brief) - prints the stack; if brief is zero includes the gory details o fteh contents of each stack frame
dumpAllToFile(1) - prints the stack to a file called stack.txt in the current directory
briefPrintFrames() - prints the frames in the current stack segment in brief format
printAllFramesAndContexts() - prints the entire stack
printAllFramesAndContextsFrom(frame *sfp, oopSized *esp, bool brief) - prints the entire stack from a given frame
printAllFramesAndContextsFrom(oop ctx, bool brief) - prints the entire stack from a given context. oop is the C type of Smalltalk objects

examining Smalltalk objects

For those with the VM sources these functions live in mman/checks.c

dbObjResidence(oop o) - prints where an address lies relative to the VMs various spaces
dbObjHeader(oop o) - prints the header of an object. This is similar to an object table entry, but the VM doesn't really have an object table, so "header" is preferred.
dbObjClassName(oop o) - prints the class name of an object
dbObjData(oop o) - prints an objects' contents

examining native code

For those with the VM sources these functions live in tran/tmain.c and elsewhere

nMethodForPC(void *pc) - returns the native method for a given PC, if PC is within an nMethod
dbPrintNMethod(nMethod *nmeth) - prints the header of an nmethod including its selector
printEntireNMethod(nMethod *nmeth, memIndex *pcMap, FILE *out, bool regNames) - disassembles an nmethod

The Debug Engine

...is primarily for VM debugging as opposed to application debugging. It is slow but its assertion checks tend to catch errors early. It also includes more debugging functions which are compiled in only in an engine with ASSERTs. The assertFail routine is in stack/assertFail.c, is called whenever an assert fails, and prints the expression that failed, and the file and line number where the expression failed. A common approach to debugging is to put a breakpoint in assertFail, run until a crash causes an assertFail and then start looking around.

Why and How to Create Reproducible Crashes

In Eliot's book a reproducible crash is one which can be provoked without user interaction. Anything from moving the mouse to covering a window with another can send events via the engine up into Smalltalk and causes the system to "do stuff". Such "stuff" inevitably results in objects being created. So the slightest change in user interaction will change the number and order of object allocations and can make keeping track of things from run to run a real headache.

Life is much easier if one can cause the system to crash simply by starting up an image and letting it run to destruction. One can then much more easily observe it from run to run building up a complete picture. An easy way to do this is to try to develop a test case as a doit in a workspace and then add a "saveAs:..." expression to the doit, save the image and hope it crashes on start-up without one having to intervene. e.g.

| a b c |
ObjectMemory saveAs: 'bug' thenQuit: true.
b := (1 to: 100) collect: [:i| Array new: 1].
a := (1 to: 1000000) collect: [:ign| Object new].
a size.
c := (1 to: 100) collect: [:i| Array new: 1].
1 to: b size by: 2 do: [:i| b at: i put: nil].
1 to: c size by: 2 do: [:i| c at: i put: nil].
a := nil.
ObjectMemory garbageCollect.
ScheduledControllers restore.
ObjectMemory verboseGlobalCompactingGC

ObjectMemory saveAs: 'crash' thenQuit: true.
'load.st' asFilename fileIn

Apologies for this being such a brain dump. Hopefully it'll improve over time.

Eliot Miranda Roland Wagener and, perhaps others...

Copied from the wayback machine web.archive.org/web/*/http://wiki.cs.uiuc.edu/VisualWorks/debugging+at+the+virtual+machine+level