Programming Manual Chap. 16

Debugging and Tuning in Test Mode[]

The development environment contains a menu item Debug/Begin Test. This launches your game in "test mode." Running your game in test mode is similar to, but not exactly the same as running your game in production mode. Production mode is used when you embed your game on a webpage, or publish it in the gallery. Here are the key differences between test mode and production mode:
*Debugger and Breakpoints. In test mode, the debugger can be triggered by pressing the [Pause] key, or by setting a breakpoint. In production mode, the debugger will not appear, and breakpoints are ignored.

Local Folder. In test mode, the engine will load scripts and assets from both your server folder and your local folder, on your own hard drive. In production mode, the local folder is ignored, only the server folder is used. See the chapter on the filesystem for more information.
Performance Tuning. The performance monitoring tools can easily be invoked from inside test mode. It is possible to invoke them from production mode, but it is not as easy.
Preview Models. In test mode, you can freely use models that are marked as 'preview-only'. In production mode, you cannot use preview-only models. This allows you to try these models in your game before you purchase them. For more information, see the chapter on access control.
The Console. In test mode, the icon for the debugging console appears. Clicking on it will bring up the console, and you can type expressions. The console is not available in production mode.
Version Locking. In test mode, your game will generally use the latest versions of all assets and scripts. In production mode, your game will generally use "known-good" versions - specifically, those versions that were tested successfully. For a more precise explanation of version locking, see the chapter on version locking.

The following sections will describe all the useful testing, debugging, and performance tuning features provided by the Wild Pockets system.

The Debugger[]

Starting the Debugger[]

The debugger is a pop-up window that can appear when your program is running in test mode. When the debugger pops up, your program freezes. Your program remains frozen until the debugger is dismissed.

There are several ways to pop up the debugger. One is to simply hit the "pause-break" key on the keyboard. Another is to set a breakpoint using the breakpoints dialog in the development environment. Another is to insert a call to Debugger.breakpoint directly into your code.

You can also pop up the debugger in the development environment itself (by pressing pause-break). This is not very useful, but it does make it easy to familiarize yourself with the debugging window.

The Debugger Window[]

The debugging window contains the following major parts:
*Menu Bar. The window contains its own little menu bar at the top.

Call Stack. The call stack shows where your code's execution is stopped.
Debug Console. The debug console allows you to evaluate expressions.
Stepping Buttons. Buttons that cause the program to continue.
Execution Buttons. Buttons that print information into the debug console.
Resizing Corners. You can resize the window by dragging the corners.\

The Call Stack[]

The call stack shows where your program is stopped. The topmost line in the call stack is the currently-executing function. The next line is the function that called the currently-executing function, and so forth.

You will notice that sometimes, some of the lines in the call stack may be labeled "SYS". This is because you may at times be executing not only your own Lua code, but also Lua code that came with Wild Pockets itself. The "SYS" lines represent lua modules that came with Wild Pockets. Likewise, you may at times see a line labeled "System Kernel." This, too, is Lua code that comes with Wild Pockets.

Sometimes the function names in the call stack may be wrong. This is because Lua unfortunately does not record the function name inside the function declaration - it only records the name from the call site. Consider this code:

function foo() print("hi") end
bar = foo
bar()

Here, the function was declared as "foo," but we then copied the function into a variable "bar", and the call-site is invoking "bar." If you break, the debugger will show "bar", the name from the call-site. We hope, in the future, to modify the Lua system to fix this. Fortunately, the filename and line number are always correct. You can use this to find the real function.

The last line in the call stack is always "Debugging Scope."

Some visual debuggers will actually show you the code and put a little arrow indicating where the program is stopped. Unfortunately, we haven't implemented that yet. We hope to provide this functionality soon.

Scoping and the Debug Console[]

The debug console inside the debugger is different from the standard debug console in that it evaluates expressions inside the current scope. For example, let's say your program contains this code:

function foo()
    local a = 3
    Debugger.breakpoint()
end

When Debugger.breakpoint is executed, the debugger pops up. The program is now stopped inside function foo. You can now type this into the debug console:

print(a)

And it will print "3" - the value of the local variable inside function foo.

The there are often radio buttons next to the lines of the call stack. By clicking one of these radio buttons, you change what scope the debugging console is operating in. By clicking on the bottommost radio button (labeled 'Debugging Scope'), the debugging console becomes a regular debugging console that operates in its own global scope.

Underneath the debug console is a button labeled locals. Clicking this causes all the local variables of the currently-selected scope to be printed into the debug console.

In addition to being able to print out values, you can also modify them. For example, if you were running function foo (above), and if you were to then type this into the debug console:

a=3

This would modify the value of the local variable 'a'.

We have noticed a bug in the local variable access: local variables defined inside if-statements, while-loops, and the like are not detected by the debugger. Only locals declared at function scope are seen. We hope to fix this weakness in a future release of the debugger.

Code Execution with Caveats[]

You can type any expression at all into the debug console, and it will try to evaluate it. For example, if your game contains a function killRobots, you may very well be able to use it directly from inside the debug console to kill the robots in your game. However, there are some serious caveats.

The first thing you need to remember is that your program is in the middle of executing some subroutine. Your program may not be designed to handle arbitrary modifications in the middle of a subroutine - there is no guarantee that suddenly removing all the robots won't confuse the heck out of your program.

Second, remember that your program is frozen in time. If you execute a command like "Timer.later" inside the debug console, later will never arrive (unless you close the debugger).

Basically, most code will work in the debug console, but if you do very intrusive alterations of the game state, it may have unexpected repercussions.

Accessing Functions in another Module[]

The import statement works inside the debug console. With the help of the import statement, you can call functions inside any module.

For example, let's say your game contains a module bob/robots.lua, and this module contains a function killRobots. Simply typing this won't work:

killRobots()

Because 'killRobots' is defined only inside the module, and you're not inside that module. However, this code will work:

import("bob/robots.lua").killRobots()

The import pulls up the scope for that module, and the dot-operator pulls the necessary function out of that scope.

In the future, we will probably include a drop-down in the debugging window to make it easier to get at the scopes of your many modules.

Stepping[]

When you're done examining your program's state, you can click the "cont" button (which stands for 'continue') to simply resume execution. If you do so, the debugger will disappear and the program will continue executing until such time as somebody presses pause-break again or another breakpoint is hit.

It is also possible to single-step:

Step Into. The debugger allows the program to execute exactly one line of lua code. If that line of code happens to be a function call, the program freezes just inside the called function.
Step Over. The debugger allows the program to execute exactly one line of lua code. If that line of code happens to be a function call, the program freezes just after the function call is done.
Step Out. The debugger allows the program to execute until the currently-executing function returns. The program freezes just after the return.

Some visual debuggers will actually show you the code and put a little arrow indicating where the program is stopped. This is particularly useful when single-stepping, because you can watch the execution proceed through the code. Unfortunately, we haven't implemented that yet. We hope to provide this functionality soon.

Setting Breakpoints[]

The breakpoint window can be accessed in two places: in the development environment in the "debug" window, or in the debugger itself inside the "edit" menu. Either way, it brings up the same breakpoints dialog.

The breakpoints window requires you to type in a filename and a line number. After typing them in, click the add-button. The breakpoint is added to the list. A small check-box can be used to remove the breakpoint from the list.

Once set, the breakpoint stays set no matter what game you are testing or editing. Breakpoints remain set even if you leave the development environment and come back, or if you switch to editing a different game. The only way to get rid of a breakpoint is to go into the breakpoint editor and remove it.

You can also ask your game to break when test-mode starts, or whenever there is an error thrown by any lua function.

When breakpoints are set, Lua execution is significantly slower.

Performance Tuning[]

Identify the Cause[]

It may amuse you to learn that the most common cause of poor performance in Wild Pockets games is too many print-statements. The debug console is not designed to handle thousands of prints-per-second, and programmers often fail to realize just how many megabytes of debugging data they're pouring into that hapless little debug console.

Many inexperienced developers, seeing a performance problem, will immediately blame the renderer. But in reality, any subsystem can bog down: the renderer, the collision detection, the animation code, the lua execution. Even trivial subsystems like the debug console or the GUI renderer can bog down if you push them to do what they weren't designed to do. The renderer is no more likely to be the problem than any of the others. Blaming the wrong subsystem can waste a tremendous amount of time "optimizing" the subsystem that's already blazing-fast, while ignoring the subsystem that's running slow.

If the problem subsystem does in fact turn out to be the renderer, the cause needs to be narrowed further. Often, people will try reducing polygon counts, and will be suprised to find this has little effect. The reason for this is that modern video cards can handle an awful lot of polygons without breaking a sweat. But those same video cards may choke if you give them too many textures - combining textures may be far more effective than reducing polygons. Textures, polygons, models, meshes, state changes - too many of any of these will cause problems.

Long story short, it is absolutely essential to isolate the cause of a problem before attempting repairs.

For this, we provide the profiling and performance monitoring tools.

Accessing the Profiling Tools[]

Most of the profiling tools can be accessed by popping up the debugger (see the previous section), and then clicking on the Profiling menu.

The menu items labeled "report" will cause a report to be printed into the debugging console. The same report also appears inside the program's debug output, which can be accessed by right-clicking on the Wild Pockets icon in the Windows notification area (usually, the lower-right corner of your screen). The debug output is often more readable than the debug console, since it uses a fixed-width font.

The Memory Usage Report[]

One of the main reasons games can slow down is that they're using more memory than your computer has. To determine if this is the case, use the operating system task manager. Look for "WildPocketsEngine.exe" in the report, and see how much memory it is using. If the number seems unreasonable, it is time to look at the memory usage report. Bring up the debugger, and select Profiling/Memory Usage Report. It will print a report into the debug output.

The first line in the report is "Total of Tracked Memory." Every time the Wild Pockets engine calls "malloc", it increments a total, and every time it calls "free," it decrements the total. This is the "tracked memory."

Microsoft Windows stores memory in "heaps." There are two big heaps - the malloc heap, and the general heap. Microsoft windows provides a means to query how much data is being stored in the malloc and general heaps. The tracked memory is allocated with "malloc," so it is included in the malloc heap. The total reported by the malloc heap is usually slightly larger than the total of tracked memory. The general heap is usually pretty small.

You will notice that the two heaps (malloc + general) are usually not as large as the number you see in the task manager. This is because drivers (especially the OpenGL driver) usually do not use the malloc or general heaps. So the memory they use is not reported anywhere but in the task manager. If you see a big discrepancy between the heap sizes and the task manager size, then there's probably a lot of memory being used by the OpenGL driver - usually, to store textures.

The remaining lines of the memory usage report break down what the tracked memory is being used for.

Curl is a networking library we use. Usually very small.
Lua Heap is the memory used by lua data structures. Can be large if your program is using lots of lua data.
ODE internal is the memory used for the physics engine. Usually very small.
OPENSSL is another networking library we use. Usually very small.
Resource Type: Geom is memory used to store 3D models. Can be large.
Resource Type: Texture is memory used to store compressed textures. Can be large.
Resource Type: Other is memory used to store scripts, scenes, and other assets. Usually small.
Texture Atlas is memory used to store decompressed textures. Can be very large.
Unclassified is memory used in ways we haven't specifically tracked. Usually small.

The big memory risks are: lua data structures, textures, 3D models, and driver-related memory which is again usually textures.

Timers and Counters[]

The timers report shows how much execution time is being taken by each subsystem of Wild Pockets. The counters report is simply a list of counts of various things that might affect execution time - number of polygons, number of models, and so forth. The two are combined into the timers and counters report.

Wild Pockets is a multithreaded engine: the renderer renders while the engine simulates. Unfortunately, multithreading can make it hard to get a reliable performance measurement. When an engine is multithreading, the operating system may decide to simply stop executing one thread for a while. This delay shows up in the performance report, but it's not the fault of the module that was executing - it only reflects the operating system's decision to split its effort between the various threads. Because of this, it is best to turn off multithreading while using the timers and counters report. To do so, select Profiling/Turn Profiling mode On.

After you turn on profiling mode, you need to exit the debugger and let the program run for a few seconds in profiling (non-multithreaded) mode. Then, you can reenter the debugger and get a report.

To obtain a report, click Profiling/Print Timers and Counters. This prints both the timers report and the counters report to the debug console and the debug output. It is much better to use the debug output, since the debug output uses a fixed-width font and the columns line up properly.

The timers report breaks the engine into several major categories:

Physics. Physics includes all the calculations done by the physics engine. Collision detection is the amount of time needed to determine what is touching what. Once this is known, the analysis phase determines what collision handlers need to be called. The collision analysis and handlers category includes both the time for the analysis and the time needed to execute the lua collision handlers. The world update step is time spend adding up forces and calculating inertial movement. Normally, the physics engine should disable objects that aren't moving - this will reduce the time taken for world update (but not collision detection). Failure to disable can cause slow world-update.

Render. Render is a big category including all parts of both the GUI and 3D renderers. Many lines in this category are trivial tasks that should always take close to zero time. Those that can be more expensive include the following. Clear buffers is the amount of time it takes to clear the screen before rendering each frame. Culling is the amount of time needed to decide what to render and what not to. GUI rendering can be expensive, especially if you make the mistake of tiling a tiny image. Render objects is a misleading name, it only queues objects for rendering, so it is almost always close to zero. Send Vertex Arrays is the actual time needed to send the objects to the video card for rendering. glFinish is the most interesting thing here: this measures the amount of time spent by the renderer waiting for the GPU to finish its job, in the event that the GPU couldn't render as fast as the CPU was asking things to be rendered.

GuiManager. If your game has a substantially complex GUI with buttons, menus, and so forth, it may take significant CPU time to process the GUI every frame.

Timer. Most lua code is triggered, in one way or another, by the timer. This is true if the lua code was scheduled with Timer.later, Timer.every, or Timer.sleep. This category includes time taken to execute such lua code. If this number is large, it generally means your Lua code is taking a lot of time to execute.

The counters report shows many things that can affect the performance of the engine:

Apply State. The renderer, before rendering a model, must set the current color to the model's color, it must set the current texture to the model's texture, and so forth. But if it renders two models in a row with the same texture, it doesn't have to set the current texture prior to rendering the second model - the texture is already set. State changes are expensive - especially texture changes. Try to keep the number of texture changes under 100, and the number of other state changes under 300.

Models sorted by State / Models sorted by Depth. The engine tries hard to minimize the number of state changes. It does so by sorting the models into groups with the same texture and color (sorting by state). However, when models are transparent, they cannot be sorted by state, instead, they must be sorted back-to-front. This can lead to much more state changing. If you see lots of models sorted by depth, it means you have lots of transparency. If this is unintentional, fix it, you will get a performance boost.

Buffer Update Counter. When a model is animated, the vertex positions must be recalculated every frame. This is called a "buffer update." Animation can be expensive. This time will show up in the timers report under "Vertex Arrays: Store in Main Memory."

Model Counter. The total number of models in the scene. This has a signficant impact on renderer performance, even if the models are low-poly, because there is per-model overhead. Try to keep the number of models under 100.

Model Mesh Counter. The total number of meshes in the scene. A model with three textures contains three meshes. So the meshes can far exceed the number of models. This is the single biggest determinant of engine performance: mesh setup time is significant. Try to keep the number of meshes under 300.

Vertex Counter. People often talk about 'polygons', but in fact, they are speaking of number of unique vertices. The vertex counter tells you this. The index counter tells you how many times each vertex was used, which is less important. Try to keep the number of vertices under 50,000.

Timer Tasks. The total number of tasks scheduled through the timer module. Usually this is quite small and not a problem, but bugs in your code can cause this number to skyrocket - if you see a big number here, you may have a bug.

Version Locking[]

The Potential for Broken Links[]

When you build a website, and you link to an image on somebody else's web server, they could potentially break your website by removing the image from their web server. Likewise, when you build a Wild Pockets game, and you refer to an asset in somebody else's folder, they could potentially break your game by removing the asset. Version locking is a mechanism to make sure this doesn't occur.

Removing an asset isn't the only way to break a game. Simply replacing an asset with a more recent version could break a game, if the new version isn't compatible with the old one. You don't even have to link to somebody else's folder to put yourself at risk. Even referencing your own assets can cause problems, if you intend to continue tweaking and improving those assets.

Version locking is designed to solve problems associated with asset and script updates unexpectedly causing a game to break.

The Core Idea of Version Locking[]

When you upload a file to the file server for the first time, the file server assigns the file a version number of 0. If you upload a new version, that version has a version number of 1. If you then delete the file, the version sequence is maintained. The file is now null, but if you recreate the file, that is now version 2. The version sequence never clears and never resets, the server saves all versions, indefinitely.

The key idea behind version locking is that when your game is in the gallery, it does not necessarily use the latest version of every asset. Instead, it uses the last-known-good version. To make this possible, the scene file contains a table that lists, for every asset, the last known good version. This is called the version lock table.

Having a table of last-known-good versions makes it possible to run the game in a safe mode where it always uses known good asset versions.

How the Version Lock Table is Built[]

Open the builder, then save a new game in your server folder. This is essential, you must save the game in your folder - it cannot build a version lock table unless you have write-access to the game. Your game initially has a blank version lock table, which you can see by opening the version lock editor. Do so by clicking in the File menu of the development environment.

The version lock table is built primarily by testing your game. To do so, click Debug/Begin Test. Each time you test your game, the development environment attempts to update the entries in the version lock table, and add new entries. The update process works as follows:

Any asset that gets used during testing that is not in the table is added to the table, and the version is set to the latest version.
If the testing process uses an asset that's already in the table, the version is updated to the latest version, unless the asset is marked frozen.

Because the development/testing environment is constantly updating the entries in the table to the latest version, you pretty much always see the latest version of every asset during the testing process. The only exception is when an entry in the table has the freeze checkbox next to it. This prevents the entry from getting updated. If nothing in the table is frozen (as is commonly the case), then testing works as if there were no version locking at all, always showing the latest version of everything.

When you publish your game to the gallery, or send it to a friend, the game runs in a different mode: production mode. Production mode differs from testing mode in that it does not attempt to update the table. Instead, it uses whatever versions are already recorded in the table.

The net effect is this: when you run the game in test mode, it uses the latest version of everything. When you run the game in production mode, it uses the same versions you last used during testing. Or to put it differently, the production game will always look exactly as it did during testing - even if the assets are being altered or deleted by their authors.

Tweaking the Version Lock Table[]

Suppose that you use a friend's dragon model in your game, and it looks great. But one day, you're testing your game and you see that your friend has recolored the dragon, and you like the old colors better. The system doesn't know that this update is undesired. You must tell it, by manually tweaking the version lock table.

Go into the version lock table, and find the entry for the dragon. Dial back the version number by clicking on it and selecting a different version. This will automatically "freeze" the entry at that version. After dialing back the version, test the game again - if the dragon looks right, you're done. If not, you may need to play with the version number some more until you find the version that was good.

Disabling Version Locking[]

Sometimes you really do want your game to use the latest version no matter what. For example, suppose that you intend to deliberately publish new textures every week, just to keep your game looking fresh. In that case, version locking is going to get in your way - it's going to keep the textures locked down to how they looked the last time you tested the game.

To disable version locking for an asset, go into the version lock editor and find the asset you intend to update frequently. Click the disable-icon next to the asset. This will grey out the line in the version lock editor. The result will be that the game will use the latest version no matter what, even if the latest version is broken.

The Local Folder[]

The file server assigns a version number to everything. When an asset is stored in your local folder, it doesn't have a version number yet. If a file has no version number, it is impossible to put the correct version number into the version lock table.

However, a file which is in your local folder will probably get uploaded eventually. At that time, it will be assigned a version number. The version locking system knows this and waits for that moment. As soon as the file is uploaded (and hence, assigned a version number), the version lock system inserts the correct version number into the table.

This process can fail under an extremely obscure set of circumstances. The sync tool leaves behind records showing what was uploaded, and what version number was assigned. These records are used by the version lock system to update the version lock table. If those records are deleted or overwritten before the version lock table can be updated, the version lock table can be left not knowing what the correct version number should be. It is very hard to get into this state. If you manage it, the solution is to test the game a second time. Testing the game a second time will clear up the problem.

Of course, during normal development you tend to test your game over and over, so this is not often an issue.

Versions of Versions[]

The file server saves old versions of every file. This is true for scene files as well. That has interesting consequences.

Suppose you're in the development environment, and you click File/Save As, and type in a filename: "mygame.scene". You have now created a file on the file server: mygame.scene version 0. If you then click File/Save, you have written another file to the file server: mygame.scene version 1. If you click File/Save again, you have written yet another file to the file server, mygame.scene version 2. The file server now contains three versions of your game.

Suppose that mygame.scene version 2 contains this version lock table:

12, dragon.jpg
27, lizard.jpg

And, suppose you continue development of your game - you make a lot of improvements to dragon.jpg, yielding dragon.jpg version 18. You test your game, and sure enough, the new dragon texture looks much better. So you save your game again. You have now created mygame.scene version 3, which contains this version lock table:

18, dragon.jpg
27, lizard.jpg

But remember, the file server saves everything, so mygame.scene version 2 is still being stored on the file server, and it still has this version lock table in it: