Doom 3: C++ enhanced RTTI and memory debugging

Although Doom 3 was released 13 years ago, there is still of lot of interesting stuff to find in it. The main reason for that is: it was developed in the time of changes. The ID software team had just moved from C to C++ (using Visual C++ 6). Graphic cards were moving from fixed pipeline to programming pipeline, CPUs were getting SIMD extensions. All of this caused a lot of diversity in the code: crazy mix of C++/C/asm code, several renderer backends, code acceleration for MMX/SSE/AltiVec (yeah, Macs used PowerPC at that time). Not to mention tons of scripts and defs, including C++-like scripting language, and in-game GUI system.

I have to deal with Doom 3 engine in the context of The Dark Mod (TDM for short).

This article is about memory debugging configuration in Doom 3 SDK (more precisely, "Debug with inlines and memory log").

Example

To get started, here are some messages dumped to game console in the memory debugging configuration (current version of TDM):

Also, after exiting game, a file "tdm_main_leak_size.txt" appears with this data:

size: 40 KB, allocs: 3: idlib/containers/HashIndex.cpp, line: 53
size: 24 KB, allocs: 3: idlib/containers/HashIndex.cpp, line: 50
size: 22 KB, allocs: 212: game/Game_local.cpp, line: 7116
size: 18 KB, allocs: 523: idlib/Str.cpp, line: 90
size: 8 KB, allocs: 38: c:/thedarkmod/tdm/idlib/containers/List.h, line: 377
size: 6 KB, allocs: 138: game/DarkModGlobals.cpp, line: 467
size: 3 KB, allocs: 150: , line: 0
size: 3 KB, allocs: 35: renderer/Model_md5.cpp, line: 743
size: 2 KB, allocs: 110: game/darkModLAS.cpp, line: 1282
size: 1 KB, allocs: 8: game/Objectives/ObjectiveLocation.cpp, line: 72
size: 0 KB, allocs: 39: game/SndPropLoader.cpp, line: 723
size: 0 KB, allocs: 39: game/SndProp.cpp, line: 339
size: 0 KB, allocs: 13: game/SEED.cpp, line: 3449
size: 0 KB, allocs: 8: renderer/draw_arb2.cpp, line: 841
size: 0 KB, allocs: 8: c:/thedarkmod/tdm/idlib/math/Polynomial.h, line: 601
size: 0 KB, allocs: 2: framework/I18N.cpp, line: 428
size: 0 KB, allocs: 1: framework/FileSystem.cpp, line: 2756
size: 0 KB, allocs: 3: game/EscapePointManager.cpp, line: 181
1333 total memory blocks allocated
134 KB memory allocated

Memory leak detection does not feel new, but the fact that each message about uninitialized member contains the exact name of the member is quite intriguing.

Memory debugging

First let's deal with the simpler part: memory heap debugging. Looking at Heap.h, we see that ID_DEBUG_MEMORY enables a special piece of code:

#else /* ID_DEBUG_MEMORY */

void * Mem_Alloc( const int size, const char *fileName, const int lineNumber );
void * Mem_ClearedAlloc( const int size, const char *fileName, const int lineNumber );
void Mem_Free( void *ptr, const char *fileName, const int lineNumber );
char * Mem_CopyString( const char *in, const char *fileName, const int lineNumber );
void * Mem_Alloc16( const int size, const char *fileName, const int lineNumber );
void Mem_Free16( void *ptr, const char *fileName, const int lineNumber );

#ifdef ID_REDIRECT_NEWDELETE

__inline void *operator new( size_t s, int t1, int t2, char *fileName, int lineNumber ) {
    return Mem_Alloc( s, fileName, lineNumber );
}
__inline void operator delete( void *p, int t1, int t2, char *fileName, int lineNumber ) {
    Mem_Free( p, fileName, lineNumber );
}
__inline void *operator new[]( size_t s, int t1, int t2, char *fileName, int lineNumber ) {
    return Mem_Alloc( s, fileName, lineNumber );
}
__inline void operator delete[]( void *p, int t1, int t2, char *fileName, int lineNumber ) {
    Mem_Free( p, fileName, lineNumber );
}
__inline void *operator new( size_t s ) {
    return Mem_Alloc( s, "", 0 );
}
__inline void operator delete( void *p ) {
    Mem_Free( p, "", 0 );
}
__inline void *operator new[]( size_t s ) {
    return Mem_Alloc( s, "", 0 );
}
__inline void operator delete[]( void *p ) {
    Mem_Free( p, "", 0 );
}

#define ID_DEBUG_NEW new( 0, 0, __FILE__, __LINE__ )
#undef new
#define new ID_DEBUG_NEW

#endif

#define Mem_Alloc( size ) Mem_Alloc( size, __FILE__, __LINE__ )
#define Mem_ClearedAlloc( size ) Mem_ClearedAlloc( size, __FILE__, __LINE__ )
#define Mem_Free( ptr ) Mem_Free( ptr, __FILE__, __LINE__ )
#define Mem_CopyString( s ) Mem_CopyString( s, __FILE__, __LINE__ )
#define Mem_Alloc16( size ) Mem_Alloc16( size, __FILE__, __LINE__ )
#define Mem_Free16( ptr ) Mem_Free16( ptr, __FILE__, __LINE__ )

#endif /* ID_DEBUG_MEMORY */

The first part declares debug versions of ID's memory allocation functions, each of them having additional parameters filename and lineNumber. Then global operator new and operator delete are overriden to capture all allocations done with standard new/delete. Again, they have additional parameters. After that the main source of issues comes: new keyword is redefined with a macro, which allows to capture file and line of each call automatically.

The implementation of debug memory allocation looks like:

void *Mem_AllocDebugMemory( const int size, const char *fileName,
                    const int lineNumber, const bool align16 ) {
    //...

    void *p = mem_heap->Allocate( size + sizeof( debugMemory_t ) );
    Mem_UpdateAllocStats( size );

    debugMemory_t *m = (debugMemory_t *) p;
    m->fileName = fileName;
    m->lineNumber = lineNumber;
    m->frameNumber = idLib::frameNumber;
    m->size = size;
    m->next = mem_debugMemory;
    m->prev = NULL;
    if ( mem_debugMemory )
        mem_debugMemory->prev = m;
    mem_debugMemory = m;

    return ( ( (byte *) p ) + sizeof( debugMemory_t ) );
}

Each user-returned block of memory is prepended by 32 bytes of data, stored in debugMemory_t struct. It captures some information about allocation. When this memory is freed, size data member is used to update allocation stats and to provide some limited double-free detection. Since all the blocks are always linked in a doubly linked list, it is straight-forward to dump all used dynamic memory blocks at any moment, including doing so at program termination to detect memory leaks.

In order to see memory stats constantly during playing, you can type com_showMemoryUsage 1 into the game console, and you'll see something like this:

The main benefit of implementing memory profiling system yourself versus taking a ready third-party solution is that you can see exactly the data you need. In this case, per-frame allocation data is recorded and shown (which works thanks to Mem_ClearFrameStats being called each frame).

As for dumping memory blocks, there are three ways to do it:

  1. Exit game and look into file [tdm_game]_leak_location.txt (contents were shown above): this way is used to detect memory leaks.
  2. Write memoryDump into game console and look into created file memoryDump.txt. This file contains all blocks alive, and can be used to perform additional offline analysis (sample).
  3. Write memoryDumpCompressed -s into game console and look into file memoryDump.txt. Unlike the previous command, this one shows per-allocation-site statistics sorted according to parameter (-s = by size), which is rather compact (300 lines):
memoryDump.txt contents (click to display)

size: 256912 KB, allocs: 354: c:/thedarkmod/tdm/idlib/Allocators.h, line: 613
size: 133217 KB, allocs: 4206: renderer/tr_main.cpp, line: 297
size: 27157 KB, allocs: 34927: c:/thedarkmod/tdm/idlib/containers/List.h, line: 377
size: 10240 KB, allocs: 10: c:/thedarkmod/tdm/idlib/Allocators.h, line: 378
size: 5494 KB, allocs: 11682: framework/DeclManager.cpp, line: 1907
size: 4466 KB, allocs: 2358: game/gamesys/Class.cpp, line: 468
size: 2813 KB, allocs: 41174: cm/CollisionModel_load.cpp, line: 660
size: 2693 KB, allocs: 103: c:/thedarkmod/tdm/idlib/Allocators.h, line: 80
size: 2684 KB, allocs: 337: cm/CollisionModel_load.cpp, line: 2913
size: 2449 KB, allocs: 46008: idlib/Str.cpp, line: 90
size: 2086 KB, allocs: 309: renderer/Model_md5.cpp, line: 195
size: 2048 KB, allocs: 2: renderer/CinematicFFMpeg.cpp, line: 607
size: 1983 KB, allocs: 769: cm/CollisionModel_files.cpp, line: 337
size: 1950 KB, allocs: 960: game/BrittleFracture.cpp, line: 331
size: 1872 KB, allocs: 1546: renderer/Image_init.cpp, line: 1361
size: 1762 KB, allocs: 764: cm/CollisionModel_files.cpp, line: 390
size: 1679 KB, allocs: 1819: cm/CollisionModel_load.cpp, line: 596
size: 1677 KB, allocs: 6173: idlib/containers/HashIndex.cpp, line: 50
size: 1487 KB, allocs: 528: game/physics/Clip.cpp, line: 102
size: 1131 KB, allocs: 13795: idlib/MapFile.cpp, line: 316
size: 1121 KB, allocs: 12488: framework/DeclManager.cpp, line: 1701
size: 1043 KB, allocs: 309: renderer/Model_md5.cpp, line: 196
size: 1024 KB, allocs: 1: renderer/tr_main.cpp, line: 245
size: 902 KB, allocs: 1200: cm/CollisionModel_load.cpp, line: 566
size: 814 KB, allocs: 337: cm/CollisionModel_load.cpp, line: 2905
size: 810 KB, allocs: 1572: renderer/RenderWorld.cpp, line: 254
size: 759 KB, allocs: 3471: c:/thedarkmod/tdmframework/DeclManager.h, line: 228
size: 691 KB, allocs: 680: cm/CollisionModel_files.cpp, line: 316
size: 683 KB, allocs: 5212: idlib/containers/HashIndex.cpp, line: 53
size: 669 KB, allocs: 27325: , line: 0
size: 651 KB, allocs: 704: ui/Window.cpp, line: 2188
size: 625 KB, allocs: 1: game/Game_local.cpp, line: 554
size: 620 KB, allocs: 980: idlib/containers/HashIndex.cpp, line: 101
size: 594 KB, allocs: 15207: c:/thedarkmod/tdm/idlib/containers/StrPool.h, line: 106
size: 590 KB, allocs: 2200: c:/thedarkmod/tdm/idlib/containers/HashIndex.h, line: 159
size: 512 KB, allocs: 1: renderer/CinematicID.cpp, line: 71
size: 458 KB, allocs: 3912: game/anim/Anim_Blend.cpp, line: 3073
size: 401 KB, allocs: 123: game/gamesys/Class.cpp, line: 161
size: 367 KB, allocs: 7238: tools/compilers/aas/AASFile.cpp, line: 913
size: 354 KB, allocs: 3776: sound/snd_cache.cpp, line: 98
size: 309 KB, allocs: 769: cm/CollisionModel_files.cpp, line: 433
size: 285 KB, allocs: 4876: idlib/MapFile.cpp, line: 540
size: 283 KB, allocs: 1339: c:/thedarkmod/tdm/idlib/containers/List.h, line: 536
size: 247 KB, allocs: 2181: c:/thedarkmod/tdm/idlib/containers/HashIndex.h, line: 166
size: 215 KB, allocs: 4598: ui/GuiScript.cpp, line: 375
size: 205 KB, allocs: 6590: game/OverlaySys.cpp, line: 245
size: 205 KB, allocs: 6560: game/script/Script_Program.cpp, line: 1256
size: 199 KB, allocs: 379: ui/Window.cpp, line: 2196
size: 178 KB, allocs: 251: game/AF.cpp, line: 676
size: 175 KB, allocs: 1000: idlib/MapFile.cpp, line: 125
size: 172 KB, allocs: 1107: cm/CollisionModel_load.cpp, line: 530
size: 164 KB, allocs: 860: framework/CVarSystem.cpp, line: 653
size: 161 KB, allocs: 2292: idlib/MapFile.cpp, line: 379
size: 159 KB, allocs: 1: game/physics/Clip.cpp, line: 714
size: 159 KB, allocs: 3143: ui/RegExp.cpp, line: 246
size: 159 KB, allocs: 1167: game/Entity.cpp, line: 5878
size: 148 KB, allocs: 73: sound/snd_world.cpp, line: 200
size: 142 KB, allocs: 111: renderer/RenderWorld.cpp, line: 408
size: 136 KB, allocs: 4368: ui/Window.cpp, line: 1625
size: 131 KB, allocs: 960: game/BrittleFracture.cpp, line: 1171
size: 128 KB, allocs: 1: renderer/CinematicID.cpp, line: 70
size: 121 KB, allocs: 3: game/ai/AAS_routing.cpp, line: 241
size: 114 KB, allocs: 4499: c:/thedarkmod/tdm/idlib/math/Vector.h, line: 1727
size: 107 KB, allocs: 3444: game/anim/Anim_Blend.cpp, line: 88
size: 105 KB, allocs: 406: cm/CollisionModel_load.cpp, line: 625
size: 91 KB, allocs: 835: renderer/ModelManager.cpp, line: 336
size: 87 KB, allocs: 40: game/anim/Anim_Blend.cpp, line: 3289
size: 83 KB, allocs: 1190: idlib/math/Ode.cpp, line: 32
size: 77 KB, allocs: 2198: game/script/Script_Program.cpp, line: 1223
size: 75 KB, allocs: 807: game/script/Script_Program.cpp, line: 1138
size: 74 KB, allocs: 2260: c:/thedarkmod/tdm/idlib/containers/List.h, line: 419
size: 69 KB, allocs: 1019: c:/thedarkmod/tdm/idlib/math/Matrix.h, line: 2308
size: 68 KB, allocs: 2197: game/Entity.cpp, line: 1006
size: 65 KB, allocs: 580: renderer/Material.cpp, line: 1540
size: 65 KB, allocs: 1392: ui/Window.cpp, line: 1862
size: 64 KB, allocs: 1: renderer/CinematicID.cpp, line: 68
size: 60 KB, allocs: 3: game/ai/AAS_routing.cpp, line: 128
size: 58 KB, allocs: 426: game/StimResponse/StimResponseCollection.cpp, line: 105
size: 57 KB, allocs: 733: idlib/geometry/Winding.cpp, line: 36
size: 53 KB, allocs: 55: ui/Window.cpp, line: 2253
size: 52 KB, allocs: 450: game/anim/Anim_Blend.cpp, line: 3359
size: 52 KB, allocs: 166: game/AF.cpp, line: 793
size: 52 KB, allocs: 215: game/StimResponse/StimResponseCollection.cpp, line: 91
size: 51 KB, allocs: 397: game/anim/Anim.cpp, line: 976
size: 49 KB, allocs: 38: ui/Window.cpp, line: 2220
size: 48 KB, allocs: 512: game/script/Script_Program.cpp, line: 1124
size: 45 KB, allocs: 162: game/physics/Physics_AF.cpp, line: 1061
size: 44 KB, allocs: 369: game/ai/AAS_routing.cpp, line: 52
size: 38 KB, allocs: 64: framework/DeclAF.cpp, line: 786
size: 37 KB, allocs: 599: framework/DeclManager.cpp, line: 998
size: 35 KB, allocs: 1148: ui/GuiScript.cpp, line: 520
size: 34 KB, allocs: 251: game/AF.cpp, line: 673
size: 32 KB, allocs: 299: renderer/ModelManager.cpp, line: 282
size: 32 KB, allocs: 1: cm/CollisionModel_load.cpp, line: 3385
size: 32 KB, allocs: 1: renderer/CinematicID.cpp, line: 69
size: 31 KB, allocs: 230: game/Moveable.cpp, line: 193
size: 28 KB, allocs: 1961: c:/thedarkmod/tdm/ui/Winvar.h, line: 44
size: 23 KB, allocs: 82: framework/DeclParticle.cpp, line: 213
size: 23 KB, allocs: 1190: game/physics/Physics_RigidBody.cpp, line: 1164
size: 22 KB, allocs: 369: game/ai/AAS_routing.cpp, line: 50
size: 22 KB, allocs: 212: game/Game_local.cpp, line: 7116
size: 20 KB, allocs: 39: renderer/RenderWorld_load.cpp, line: 623
size: 19 KB, allocs: 60: game/AF.cpp, line: 826
size: 18 KB, allocs: 30: framework/DeclAF.cpp, line: 1079
size: 17 KB, allocs: 25: game/AFEntity.cpp, line: 1524
size: 16 KB, allocs: 60: game/physics/Physics_AF.cpp, line: 1697
size: 16 KB, allocs: 26: framework/DeclAF.cpp, line: 1157
size: 15 KB, allocs: 397: c:/thedarkmod/tdm/idlib/containers/HashTable.h, line: 191
size: 14 KB, allocs: 208: game/script/Script_Program.cpp, line: 2076
size: 13 KB, allocs: 564: game/ai/EAS/EAS.cpp, line: 614
size: 13 KB, allocs: 564: game/ai/EAS/EAS.cpp, line: 616
size: 12 KB, allocs: 263: game/ai/AAS_routing.cpp, line: 894
size: 12 KB, allocs: 3: framework/FileSystem.cpp, line: 1253
size: 12 KB, allocs: 3: game/ai/AAS_routing.cpp, line: 238
size: 12 KB, allocs: 13: ui/UserInterface.cpp, line: 270
size: 11 KB, allocs: 86: game/Entity.cpp, line: 5868
size: 11 KB, allocs: 60: framework/CVarSystem.cpp, line: 578
size: 11 KB, allocs: 10: ui/Window.cpp, line: 2231
size: 11 KB, allocs: 1: framework/KeyInput.cpp, line: 709
size: 10 KB, allocs: 185: game/anim/Anim_Blend.cpp, line: 4637
size: 10 KB, allocs: 684: ui/Window.cpp, line: 1694
size: 10 KB, allocs: 649: renderer/tr_polytope.cpp, line: 91
size: 9 KB, allocs: 133: game/StimResponse/Response.cpp, line: 212
size: 9 KB, allocs: 615: ui/Window.cpp, line: 1628
size: 9 KB, allocs: 68: game/Mover.cpp, line: 380
size: 9 KB, allocs: 1158: game/PVSToAASMapping.cpp, line: 197
size: 8 KB, allocs: 19: game/ai/AI.cpp, line: 1652
size: 8 KB, allocs: 271: framework/CmdSystem.cpp, line: 371
size: 8 KB, allocs: 53: renderer/ModelManager.cpp, line: 288
size: 8 KB, allocs: 3: game/ai/AAS_routing.cpp, line: 218
size: 8 KB, allocs: 173: ui/GuiScript.cpp, line: 450
size: 7 KB, allocs: 1: game/Game_local.cpp, line: 585
size: 6 KB, allocs: 12: game/ModelGenerator.cpp, line: 296
size: 6 KB, allocs: 138: game/DarkModGlobals.cpp, line: 467
size: 6 KB, allocs: 271: framework/CmdSystem.cpp, line: 366
size: 6 KB, allocs: 3: game/ai/AAS_routing.cpp, line: 247
size: 5 KB, allocs: 256: framework/DeclManager.cpp, line: 408
size: 4 KB, allocs: 255: framework/DeclManager.cpp, line: 418
size: 4 KB, allocs: 106: game/ai/AAS_routing.cpp, line: 1033
size: 4 KB, allocs: 16: c:/thedarkmod/tdm/idlib/containers/List.h, line: 60
size: 4 KB, allocs: 4: ui/Window.cpp, line: 2209
size: 4 KB, allocs: 4: ui/EditWindow.cpp, line: 99
size: 4 KB, allocs: 294: c:/thedarkmod/tdm/ui/Window.h, line: 110
size: 4 KB, allocs: 141: game/anim/Anim_Blend.cpp, line: 343
size: 4 KB, allocs: 72: game/anim/Anim_Blend.cpp, line: 4597
size: 4 KB, allocs: 2: sound/snd_world.cpp, line: 81
size: 3 KB, allocs: 21: idlib/math/Lcp.cpp, line: 1603
size: 3 KB, allocs: 11: game/Player.cpp, line: 1325
size: 3 KB, allocs: 14: game/Missions/MissionDB.cpp, line: 74
size: 3 KB, allocs: 238: ui/Window.cpp, line: 1632
size: 3 KB, allocs: 34: renderer/Model_md5.cpp, line: 743
size: 3 KB, allocs: 105: ui/Window.cpp, line: 2305
size: 3 KB, allocs: 30: game/SndProp.cpp, line: 332
size: 3 KB, allocs: 294: ui/Window.cpp, line: 2335
size: 3 KB, allocs: 25: game/AFEntity.cpp, line: 1481
size: 3 KB, allocs: 271: framework/CmdSystem.cpp, line: 367
size: 3 KB, allocs: 23: renderer/ModelManager.cpp, line: 294
size: 3 KB, allocs: 3: framework/FileSystem.cpp, line: 1252
size: 2 KB, allocs: 13: ui/UserInterface.cpp, line: 157
size: 2 KB, allocs: 20: game/Entity.cpp, line: 5939
size: 2 KB, allocs: 20: game/Actor.cpp, line: 2427
size: 2 KB, allocs: 76: game/ai/EAS/EAS.cpp, line: 104
size: 2 KB, allocs: 13: renderer/Cinematic.cpp, line: 57
size: 2 KB, allocs: 19: game/ai/AI.cpp, line: 1986
size: 2 KB, allocs: 19: game/AFEntity.cpp, line: 702
size: 2 KB, allocs: 110: game/darkModLAS.cpp, line: 1282
size: 2 KB, allocs: 4: framework/DeclAF.cpp, line: 961
size: 2 KB, allocs: 19: game/ai/AI.cpp, line: 1655
size: 2 KB, allocs: 3: game/ai/AAS_routing.cpp, line: 244
size: 2 KB, allocs: 19: renderer/Model_prt.cpp, line: 95
size: 1 KB, allocs: 30: game/SndProp.cpp, line: 384
size: 1 KB, allocs: 14: game/Mover.cpp, line: 3391
size: 1 KB, allocs: 8: game/physics/Physics_AF.cpp, line: 1872
size: 1 KB, allocs: 1: ui/Window.cpp, line: 2286
size: 1 KB, allocs: 105: c:/thedarkmod/tdm/ui/Window.h, line: 129
size: 1 KB, allocs: 7: game/Inventory/Inventory.cpp, line: 656
size: 1 KB, allocs: 1: game/SndProp.cpp, line: 366
size: 1 KB, allocs: 20: game/StimResponse/Response.cpp, line: 189
size: 1 KB, allocs: 3: tools/compilers/aas/AASFileManager.cpp, line: 50
size: 1 KB, allocs: 1: ui/Window.cpp, line: 2264
size: 1 KB, allocs: 4: game/physics/Physics_AF.cpp, line: 1084
size: 1 KB, allocs: 81: game/darkModLAS.cpp, line: 1290
size: 1 KB, allocs: 16: game/ai/States/IdleState.cpp, line: 174
size: 1 KB, allocs: 1: game/SndProp.cpp, line: 314
size: 1 KB, allocs: 5: game/physics/Physics_AF.cpp, line: 3102
size: 1 KB, allocs: 1: cm/CollisionModel_load.cpp, line: 745
size: 1 KB, allocs: 1: ui/ListWindow.cpp, line: 46
size: 1 KB, allocs: 16: game/ai/Tasks/IdleAnimationTask.cpp, line: 437
size: 1 KB, allocs: 4: game/AF.cpp, line: 748
size: 1 KB, allocs: 8: game/Objectives/ObjectiveLocation.cpp, line: 72
size: 1 KB, allocs: 14: game/Missions/MissionDB.cpp, line: 71
size: 1 KB, allocs: 33: game/anim/Anim_Blend.cpp, line: 323
size: 1 KB, allocs: 1: c:/thedarkmod/tdm/idlib/containers/HashTable.h, line: 86
size: 0 KB, allocs: 1: game/SndProp.cpp, line: 319
size: 0 KB, allocs: 39: game/SndPropLoader.cpp, line: 723
size: 0 KB, allocs: 39: game/SndProp.cpp, line: 339
size: 0 KB, allocs: 43: ui/Window.cpp, line: 2433
size: 0 KB, allocs: 26: game/ai/Conversation/Conversation.cpp, line: 475
size: 0 KB, allocs: 6: game/ai/AAS.cpp, line: 30
size: 0 KB, allocs: 1: cm/CollisionModel_load.cpp, line: 742
size: 0 KB, allocs: 24: game/anim/Anim_Blend.cpp, line: 702
size: 0 KB, allocs: 19: game/ai/AI.cpp, line: 1656
size: 0 KB, allocs: 19: game/ai/AI.cpp, line: 1657
size: 0 KB, allocs: 19: game/ai/AI.cpp, line: 1658
size: 0 KB, allocs: 19: game/ai/AI.cpp, line: 1659
size: 0 KB, allocs: 23: game/anim/Anim_Blend.cpp, line: 826
size: 0 KB, allocs: 23: game/anim/Anim_Blend.cpp, line: 724
size: 0 KB, allocs: 3: framework/minizip/unzip.cpp, line: 792
size: 0 KB, allocs: 42: renderer/RenderWorld_load.cpp, line: 295
size: 0 KB, allocs: 42: idlib/geometry/Winding.cpp, line: 469
size: 0 KB, allocs: 5: framework/CVarSystem.cpp, line: 151
size: 0 KB, allocs: 2: sound/snd_system.cpp, line: 960
size: 0 KB, allocs: 20: game/anim/Anim_Blend.cpp, line: 686
size: 0 KB, allocs: 7: game/ai/Conversation/ConversationSystem.cpp, line: 242
size: 0 KB, allocs: 14: game/SEED.cpp, line: 972
size: 0 KB, allocs: 19: game/anim/Anim_Blend.cpp, line: 811
size: 0 KB, allocs: 13: game/SEED.cpp, line: 3449
size: 0 KB, allocs: 8: framework/DeclManager.cpp, line: 975
size: 0 KB, allocs: 13: framework/DeclManager.cpp, line: 943
size: 0 KB, allocs: 16: game/anim/Anim_Blend.cpp, line: 359
size: 0 KB, allocs: 10: ui/Window.cpp, line: 2107
size: 0 KB, allocs: 29: game/darkModLAS.cpp, line: 1299
size: 0 KB, allocs: 14: game/anim/Anim_Blend.cpp, line: 834
size: 0 KB, allocs: 14: game/anim/Anim_Blend.cpp, line: 849
size: 0 KB, allocs: 7: game/Inventory/Inventory.cpp, line: 393
size: 0 KB, allocs: 2: renderer/RenderSystem.cpp, line: 1067
size: 0 KB, allocs: 6: game/ai/AAS.cpp, line: 48
size: 0 KB, allocs: 21: game/physics/Physics_AF.cpp, line: 7366
size: 0 KB, allocs: 16: game/ai/States/IdleState.cpp, line: 383
size: 0 KB, allocs: 1: game/Pvs.cpp, line: 798
size: 0 KB, allocs: 1: game/SndProp.cpp, line: 363
size: 0 KB, allocs: 1: cm/CollisionModel_load.cpp, line: 682
size: 0 KB, allocs: 9: game/anim/Anim_Blend.cpp, line: 605
size: 0 KB, allocs: 2: game/IK.cpp, line: 539
size: 0 KB, allocs: 8: game/anim/Anim_Blend.cpp, line: 504
size: 0 KB, allocs: 3: game/ai/States/IdleSleepState.cpp, line: 106
size: 0 KB, allocs: 1: game/Player.cpp, line: 1887
size: 0 KB, allocs: 7: game/anim/Anim_Blend.cpp, line: 437
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 625
size: 0 KB, allocs: 8: game/ai/MovementSubsystem.cpp, line: 423
size: 0 KB, allocs: 16: game/ai/Tasks/RandomHeadturnTask.cpp, line: 138
size: 0 KB, allocs: 8: renderer/draw_arb2.cpp, line: 841
size: 0 KB, allocs: 2: framework/FileSystem.cpp, line: 2889
size: 0 KB, allocs: 2: framework/FileSystem.cpp, line: 2756
size: 0 KB, allocs: 8: c:/thedarkmod/tdm/idlib/math/Polynomial.h, line: 601
size: 0 KB, allocs: 1: game/darkModLAS.cpp, line: 1209
size: 0 KB, allocs: 1: game/Pvs.cpp, line: 793
size: 0 KB, allocs: 1: game/PVSToAASMapping.cpp, line: 115
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 6588
size: 0 KB, allocs: 3: game/ai/MovementSubsystem.cpp, line: 330
size: 0 KB, allocs: 1: game/Entity.cpp, line: 5823
size: 0 KB, allocs: 1: game/AFEntity.cpp, line: 1262
size: 0 KB, allocs: 1: game/Player.cpp, line: 5738
size: 0 KB, allocs: 1: game/Player.cpp, line: 8806
size: 0 KB, allocs: 2: framework/FileSystem.cpp, line: 2053
size: 0 KB, allocs: 16: game/Pvs.cpp, line: 809
size: 0 KB, allocs: 4: game/anim/Anim_Blend.cpp, line: 566
size: 0 KB, allocs: 8: game/ai/Memory.cpp, line: 446
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 607
size: 0 KB, allocs: 1: game/ai/Tasks/HandleDoorTask.cpp, line: 2963
size: 0 KB, allocs: 1: renderer/ModelManager.cpp, line: 206
size: 0 KB, allocs: 1: renderer/ModelManager.cpp, line: 214
size: 0 KB, allocs: 1: renderer/ModelManager.cpp, line: 220
size: 0 KB, allocs: 1: renderer/ModelManager.cpp, line: 309
size: 0 KB, allocs: 3: game/Inventory/Inventory.cpp, line: 904
size: 0 KB, allocs: 2: framework/I18N.cpp, line: 428
size: 0 KB, allocs: 3: game/anim/Anim_Blend.cpp, line: 754
size: 0 KB, allocs: 3: game/anim/Anim_Blend.cpp, line: 734
size: 0 KB, allocs: 1: ui/UserInterface.cpp, line: 210
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 1546
size: 0 KB, allocs: 1: game/Objectives/MissionData.cpp, line: 1697
size: 0 KB, allocs: 2: renderer/CinematicFFMpeg.cpp, line: 606
size: 0 KB, allocs: 2: game/anim/Anim_Blend.cpp, line: 744
size: 0 KB, allocs: 1: game/LightGem.cpp, line: 101
size: 0 KB, allocs: 3: game/ai/States/IdleSleepState.cpp, line: 143
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 623
size: 0 KB, allocs: 1: game/Entity.cpp, line: 10650
size: 0 KB, allocs: 2: game/ai/MovementSubsystem.cpp, line: 348
size: 0 KB, allocs: 1: renderer/RenderSystem_init.cpp, line: 2464
size: 0 KB, allocs: 1: renderer/RenderSystem_init.cpp, line: 2467
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 591
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 611
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 615
size: 0 KB, allocs: 3: game/EscapePointManager.cpp, line: 181
size: 0 KB, allocs: 1: game/Pvs.cpp, line: 792
size: 0 KB, allocs: 3: framework/FileSystem.cpp, line: 2079
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 589
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 619
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 1612
size: 0 KB, allocs: 1: game/anim/Anim_Blend.cpp, line: 776
size: 0 KB, allocs: 1: game/anim/Anim_Blend.cpp, line: 786
size: 0 KB, allocs: 1: game/anim/Anim_Blend.cpp, line: 525
size: 0 KB, allocs: 1: renderer/tr_main.cpp, line: 242
size: 0 KB, allocs: 2: framework/FileSystem.cpp, line: 2052
size: 0 KB, allocs: 1: sound/snd_system.cpp, line: 444
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 590
size: 0 KB, allocs: 1: ui/Window.cpp, line: 2090
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 8065
size: 0 KB, allocs: 1: game/Missions/MissionManager.cpp, line: 40
size: 0 KB, allocs: 1: renderer/CinematicFFMpeg.cpp, line: 249
size: 0 KB, allocs: 1: idlib/math/Simd.cpp, line: 43
size: 0 KB, allocs: 1: idlib/math/Simd.cpp, line: 161
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 588
size: 0 KB, allocs: 1: game/Game_local.cpp, line: 631
338574 total memory blocks allocated
486355 KB memory allocated

Uninitialized members

Leaving class members of primitive type uninitialized can cause annoying and non-reproducible bugs. That's why Visual C++ fills allocated memory with magic values like 0xCD or 0xCC in debug build, to increase the chances of program crashing due to such an error. For some reason, ID devs decided to implement their own version of this system for game objects.

In the game module of Doom 3, most of the classes inherit from idClass base, which allows to create object instance by type name and supports events for scripting system. All objects in this hierarchy are manually filled with 0xCD because idClass overloads operator new and operator delete:

void * idClass::operator new( size_t s ) {
    int *p;

    s += sizeof( int );
    p = (int *)Mem_Alloc( s );
    *p = s;
    memused += s;
    numobjects++;

#ifdef ID_DEBUG_UNINITIALIZED_MEMORY
    unsigned int *ptr = (unsigned int *)p;
    int size = s;
    assert( ( size & 3 ) == 0 );
    size >>= 2;
    for ( int i = 1; i < size; i++ ) {
        ptr[i] = 0xcdcdcdcd;
    }
#endif

    return p + 1;
}

Now the remaining question is: where are the messages about uninitialized class members coming from?

Quick search by the text of message gives the following bits of code:

#define CLASS_DECLARATION( nameofsuperclass, nameofclass ) \
    //...some more ...                                     \
    idClass *nameofclass::CreateInstance( void ) {         \
        try {                                              \
            nameofclass *ptr = new nameofclass;            \
            ptr->FindUninitializedMemory();                \
            return ptr;                                    \
        }                                                  \
        catch( idAllocError & ) {                          \
            return NULL;                                   \
        }                                                  \
    }                                                      \

void idClass::FindUninitializedMemory( void ) {
#ifdef ID_DEBUG_UNINITIALIZED_MEMORY
    unsigned int *ptr = ( ( unsigned int * )this ) - 1;
    int size = *ptr;
    assert( ( size & 3 ) == 0 );
    size >>= 2;
    for ( int i = 0; i < size; i++ ) {
        if ( ptr[i] == 0xcdcdcdcd ) {
            const char *varName = GetTypeVariableName( GetClassname(), i << 2 );
            gameLocal.Warning(
                "type '%s' has uninitialized variable %s (offset %d)",
                GetClassname(), varName, i << 2
            );
        }
    }
#endif
}

The method CreateInstance (wrapped into macro CLASS_DECLARATION) allows creating game objects by type, and it is always called when a game object is created. As you see, it calls FindUninitializedMemory method after creation, which checks which words still have values 0xCDCDCDCD, and reports them. The only thing remaining is the GetTypeVariableName function, and this is where the truly dark magic is.

TypeInfo

In order to obtain name of the class member from its offset, information about all the member offsets must be embedded into the code. Currently, C++ compilers do not do that (although there are proposals for something like that). Standard C++ RTTI only stores class name and inheritance hierarchy for polymorphic types, it does not store any info about members. There are several approaches for implementing reflection in C++. Preprocessor can generate the necessary information, given that you use macros for declaring members (example), or specialized parser can generate the necessary information (e.g. Qt MOC). Doom 3 uses the latter way: homebrew parser does that.

Recall that Doom 3 uses its own C++-like scripting language, which is parsed, compiled and interpreted by custom code (see review). The lexer and parser are reused for several different tasks. The parser also performs all the job of C preprocessor: it #includes files, checks #if-s and #ifdef-s, manages #define-s and performs macro substitution. This same parser is reused in the specialized TypeInfo compiler, whose only purpose is to generate a header with all the desired type information. It parses a single cpp file of the game module, which is supposed to include headers with all the declarations. So: ID's script parser is good enough to parse Doom 3 game sources and extract data from them! Well, it has its own issues (no pragma once support, some C++11 syntax not supported), but this is a remarkable achievement nonetheless.

After having parsed everything, TypeInfo compiler generates a header GameTypeInfo.h in game/gamesys/. This header contains the following information (you can download resulting GameTypeInfo.h for TDM):

  1. constants: type, name, value (including enum values and static const members);
  2. enums: list of all enums, list of name/value pairs for all enums;
  3. class members: type, name, offset, size;
  4. classes: full list, name, base class name, pointer to members list;

Here is an example of information for members of idStr class:

static classVariableInfo_t idStr_typeInfo[] = {
    { "int", "len", (int)(&((idStr *)0)->len), sizeof( ((idStr *)0)->len ) },
    { "char *", "data", (int)(&((idStr *)0)->data), sizeof( ((idStr *)0)->data ) },
    { "int", "alloced", (int)(&((idStr *)0)->alloced), sizeof( ((idStr *)0)->alloced ) },
    { "char[20]", "baseBuffer", (int)(&((idStr *)0)->baseBuffer), sizeof( ((idStr *)0)->baseBuffer ) },
    { NULL, 0 }
};

Clearly, it contains enough information to make it possible to get member name from offset. Note that sizes and offsets are written as C++ expressions evaluated to proper values, instead of plain integer constants. This allows to avoid messing with struct alignment rules of Visual C++ compiler. However, it also requires to do an ugly hack (called "Public Morozov" in Russia) to make sure that private and protected members are accessible in TypeInfo.cpp:

// This is real evil but allows the code
// to inspect arbitrary class variables.
#define private     public
#define protected   public

Here is the full diagram for building TypeInfo in memory debugging configuration (for the case of TDM):

Saving game state

Is it worth to implement the whole TypeInfo thing just for printing the name of an uninitialized member? I think not. But this is not the only thing which relies on TypeInfo. Another thing made possible by TypeInfo is printing the whole game state in a readable text format. Normally, all the savegame data is stored in binary format, because it is much faster to do so and the resulting file is smaller (although I added compression on top of it in TDM). But text representation may be very useful for debugging issues (e.g. forgot to save/restore some member), maybe even maintaining backwards compatibility of savegame files.

The code is located in the same TypeInfo.cpp file. It contains several methods which traverse all the game objects and do something with them. All of them rely on the methods idTypeInfoTools::WriteClass_r and  idTypeInfoTools::WriteVariable_r, which implement the traversal itself. The latter method is responsible for interpreting every possible type of class member; for instance it contains special code which supports traversing all elements of idList container (which is ID's equivalent to std::vector):

int idTypeInfoTools::WriteVariable_r(
    const void *varPtr, const char *varName, const char *varType,
    const char *scope, const char *prefix, const int pointerDepth
) {
    //...(more code)...
    } else if ( token == "idList" ) {
        idList<int> *list = ((idList<int> *)varPtr);
        Write(varName, varType, scope, prefix, ".num", va( "%d", list->Num() ), NULL, 0);
        Write(varName, varType, scope, prefix, ".granularity", va( "%d", list->GetGranularity() ), NULL, 0);
        if ( list->Num() && ParseTemplateArguments( typeSrc, templateArgs ) ) {
            void *listVarPtr = list->Ptr();
            for ( i = 0; i < list->Num(); i++ ) {
                idStr listVarName = va( "%s[%d]", varName, i );
                int size = WriteVariable_r(listVarPtr, listVarName, templateArgs, scope, prefix, pointerDepth);
                if ( size == -1 )
                    break;
                listVarPtr = (void *)( ( (byte *) listVarPtr ) + size );
            }
        }

        typeSize = sizeof( idList<int> );
    } else if ( token == "idStaticList" ) {
    //...(more code)...
}

During traversal, method Write is called, as you can see on lines 5 and 8 of the last listing. This is actually not a method, but a function pointer, which is initialized differently depending on the current needs. This method accepts seven parameters:

varType | prefix | scope | varName | postfix | value | varSize
====================================================================================================
"float" | "" | "idEntity" | "m_MinLODBias" | "" | "0" | 4
"idBounds" | "idEntity::renderEntity." | "renderEntity_t" | "bounds" | "" | "(-6.75103283 -2.63098955 -20)-(7 2.5482502 49.51835632)" | 24
"idDict" | "" | "idEntity" | "spawnArgs" | "[7]" | "'noclipmodel'  '0'" | 0
"idEntity::entityFlags_s" | "" | "idEntity" | "fl" | ".invisible" | "false" | 0
"idList < idEntityPtr < idActor > >" | "idEntity::m_userManager." | "UserManager" | "m_users" | ".num" | "0" | 0

In this text, first line specifies order of seven parameters, and each of the next lines shows one example of their values, seven values per line, splitted with vertical dash. The first example corresponds to member idEntity::m_MinLODBias of float type, having zero value. The second one corresponds to member renderEntity_s::bounds of type idBounds (coordinates of two vertices are given), where renderEntity_s itself is a member idEntity::renderEntity. The third one shows 7-th entry of idEntity::spawnArgs dictionary (idDict) as 'key'-'value' pair. The forth example is a boolean value entityFlags_s::invisible in bitset structure idEntity::fl. The last one corresponds to number of elements in UserManager::m_Users, which has type idList (aggregated in idEntity::m_userManager).

Now let's return back to Doom 3 and game state saving. Whenever you save game in memory debugging configuration, a file named like "Quicksave_1_save.gameState.txt" is created near the ordinary savegame file. This is a text file of size about 40MB, where each member of each game entity is fully described. The full version of this file can be downloaded, and here is a small excerpt from it:

entity 36 idStaticEntity {
idEntity::entityNumber = "36"
idEntity::entityDefNumber = "28"
idEntity::spawnNode = "<unknown type 'idLinkList < idEntity >'>"
idEntity::activeNode = "<unknown type 'idLinkList < idEntity >'>"
idEntity::snapshotNode = "<unknown type 'idLinkList < idEntity >'>"
idEntity::snapshotSequence = "-1"
idEntity::snapshotBits = "0"
idEntity::name = "func_static_5"
idEntity::spawnArgs[0] = "'classname'  'func_static'"
idEntity::spawnArgs[1] = "'name'  'func_static_5'"
idEntity::spawnArgs[2] = "'model'  'models/darkmod/furniture/seating/chair_simple01.lwo'"
idEntity::spawnArgs[3] = "'origin'  '216.743 -47.89 20'"
idEntity::spawnArgs[4] = "'rotation'  '0.707107 -0.707107 0 0.707107 0.707107 0 0 0 1'"
idEntity::spawnArgs[5] = "'inline'  '0'"
idEntity::spawnArgs[6] = "'spawnclass'  'idStaticEntity'"
idEntity::spawnArgs[7] = "'solid'  '1'"
idEntity::spawnArgs[8] = "'noclipmodel'  '0'"
idEntity::scriptObject.idScriptObject::type = "<pointer type 'idTypeDef *' not listed>"
idEntity::scriptObject.idScriptObject::data = "<NULL>"
idEntity::thinkFlags = "0"
idEntity::dormantStart = "-2920"
idEntity::cinematic = "0"
idEntity::cameraTarget = "<NULL>"
idEntity::targets.num = "0"
idEntity::targets.granularity = "16"
idEntity::health = "0"
idEntity::maxHealth = "0"
idEntity::fl.notarget = "false"
idEntity::fl.noknockback = "false"

Whenever you restore game from a save in this configuration, full game state after loading is saved into file "Quicksave_1_restore.gameState.txt". Moreover, this restored state is compared field-by-field with the old one (which was written on save) using the same game state traversal code, and messages about mismatches (i.e. state diffs) are printed to the game console:

Conclusion

So all of this was about a tiny little part of Doom 3 source code: memory debugging and typeinfo processing. It is worth noting that typeinfo stuff was completely removed by ID from Doom3 BFG (it is missing in the sources). Most likely this is because of the maintenance burden it causes. You can read more about the issues caused by all this hackery in the corresponding TDM wiki article.

Speaking of TDM, now we have to understand how much value this system has. The first moves would be to fix all the issues detected currently, and to add support for some new data structures (include STL stuff). The TypeInfo system would either give some benefit or be removed, and it is not clear yet which path would be taken.

Share on:
TwitterFacebookGoogle+Diaspora*HackerNewsEmail
Comments (0)
atom feed: comments

There are no comments yet.

Add a Comment



?

social