Reverse Engineering rpcss.dll: Hunting for the ROT's Hidden Structure

Part IV(?) of the "Inside the Running Object Table" series

When I started this research, I assumed finding the internal layout of the Running Object Table would be straightforward. Microsoft documents IRunningObjectTable at the API level, surely the implementation couldn't be far behind. I was wrong. What followed was a Ghidra session that turned up something far more interesting than I expected.

Where does the ROT actually live?
First surprise: there is no rpcss.exe on disk.
What Windows Task Manager shows as rpcss.exe is actually svchost.exe hosting rpcss.dll, classic service host pattern. The DLL lives at C:\Windows\System32\rpcss.dll and gets mapped into a dedicated svchost instance at boot. Every ROT operation your process makes through ole32.dll crosses an ALPC channel and lands in that DLL's code.
So that's our target.

Loading rpcss.dll into Ghidra
Standard procedure -drag, drop, let the auto-analyzer run. The interesting part comes when you load the PDB. Microsoft publishes symbol files for most system DLLs through their public symbol server, and rpcss.dll is no exception.
Grab symchk.exe from your Windows SDK Debuggers folder and run:

symchk C:\Windows\System32\rpcss.dll ^
    /s srv*C:\Symbols*https://msdl.microsoft.com/download/symbols

Point Ghidra at the downloaded PDB and suddenly FUN_18006XXXX becomes something readable. This is where it gets interesting.

First hit: CScmRotEntry
Searching the Symbol Table for ROT turns up the string u"ROTFlags" at address 180136070. Xref later, going inside a large function. But the real find is in the Symbol Table itself:
CScmRotEntry::GetAllowAnyClient
Not CROTEntry. Not CRunningObjectTableEntry. CScmRotEntry - SCM stands for Service Control Manager. This tells us something immediately, the ROT isn't just a COM subsystem. It's wired into the service infrastructure at a deeper level than the documentation suggests.
The decompilation of GetAllowAnyClient reveals the first concrete field offset:

return *(uint *)(this + 0x68) >> 1;  // bit 1 = AllowAnyClient flag

The real structure: LookupObjectInROT
Searching for CScmRot surfaces LookupObjectInROT - a function that takes ACTIVATION_PARAMS and walks the table looking for a matching object. From here we follow the call into CScmRot::GetObject, which is where the actual traversal happens.
The decompilation reveals the first architectural surprise:

cplVar13 = *(longlong **)(*(longlong *)(lpCriticalSection + 0x48) + uVar9 * 8);

The ROT is not a linked list. It's a hash table with 251 buckets (0xFB).
Each bucket is a pointer to a chain of CScmRotEntry objects. The hash function is a simple rolling hash over the moniker bytes:
chash = ((hash * 3) ^ byte) % 0xFB;
Aint cryptographic n complex. Fast and collision-tolerant for the expected workload of a few dozen registered objects.

Digging deeper: the class hierarchy
Listing all symbols with the CScmRot prefix turns up the full method table:

CScmRot::EnumRunning
CScmRot::GetEntryFromScmReg
CScmRot::GetTimeOfLastChange
CScmRot::IsRunning
CScmRot::Register
CScmRot::Revoke
CScmRotEntry::CScmRotEntry
CScmRotEntry::GetProcessID
CScmRotEntry::IsValid

Two things stand out. First, CScmRotEntry::CScmRotEntry it's a constructor, meaning this is a proper class with its own lifecycle. Second, CScmRotEntry::GetProcessID -every ROT entry stores the PID of the registering process. That field doesn't appear anywhere in the public API documentation.
Opening the constructor reveals something else entirely: CScmRotEntry is not a flat structure. It inherits from a base class:

CScmRotEntry
    └── CScmRotMgotEntryBase

"Mgot" itsa likely Marshaled Global Object Table. The base class constructor initializes most of the core fields:

CScmRotMgotEntryBase::CScmRotMgotEntryBase(
    CScmRotMgotEntryBase *this,
    ulong param_1, ulong param_2,
    _MnkEqBuf *param_3,
    CToken *param_4,
    ushort *param_5,
    tagInterfaceData *param_6,
    bool param_7,
    ulong param_8)
{
    *(undefined8 *)(this + 0x08) = 0;       // pNext = NULL
    *(undefined4 *)(this + 0x10) = 1;       // ref count = 1
    *(tagInterfaceData **)(this + 0x18) = param_6;
    *(ushort **)(this + 0x30) = param_5;    // display name
    this[0x40] = ... | param_7;             // flags
    *(_MnkEqBuf **)(this + 0x20) = param_3; // moniker eq buffer
    *(CToken **)(this + 0x28) = param_4;    // integrity token
    if (param_4 != NULL) {
        LOCK();
        *(int *)(param_4 + 8) += 1;         // manual AddRef
        UNLOCK();
    }
}

And CScmRotEntry adds its own fields on top:

*(ulong *)(this + 0x50) = 0x746f7263;   // magic: "crot"
*(ulong *)(this + 0x54) = param_5;
*(_FILETIME *)(this + 0x58) = *param_4; // last change time
*(tagInterfaceData **)(this + 0x60) = param_12;
*(ulong *)(this + 0x68) = param_7;      // ROTFlags

The complete structure map is being combining both constructors gives us the full layout:

struct CScmRotMgotEntryBase {
    void            *vtable;       // +0x00
    longlong         pNext;        // +0x08  next in bucket chain
    DWORD            dwRefCount;   // +0x10  starts at 1
    tagInterfaceData *pIFaceData;  // +0x18  marshaled interface
    _MnkEqBuf       *pMnkEqBuf;   // +0x20  moniker comparison buffer
    CToken          *pToken;       // +0x28  integrity level token
    ushort          *pwszName;     // +0x30  display name (wide string)
    longlong         unk_0x38;     // +0x38
    BYTE             bFlags;       // +0x40  bit 0 = various flags
    DWORD            dwField_44;   // +0x44
    DWORD            dwField_48;   // +0x48
    DWORD            dwField_4c;   // +0x4c
};
// Derived class
struct CScmRotEntry : CScmRotMgotEntryBase {
    DWORD            dwMagic;      // +0x50  0x746F7263 = "crot"
    DWORD            dwField_54;   // +0x54
    _FILETIME        ftLastChange; // +0x58  registration timestamp
    tagInterfaceData *pIFaceData2; // +0x60  second interface data
    DWORD            dwROTFlags;   // +0x68  bit 1 = AllowAnyClient
};

Three findings worth highlighting
The "crot" magic signature at +0x50. Every valid CScmRotEntry in memory carries the ASCII bytes c, r, o, t at this offset. This is almost certainly what CScmRotEntry::IsValid checks. For a memory scanner, this is a gift: you can locate entries by signature without knowing the head of the hash table.
CToken uses manual reference counting. The increment at param_4 + 8 is wrapped in LOCK/UNLOCK - interlocked, not COM AddRef. This means tokens are shared across multiple ROT entries from the same process using a private refcount mechanism entirely separate from COM lifetime management.
pwszName at +0x30 is the display name. This is the wide string you'd normally get from IMoniker::GetDisplayName. Reading it directly means a memory scanner doesn't need to deserialize tagInterfaceData or call any COM API to get human-readable moniker names — the string is right there in the entry.

The security picture
Walking CScmRot::GetObject surfaces three distinct access control mechanisms operating in sequence:

cRotEntryIsTrusted(plVar11, token)
RotMgotEntryIsAccessible(plVar11, token, ...)
CToken::IsTokenILLower(plVar11[5], plVar13[5], ...)

Trust check, accessibility check, integrity level comparison in that order. When multiple entries match the same moniker, the one with the higher integrity level wins. A sandboxed process cannot shadow a medium-integrity registration. AppContainer entries get discarded entirely for out-of-container clients — we even found the log string in the code:

"Disregarding ROT entry registered by AppContainer"

The ROT has a real access control model. It's just not documented anywhere near as clearly as the public API suggests.

Code and tooling: github.com/ssteelfactor-oss

[to be continued...]