Fusion Unreal kit (v0)

Makes an Unreal Engine game a Fusion guest. Built in BUILD.md step 1.4 with VEIN Demo (Unreal 5.6, D3D12) as the first game. It shows the game's world inside Fusion, drawn from Fusion's camera, with correct depth against Godot's own objects.

It is a UE4SS mod named FusionKit, made of three parts:

How a frame gets to Fusion¶

1. Lua, every frame: FusionKit_Camera() gives Fusion's camera already converted to the game's world (cm, Z up, left-handed, horizontal FOV; src/units.rs). Both capture actors move there.
2. Unreal renders the two captures: SCS_FinalColorLDR into an RGBA8 target (the game's finished picture, gamma-encoded) and SCS_SceneDepth into an R32F target (view depth in cm). The color capture keeps its history (bAlwaysPersistRenderingState), or auto exposure and TAA don't work and the picture is far too dark.
3. main.dll recognised those two render targets when Unreal created them (FusionKit_ExpectTargets(w, h) arms a hook on D3D12 resource creation, matched by size and format). It follows their state through Unreal's ResourceBarrier calls, applied in submission order.
4. Right after Unreal submits the command list that finishes them, main.dll runs its own tiny command list on the same queue: move both to COPY_SOURCE, copy into a free Fusion buffer (shared, simultaneous-access textures), move them back, signal the layer's shared fence. The buffer is published as READY; Fusion's FusionGuestView takes it as usual.

Nothing waits: if Fusion holds every buffer, that frame is skipped. Once frames flow, Lua turns the game's own view off (show Rendering) and main.dll moves the game window off-screen; both come back when Fusion goes away.

Direct3D 11 games (step 7.3)¶

Checked with Hello Neighbor Alpha 2 (Unreal 4.12, D3D11; router conform PASS, 54 pictures a second, 2.1 frames behind Fusion's camera). main.dll watches both APIs (a D3D11 game may load d3d12.dll too: 4.12 does), and the one that sees the targets created wins:

• src/hooks11.rs hooks ID3D11Device::CreateTexture2D to recognise the targets;
• src/capture11.rs copies them at each Present (the render thread, which owns the immediate context) through fusion_win::d3d11::Ring: a compute shader writes them into shared textures made with D3D12 (so Fusion opens them like any kit's) and signals the shared fence.

Present is hooked in its code (src/present.rs, MinHook), for both APIs, not in the swap chain's function table. The Steam overlay hooks that table too and keeps one saved original, which it overwrites whenever a new swap chain appears: with a table hook of the kit's in between, the two called each other until the stack ran out (Hello Neighbor crashed in gameoverlayrenderer64.dll).

Older engines (the kit's Lua adapts by itself): capture sources are looked up by name (their numbers changed); without KismetRenderingLibrary.CreateRenderTarget2D (before 4.13) the targets are CanvasRenderTarget2Ds; line traces use whichever of LineTraceSingleByProfile, LineTraceSingle, LineTraceSingle_NEW the engine has; HideActorComponents with one argument or two. Before 4.13 there is no scene depth capture (SCS_SceneDepth): the picture goes without depth (depth_format 0) and Fusion draws it behind its own things; the router says gives = ["layer:color", ...]. (The depth could come from the engine's own depth buffer during the capture: not done, the platform comes first.)

For experiments, a debug.txt next to main.dll turns parts off, one word a line: no_d3d12, no_texture11, no_present11.

Collision, entities and spawning (step 1.4b)¶

• Collision for Fusion's Collision Hub. Every frame Lua does up to 48 vertical line traces where main.dll asks (FusionKit_NextTrace / FusionKit_TraceHit), with the "Pawn" collision profile (what blocks a character), ignoring characters. main.dll samples a 0.5 m grid in Fusion's space, one 16 m column of chunks at a time, nearest to Fusion's camera first, up to 40 m away; a finished column becomes a height surface sent as CollisionChunks. Columns farther than 64 m are cleared, and sampled again if the camera comes back. About 40 us a trace: 2.3 ms of the game thread per frame while sampling, nothing once the area is done.
• Entities. Four times a second Lua lists the pawns within 150 m of Fusion's camera: the router says what each is (or leaves it out); characters get their capsule as their box (listed at their feet), other pawns their mesh's bounds. main.dll sends them as one EntityList. About 1.5-3 ms per list with ~50 pawns.
• Spawning. A Spawn message from Fusion reaches the router (spawn); the kit answers with an ActionResult.
• Questions (step 5.4). Ten times a second Lua answers Query messages from Router Studio and fusion-cli game (router_api.lua, M.answer): live classes, objects, an object's fields through UE4SS's reflection (values only for plain types: reading some structs can crash the game), test (a draft router tried on the game) and reload (the game uses a new router.toml now). In VEIN: 300 classes in 0.3 s.
• The game's own character follows Fusion's camera (bind player_follows_camera): Unreal 5 loads the world around it (World Partition), so it's moved onto the ground under the camera when the camera is 30 m away. It's hidden from the captures.
• The kit logs what this costs the game thread every 10 s (cost over 10 s: in fusion_kit.log).

Actions and teleports¶

Fusion's context menu, Lua and steering run a block's declared actions on its entities (Action messages, by the id the kit listed them with): the router's Lua actions first, then the kit's own remove (K2_DestroyActor), set_health (the matching bind.entity rule's health field), move_to (AIBlueprintHelperLibrary.SimpleMoveToLocation, the target turned into the game's world by FusionKit_ToUnreal) and stop (StopMovement). Unknown actions and entities are refused. A Teleport moves the player's feet there. router conform tries a block's stop and move_to on a real entity (spawning one of its kinds if none is near).

Possess: playing as the game's character¶

A router's character block (VEIN's vein.player) lets Fusion play as the game's own player. Each frame FusionKit_Input() gives Lua Fusion's movement axes, the look direction (converted to Unreal's pitch and yaw) and the buttons; Lua drives the pawn through Unreal's own functions: PlayerController:SetControlRotation, Pawn:AddMovementInput relative to the look, and Character:Jump / StopJumping. So the game moves its character with its own movement, physics and animation, and no key reaches the game window. FusionKit_SetView sends back the eyes (the game's PlayerCameraManager) and the feet; Fusion's camera follows the eyes, and the captures follow Fusion's camera as always. While possessed, player_follows_camera is off.

Why a hand-written UE4SS interface¶

Normal UE4SS C++ mods need UE4SS's full source, including a private submodule (UEPseudo) that requires an Epic-linked GitHub account. The kit needs only UE4SS's mod base class (RC::CppUserModBase, re-declared with the same layout in cpp/ue4ss_abi.hpp) and a few exported LuaMadeSimple::Lua methods to add Lua functions (imported through cpp/UE4SS.def). Everything game-related goes through UE4SS's Lua, which already knows how to call Unreal functions. If a UE4SS update changes CppUserModBase, re-copy its declaration. See docs/decisions/0003-unreal-kit.md.

Lua functions (from main.dll)¶

| FusionKit_Router() | router.toml (next to the scripts) as Lua source returning a table, or nil and the problem |

Routers are written against the Router API, not these functions: see docs/router-api.md.

Build, install, test¶

cargo build -p fusion-unreal
kits\unreal\install.ps1 -Game vein            # game must be closed; -Dev adds LiveConsole
target\debug\fusion-cli.exe host-test --game-id vein --eye 0,1.7,0 --look 0,1.7,-10 --depth-scale 0.01 --depth-fade 60 --out <dir>
Godot_v4.7.2-stable_win64.exe --path godot --log-file <file> res://scenes/tests/vein_test.tscn -- --auto --shots=<dir>
Godot_v4.7.2-stable_win64.exe --path godot --log-file <file> res://scenes/tests/vein_hub_test.tscn -- --shots=<dir>

The Godot check starts VEIN through Steam if needed (about 50 s until the first picture), puts a magenta panel where about half of VEIN's pixels behind it are nearer, and checks every pixel: the panel must show exactly where it is nearer.

LiveConsole (install.ps1 -Dev) runs Lua in the running game without a restart: kits\unreal\dev\lua.ps1 -Code 'out("%s", FindFirstOf("PlayerController"):GetFullName())'. It's how routers are explored and written; never ship it to players. A command left from an earlier run isn't run again when the game starts. Read only plain numbers and object properties blindly: reading some struct properties through UE4SS can crash the game.

Known limits (v0)¶

• The depth capture renders the scene a second time (about 5 ms on an RTX 3050 Laptop); VEIN runs at ~42 FPS with both captures at 1280x720.
• Collision is sampled from above (a height surface): the ground under roofs and overhangs has no collision (the roof is what the trace sees), thin fences only catch where a sample lands on them, and walls are up to 0.5 m off (where the eaves are). Horizontal traces or the game's real collision shapes would fix this (later).
• VEIN's frame rate at 1600x900 drops from 27-39 to about 30 FPS with the sampling and the entity list running.
• One GPU crash in VEIN (DXGI_ERROR_DEVICE_HUNG, a GPU page fault, video memory at 87% of its budget) happened once, right after Fusion's camera flew 100 m away, before the game's character followed the camera. It didn't come back since.
• Unreal versions whose D3D12 backend uses enhanced barriers on the capture targets aren't followed yet (the kit logs it and stops instead of guessing).
• Unreal before 4.13: no depth (see "Direct3D 11 games"); a picture without depth on a D3D12 game isn't followed.