# Fusion Unreal kit (v0)

Makes an Unreal Engine game a Fusion guest. Built in BUILD.md step 1.4 with
**VEIN Demo** (Unreal 5.6, D3D12) as the first game. It shows the game's world
inside Fusion, drawn from Fusion's camera, with correct depth against Godot's
own objects.

It is a **UE4SS mod** named `FusionKit`, made of three parts:

| Part | Where | Does |
|---|---|---|
| `main.dll` (Rust + a little C++) | `src/`, `cpp/` | Talks to Fusion (Fusion Protocol), converts between Fusion's space and Unreal's units, finds the capture textures on the GPU and hands every finished frame to Fusion through shared D3D12 textures; turns traced collision into Collision Hub chunks and pawns into entity lists (`src/world.rs`) |
| `Scripts/main.lua` | `lua/main.lua` | Generic Unreal logic: two `SceneCapture2D`s (color + depth) that follow Fusion's camera every frame; the collision traces; the list of pawns near Fusion's camera; spawn requests; keeping the game's own character near Fusion's camera |
| `Scripts/router_api.lua` | `lua/router_api.lua` | The Router API ([`docs/router-api.md`](https://steonmod.com/docs/router-api)): `Game.*`, `Router.block`, the router script's sandbox, and the kit's Unreal defaults (finding the player, opening the map, waiting for the world, Fusion's origin, entities, spawning) |
| `Scripts/router.toml`, `Scripts/router.lua` | `routers/<game>/` | The game's router: its blocks and `[block.bind]` data, and Lua only for what data can't say |

## How a frame gets to Fusion

1. **Lua**, every frame: `FusionKit_Camera()` gives Fusion's camera already
   converted to the game's world (cm, Z up, left-handed, horizontal FOV;
   `src/units.rs`). Both capture actors move there.
2. **Unreal** renders the two captures: `SCS_FinalColorLDR` into an RGBA8
   target (the game's finished picture, gamma-encoded) and `SCS_SceneDepth`
   into an R32F target (view depth in cm). The color capture keeps its
   history (`bAlwaysPersistRenderingState`), or auto exposure and TAA don't
   work and the picture is far too dark.
3. **main.dll** recognised those two render targets when Unreal created them
   (`FusionKit_ExpectTargets(w, h)` arms a hook on D3D12 resource creation,
   matched by size and format). It follows their state through Unreal's
   `ResourceBarrier` calls, applied in submission order.
4. Right after Unreal submits the command list that finishes them, main.dll
   runs its own tiny command list **on the same queue**: move both to
   `COPY_SOURCE`, copy into a free Fusion buffer (shared, simultaneous-access
   textures), move them back, signal the layer's shared fence. The buffer is
   published as READY; Fusion's `FusionGuestView` takes it as usual.

Nothing waits: if Fusion holds every buffer, that frame is skipped. Once frames
flow, Lua turns the game's own view off (`show Rendering`) and main.dll moves
the game window off-screen; both come back when Fusion goes away.

### Direct3D 11 games (step 7.3)

Checked with **Hello Neighbor Alpha 2** (Unreal 4.12, D3D11; `router conform` PASS, 54 pictures a
second, 2.1 frames behind Fusion's camera). main.dll watches both APIs (a D3D11 game may load
d3d12.dll too: 4.12 does), and the one that sees the targets created wins:

- `src/hooks11.rs` hooks `ID3D11Device::CreateTexture2D` to recognise the targets;
- `src/capture11.rs` copies them at each `Present` (the render thread, which owns the immediate
  context) through `fusion_win::d3d11::Ring`: a compute shader writes them into shared textures
  made with D3D12 (so Fusion opens them like any kit's) and signals the shared fence.

**`Present` is hooked in its code** (`src/present.rs`, MinHook), for both APIs, not in the swap
chain's function table. The Steam overlay hooks that table too and keeps one saved original, which
it overwrites whenever a new swap chain appears: with a table hook of the kit's in between, the two
called each other until the stack ran out (Hello Neighbor crashed in `gameoverlayrenderer64.dll`).

**Older engines** (the kit's Lua adapts by itself): capture sources are looked up by name (their
numbers changed); without `KismetRenderingLibrary.CreateRenderTarget2D` (before 4.13) the targets
are `CanvasRenderTarget2D`s; line traces use whichever of `LineTraceSingleByProfile`,
`LineTraceSingle`, `LineTraceSingle_NEW` the engine has; `HideActorComponents` with one argument
or two. **Before 4.13 there is no scene depth capture** (`SCS_SceneDepth`): the picture goes
without depth (`depth_format` 0) and Fusion draws it behind its own things; the router says
`gives = ["layer:color", ...]`. (The depth could come from the engine's own depth buffer during the
capture: not done, the platform comes first.)

For experiments, a `debug.txt` next to main.dll turns parts off, one word a line: `no_d3d12`,
`no_texture11`, `no_present11`.

## Collision, entities and spawning (step 1.4b)

- **Collision for Fusion's Collision Hub.** Every frame Lua does up to 48
  vertical line traces where main.dll asks (`FusionKit_NextTrace` /
  `FusionKit_TraceHit`), with the "Pawn" collision profile (what blocks a
  character), ignoring characters. main.dll samples a 0.5 m grid in Fusion's
  space, one 16 m column of chunks at a time, nearest to Fusion's camera
  first, up to 40 m away; a finished column becomes a height surface sent as
  `CollisionChunk`s. Columns farther than 64 m are cleared, and sampled again
  if the camera comes back. About 40 us a trace: 2.3 ms of the game thread
  per frame while sampling, nothing once the area is done.
- **Entities.** Four times a second Lua lists the pawns within 150 m of
  Fusion's camera: the router says what each is (or leaves it
  out); characters get their capsule as their box (listed at their feet),
  other pawns their mesh's bounds. main.dll sends them as one `EntityList`.
  About 1.5-3 ms per list with ~50 pawns.
- **Spawning.** A `Spawn` message from Fusion reaches the router (`spawn`);
  the kit answers with an `ActionResult`.
- **Questions (step 5.4).** Ten times a second Lua answers `Query` messages from
  Router Studio and `fusion-cli game` (`router_api.lua`, `M.answer`): live classes,
  objects, an object's fields through UE4SS's reflection (values only for plain
  types: reading some structs can crash the game), `test` (a draft router tried on
  the game) and `reload` (the game uses a new router.toml now). In VEIN: 300 classes
  in 0.3 s.
- **The game's own character follows Fusion's camera** (bind
  `player_follows_camera`): Unreal 5 loads the world around it (World
  Partition), so it's moved onto the ground under the camera when the camera
  is 30 m away. It's hidden from the captures.
- The kit logs what this costs the game thread every 10 s (`cost over 10 s:`
  in `fusion_kit.log`).

## Actions and teleports

Fusion's context menu, Lua and steering run a block's declared `actions` on its entities
(`Action` messages, by the id the kit listed them with): the router's Lua `actions` first, then
the kit's own `remove` (`K2_DestroyActor`), `set_health` (the matching `bind.entity` rule's
`health` field), `move_to` (`AIBlueprintHelperLibrary.SimpleMoveToLocation`, the target turned
into the game's world by `FusionKit_ToUnreal`) and `stop` (`StopMovement`). Unknown actions and
entities are refused. A `Teleport` moves the player's feet there. `router conform` tries a block's
`stop` and `move_to` on a real entity (spawning one of its kinds if none is near).

## Possess: playing as the game's character

A router's `character` block (VEIN's `vein.player`) lets Fusion play as the game's own player.
Each frame `FusionKit_Input()` gives Lua Fusion's movement axes, the look direction (converted
to Unreal's pitch and yaw) and the buttons; Lua drives the pawn through Unreal's own functions:
`PlayerController:SetControlRotation`, `Pawn:AddMovementInput` relative to the look, and
`Character:Jump` / `StopJumping`. So the game moves its character with its own movement,
physics and animation, and no key reaches the game window. `FusionKit_SetView` sends back the
eyes (the game's `PlayerCameraManager`) and the feet; Fusion's camera follows the eyes, and the
captures follow Fusion's camera as always. While possessed, `player_follows_camera` is off.

## Why a hand-written UE4SS interface

Normal UE4SS C++ mods need UE4SS's full source, including a private submodule
(UEPseudo) that requires an Epic-linked GitHub account. The kit needs only
UE4SS's mod base class (`RC::CppUserModBase`, re-declared with the same layout
in `cpp/ue4ss_abi.hpp`) and a few exported `LuaMadeSimple::Lua` methods to add
Lua functions (imported through `cpp/UE4SS.def`). Everything game-related goes
through UE4SS's Lua, which already knows how to call Unreal functions.
If a UE4SS update changes `CppUserModBase`, re-copy its declaration.
See [`docs/decisions/0003-unreal-kit.md`](https://steonmod.com/docs/adr-0003).

## Lua functions (from main.dll)

| Function | |
|---|---|
| `FusionKit_Connect(game_id)` | start looking for Fusion (discovery entry `game_id`) |
| `FusionKit_Status()` | plain-language connection status |
| `FusionKit_Camera()` | `nil`, or `camera_id, x, y, z, pitch, yaw, roll, fov_h, width, height` |
| `FusionKit_SetAnchor(x, y, z, yaw)` | the game-world point (cm) that is Fusion's origin, and the yaw Fusion's forward faces |
| `FusionKit_ExpectTargets(w, h[, depth])` | call right before creating the capture targets (`depth` 0: no depth target); `(0, 0)` after destroying them |
| `FusionKit_NextAction()` | `nil`, or `request_id, entity, name, a1..a8`: an action on a listed entity (answer with `FusionKit_Reply`) |
| `FusionKit_NextTeleport()` | `nil`, or `request_id, x, y, z, yaw`: move the player's feet there |
| `FusionKit_ToUnreal(x, y, z)` | a point in Fusion's space in the game's world (cm) |
| `FusionKit_Input()` | `nil`, or `move_x, move_y, pitch, yaw, buttons` while Fusion plays as the game's character |
| `FusionKit_SetView(eye_x, eye_y, eye_z, x, y, z, yaw)` | the possessed character's eyes and feet (cm) and yaw |
| `FusionKit_TargetsState()` | 0 none, 1 waiting, 2 found |
| `FusionKit_FrameCaptured(camera_id)` | the captures this frame used that camera |
| `FusionKit_Log(text)` / `FusionKit_Stats()` | `fusion_kit.log` next to main.dll / debug counters |
| `FusionKit_NextTrace()` | `nil`, or `x, y, top_z, bottom_z`: the next collision trace (cm) |
| `FusionKit_TraceHit(z)` / `FusionKit_TraceHit()` | the last trace hit at height `z` / hit nothing |
| `FusionKit_Entity(id, x, y, z, pitch, yaw, roll, min_x, min_y, min_z, max_x, max_y, max_z, health, max_health, team, kind, name)` | one entity for the list being built (cm; box in its own axes) |
| `FusionKit_EntitiesDone()` | sends that list to Fusion |
| `FusionKit_NextSpawn()` | `nil`, or `request_id, kind, x, y, z, yaw`: something Fusion asked to spawn (feet position, cm) |
| `FusionKit_Reply(request_id, ok)` | answers it (`ok` 1 or 0) |
| `FusionKit_NextQuery()` | `nil`, or `request_id, text`: a question from Router Studio or `fusion-cli game` ([`docs/protocol.md`](https://steonmod.com/docs/protocol)) |
| `FusionKit_QueryReply(request_id, ok, text)` | answers it: `ok` 1 and JSON, or 0 and what went wrong |
| `FusionKit_TomlToLua(text)` | a router.toml's text as Lua source returning its table (for `test` and `reload`) |

| `FusionKit_Router()` | `router.toml` (next to the scripts) as Lua source returning a table, or `nil` and the problem |

Routers are written against the Router API, not these functions: see
[`docs/router-api.md`](https://steonmod.com/docs/router-api).

## Build, install, test

```powershell
cargo build -p fusion-unreal
kits\unreal\install.ps1 -Game vein            # game must be closed; -Dev adds LiveConsole
target\debug\fusion-cli.exe host-test --game-id vein --eye 0,1.7,0 --look 0,1.7,-10 --depth-scale 0.01 --depth-fade 60 --out <dir>
Godot_v4.7.2-stable_win64.exe --path godot --log-file <file> res://scenes/tests/vein_test.tscn -- --auto --shots=<dir>
Godot_v4.7.2-stable_win64.exe --path godot --log-file <file> res://scenes/tests/vein_hub_test.tscn -- --shots=<dir>
```

The Godot check starts VEIN through Steam if needed (about 50 s until the first
picture), puts a magenta panel where about half of VEIN's pixels behind it are
nearer, and checks every pixel: the panel must show exactly where it is nearer.

**LiveConsole** (`install.ps1 -Dev`) runs Lua in the running game without a
restart: `kits\unreal\dev\lua.ps1 -Code 'out("%s", FindFirstOf("PlayerController"):GetFullName())'`.
It's how routers are explored and written; never ship it to players. A command
left from an earlier run isn't run again when the game starts. Read only plain
numbers and object properties blindly: reading some struct properties through
UE4SS can crash the game.

## Known limits (v0)

- The depth capture renders the scene a second time (about 5 ms on an RTX 3050
  Laptop); VEIN runs at ~42 FPS with both captures at 1280x720.
- Collision is sampled from above (a height surface): the ground under roofs
  and overhangs has no collision (the roof is what the trace sees), thin
  fences only catch where a sample lands on them, and walls are up to 0.5 m
  off (where the eaves are). Horizontal traces or the game's real collision
  shapes would fix this (later).
- VEIN's frame rate at 1600x900 drops from 27-39 to about 30 FPS with the
  sampling and the entity list running.
- One GPU crash in VEIN (`DXGI_ERROR_DEVICE_HUNG`, a GPU page fault, video
  memory at 87% of its budget) happened once, right after Fusion's camera flew
  100 m away, before the game's character followed the camera. It didn't come
  back since.
- Unreal versions whose D3D12 backend uses enhanced barriers on the capture
  targets aren't followed yet (the kit logs it and stops instead of guessing).
- Unreal before 4.13: no depth (see "Direct3D 11 games"); a picture without depth on a D3D12
  game isn't followed.