Creating an in-game map with Unreal Runtime Virtual Textures

In-Game Map: The Requirements

I wanted to make an in-game map screen. My requirements were:

• Generated from world geometry
  • My levels are procedural and modifiable so I couldn’t just manually draw it once
• Stylistic
  • I wanted it to look a bit like an old-skool TTRPG map, not just an overhead game render

Here’s the final result, which is basically what I was looking for:

You can see that the map looks hugely different from the in-game render (you don’t usually play the game from that camera angle, I’ve just moved it to roughly match for comparison). Not only is it stylistic, it categorises materials into a smaller number of types for the map, to simplify it.

My first failed idea: Render to Texture + Switch Materials

When deciding how to go about this, my first thought was to use a render-to-texture. Use an overhead orthographic camera, render to this texture from above, and change the materials during that render, right?

Wrong. Unreal Engine has no support for changing materials in a render-to-texture. And if you try to hack it by hooking into the render process and changing materials back and forth yourself between render targets, you will completely ruin performance. So you just can’t do this with a standard RTT alone, even though other engines would let you do it this way.

My second failed idea: Stencil & post-processing

My second idea was that if I can’t change materials, I could render a value into the stencil buffer indicating the type of map content, then use a post-process to read that stencil value back and completely change how I render the final pixel in the map view.

This would work, but not in my case. I was already using the stencil buffer to create “silhouettes” of characters when they’re behind geometry. Basically my characters render into the stencil buffer and if a character value exists in the stencil buffer but at a depth which is behind the scene depth, then I render the silhouette.

The problem is that if I want to write to the stencil buffer from my world geometry, I have to enable “Use Custom Depth” on that geometry. Doing so will overwrite the custom depth for characters, and therefore break my silhouette rendering. Unreal doesn’t allow you enough control over the depth/stencil really: you either write to the custom depth and stencil, or neither (also, the stencil masking options are very primitive - either all bits or one bit, which is unhelpful). If Unreal had let me tell my world geometry to render to only the stencil and not custom depth, this would have worked for me, but it doesn’t.

The solution: Runtime Virtual Textures

There is one feature in Unreal Engine that lets you render differently depending on the render target: Runtime Virtual Textures. I hadn’t considered this because it’s usually associated with landscape rendering - you’d render your landscape into the RVT and then sample it to match your grass and objects to the ground colour for example.

However the magical thing about it is that any material can have a separate set of output pins just for runtime virtual textures:

This means your single material can output one set of values for the normal camera rendering, and something else entirely for writing to an RVT. Perfect! It’s a bit like using a Render To Texture with a material swap, except there’s a level of world-tiled, pre-rendered virtual texture in between.

It does, however, require a bit more setup.

Overview

Before we get to that, let’s just explain what we’re actually achieving here.

1. The world renders to the Runtime Virtual Texture (RVT), using a different set of material output pins
   • It only does this when something requests an area of the texture
2. We render from the RVT orthographically onto a plane, which is captured into another texture (Map RTT) for the final map UI view
   • We could skip the second RTT and render directly to the view if we were willing to possess the capture camera, but since I want to embed this into a UI we need the second RTT.
   • This capture is done using an actor with both a SceneCapture component and a plane which has a material which samples the RVT
   • The plane roves the world in order to pan around the texture since its world coordinates translate to RVT UVs
   • Zooming in/out just means scaling the plane and changing the capture ortho width

Technically it’s the presence of the plane labelled “Map RTT” which is using a material which samples from the RVT which is causing the RVT to be built. The “V” in “RVT” (virtual) means that the texture is built on demand, to a resolution according to the requests.

Enabling Virtual Texture Support

The first thing you need to do is enable virtual texturing support at the project level.

1. Open Project Settings
2. In Engine > Rendering > Virtual Textures, enable the “Enable virtual texture support” checkbox
3. Restart the editor

Creating the RVT and writing to it

An RVT is just an asset, so let’s create that.

1. Right-click a content folder, then Texture > Runtime Virtual Texture.
2. Name it RVT_Map
3. Open the asset. Under Layout > Virtual Texture Content, select “Base Color” (that’s all we’re going to use)
4. Save & close RVT_Map

Next, let’s set up our world materials to write the map-specific content. Open whatever base materials you’re using and make these changes:

1. Right-click, then type “Runtime” to filter it and select “RuntimeVirtualTextureOutput”
2. Connect “BaseColor” to the colour output you want in the RVT
   • I used some parameters to sample from stylised textures based on the XZ world position for example

Creating a Runtime Texture Volume

Next, we need a Runtime Texture Volume which will tell the RVT system the outer bounds of the world it needs to update for. Just use the regular Add Actor button and search for Runtime Virtual Texture Volume.

Once created, assign it to a Runtime Virtual Texture by setting the “Virtual Texture” property to RVT_Map.

Setting RVT Volume bounds

In all the Epic examples they generally tell you to change this in the editor and use the “Set Bounds” button with e.g. Landscape actors to set this up.

You can definitely do that. To set it manually in the editor, bear in mind 2 things:

• The RVT Volume’s origin is at the bottom-left, not the centre like other volumes
• You change the bounds by scaling the actor, and 1=1cm. So to make it 5 metres it would need a scale of 500.

But if like me, you have variable size levels generated at runtime, you can’t use the editor to size the RVT Volume. Luckily you can also set RVT Volume bounds at runtime:

// We have to set the RVT Vol movable temporarily
RVTVolume->VirtualTextureComponent->SetMobility(EComponentMobility::Movable);
// RVT has its origin at the corner
FVector Min, Max;
// TODO: Determine your Min & Max world bounds here
FVector Size = Max - Min;
// RVT volume uses the actor transform
RVTVolume->SetActorLocation(Min);
RVTVolume->SetActorScale3D(Size);
RVTVolume->VirtualTextureComponent->SetMobility(EComponentMobility::Static);

Telling Primitive Components to output to RVTs

In order for an actor to render into the RVT, it needs 2 things:

1. A material with a Runtime Virtual Texture Output node
2. The output texture(s) specified in its “Draw in Virtual Textures” property

So for example, in your Static Mesh Component’s properties, you should scroll down to the “Virtual Texture” section and add an entry to the “Draw in Virtual Textures” property, pointing it at RT_Map:

The “Draw in Main Pass” property can be changed to “Never” if you only want something rendered into the RVT but not the normal game view. In our case the default is fine since world geometry appears in both.

Static Geometry Only Beyond This Point

At this point, I should point out that you should only be rendering into an RVT with (mostly) static geometry. Don’t be tempted to render moving meshes into an RVT. A static mesh attached to a moving object will work, but it will perform very poorly since the RVT will be struggling to keep up. It’s OK if things change occasionally, and in fact my map geometry is modifiable (although not movable).

There are special details about Procedural Mesh Components later in this post.

Rendering the Final Map

OK so now we have the world rendering to a runtime virtual texture. Now we need to sample a subset of the RVT to satisfy our map view, which we do using a USceneCaptureComponent2D and a plane.

Creating a Render Texture

We now need a regular render texture so that what we capture for the current view of the map can be rendered into it. This is because we want to display this in a UI image, and we can only do that from a texture. If we wanted to, we could skip the extra RTT and render to the main view directly, but for maximum flexibility this intermediate RTT is more useful.

Create a new render texture asset:

1. Right-click a content folder, select Texture > Render Target
2. Call it RT_MapRender
3. Open it and change the size to your preference (I use 1024x1024)
4. Set Addressing Modes to Clamp
5. Set Render Target Format to “RTF RGBA8” since we don’t need HDR
6. Save & close

Creating the Scene Capture Plane Material

We need a material that samples from the RVT to display our map. You simply need to create a new material, say call it “M_MapReadRVT”, and set it up as follows:

1. Material Domain = Surface
2. Shading Model = Unlit
3. Sample the RVT:
   1. Right-click the graph, type “Runtime”
   2. Pick “Runtime Virtual Texture Sample”
4. Connect BaseColor from the RVT sample to output Emissive Color

That’s it - assign this material to the plane inside your scene capture actor.

Creating The Scene Capture Actor

Importantly, the plane is the only thing that this capture component is going to render. Because I like to work in C++ and because it’s the most compact way to show you the set up, here’s how that’s created:

AMinerMapCaptureActor::AMinerMapCaptureActor()
{
	PrimaryActorTick.bCanEverTick = true;

	CaptureComp = CreateDefaultSubobject<USceneCaptureComponent2D>("Capture2D");
	SetRootComponent(CaptureComp);

	CaptureComp->ProjectionType = ECameraProjectionMode::Orthographic;
	CaptureComp->OrthoWidth = 3000;
	// Don't capture all the time
	CaptureComp->bCaptureEveryFrame = false;
	CaptureComp->bCaptureOnMovement = false;
	CaptureComp->CaptureSource = SCS_FinalColorLDR;
	// we don't want to render everything, just the plane that reads from the RVT
	CaptureComp->PrimitiveRenderMode = ESceneCapturePrimitiveRenderMode::PRM_UseShowOnlyList;
    

	// Set up the plane static mesh
	
	PlaneComp = CreateDefaultSubobject<UStaticMeshComponent>("Plane");
	PlaneComp->SetupAttachment(CaptureComp);
	PlaneComp->SetRelativeRotation(FRotator(90, 0, 0));
	PlaneComp->SetRelativeLocation(FVector(100,0,0));
	PlaneComp->SetCollisionEnabled(ECollisionEnabled::NoCollision);

	// Hide the plane at first
	PlaneComp->SetVisibility(false);
	
	// Set plane material & mesh in the BP
}

void AMinerMapCaptureActor::BeginPlay()
{
	Super::BeginPlay();
	
	// We're ONLY going to render the plane in front of us, because that will sample from the RVT
	CaptureComp->ShowOnlyActors.Add(this);

	// Always point straight down
	FRotator Rotation = UKismetMathLibrary::MakeRotationFromAxes(FVector::DownVector, FVector::RightVector, FVector::ForwardVector);
	SetActorRotation(Rotation);

    // We'll explain this later
	UpdatePlaneSize();

}

Create a Blueprint subclassed from this C++, and change some settings:

1. Set PlaneComp’s Static Mesh to the standard Engine Plane mesh
2. Set PlaneComp’s Material to M_MapReadRVT
3. Set CaptureComp’s Texture Target to RT_MapRender

You can now drag this actor into your level and it’s almost ready to go.

Matching Plane Size

We need to make sure that at all times, the plane we’re rendering matches the ortho width of the capture camera, so that it always fills the view. This was called at BeginPlay, but we’ll also call it for zooming (which is done by changing the camera ortho width):

void AMinerMapCaptureActor::UpdatePlaneSize()
{
	// Plane is 100x100
	float Mult = CaptureComp->OrthoWidth / 100.0f;
	PlaneComp->SetWorldScale3D(FVector(Mult, Mult, Mult));
}

This is a long post, so you can stop reading now if you like. But there’s some other potentially useful elements

Starting & Stopping Map Rendering

We won’t have the map view open all the time, so we need to start / stop the map render as needed. These functions handle that:

void AMinerMapCaptureActor::StartRendering()
{
	PlaneComp->SetVisibility(true);
	// We NEED capture every frame to trigger the RVT update
	CaptureComp->bCaptureEveryFrame = true;
}

void AMinerMapCaptureActor::StopRendering()
{
	CaptureComp->bCaptureEveryFrame = false;
	PlaneComp->SetVisibility(false);

}

We set the plane visibility false so that the main game camera can’t catch a view of it, and so the RVT knows not to update until required.

When you open your Map UI, you need to call StartRendering, and StopRendering when you close it. Then you just need to reference RT_MapRender in an image, and you have your map view.

Panning and Zooming

What gets rendered into that map capture plane solely depends on its position and size in the world. So all you have to do to pan and zoom is move it, and change the ortho width.

Here’s one way of doing that:

// These member variables are defined in the class header:
//    float ZoomAccumulated;
//	  FVector2D PanAccumulated;
void AMinerMapCaptureActor::JumpToBasePosition(const FVector& BasePosition)
{
	SetActorLocation(FVector(BasePosition.X, BasePosition.Y, Altitude));
	RaisePanZoomEvent();

}
void AMinerMapCaptureActor::ResetZoom()
{
	CaptureComp->OrthoWidth = DefaultOrthoWidth;
	ZoomAccumulated = 0;
	UpdatePlaneSize();
	RaisePanZoomEvent();
}
void AMinerMapCaptureActor::ResetPan()
{
	PanAccumulated = FVector2D(0, 0);
	RaisePanZoomEvent();
	
}
void AMinerMapCaptureActor::Zoom(float Delta)
{
	ZoomAccumulated += Delta;
	
	
}
void AMinerMapCaptureActor::Pan(const FVector2D& Delta)
{
	PanAccumulated += Delta;
}
void AMinerMapCaptureActor::Tick(float DeltaSeconds)
{
	Super::Tick(DeltaSeconds);

	bool bChanged = false;
	if (!PanAccumulated.IsNearlyZero())
	{
		// Pan
		FVector2D ScaledDelta = PanAccumulated * PanSpeed;
		// Scale by zoom - more zoomed out == faster, more zoomed in == slower
		ScaledDelta *= CaptureComp->OrthoWidth / DefaultOrthoWidth;
		ScaledDelta *= DeltaSeconds;
		FVector Move = ScaledDelta.X * GetActorRightVector() + ScaledDelta.Y * GetActorUpVector();
		SetActorLocation(GetActorLocation() + Move);
		// Could lerp to 0
		PanAccumulated = FVector2D(0,0);
		bChanged = true;
	}

	if (!FMath::IsNearlyZero(ZoomAccumulated))
	{
		// Zoom
		// Negative values zoom out, positive zooms in (widen ortho)
		float Adjustment = ZoomSpeed * ZoomAccumulated * DeltaSeconds; 
		float NewWidth = FMath::Clamp(CaptureComp->OrthoWidth + Adjustment, MinOrthoWidth, MaxOrthoWidth);
		if (!FMath::IsNearlyEqual(NewWidth, CaptureComp->OrthoWidth))
		{
			CaptureComp->OrthoWidth = NewWidth;
			UpdatePlaneSize();

			bChanged = true;
		}
		// Could lerp to 0
		ZoomAccumulated = 0;
	}

	if (bChanged)
	{
		// Raise event so that icons can update themselves
		RaisePanZoomEvent();

	}
}

This pans / zooms based on the input and frame time, to give a consistent movement speed. Note that the “RaisePanZoomEvent” isn’t specified here, but it’s a Gameplay Messaging Subsystem message that gets broadcast so that the UI can know the pan/zoom changed - since it overlays icons on top of the map as well. This post is officially already too long to include that part here, maybe I’ll come back to that if there’s demand.

Special Case: Procedural Mesh Components

In my project I use a custom procedural mesh component with some optimisations & custom behaviour for our needs over the regular UProceduralMeshComponent, but out of the box, none of these will render into an RVT without changes. The reason is RVTs are only really performant for static geometry, so when UE does the render path for them, it only calls a subset of the primitive component rendering functions.

Only primitive components which contain an implementation of FPrimitiveSceneProxy::DrawStaticElements will be rendered to a RVT. This means if you try to use UProceduralMeshComponent to render to an RVT, it won’t work, because PMCs only render via the FPrimitiveSceneProxy::GetDynamicMeshElements route, which the RVT render path never calls.

The way to resolve it is to add a FProceduralMeshSceneProxy::DrawStaticElements implementation:

virtual void DrawStaticElements(FStaticPrimitiveDrawInterface* PDI) override
{
	if (RuntimeVirtualTextures.Num())
	{
		for (const FProcMeshProxySection* Section : Sections)
		{
			if (!Section || !Section->bSectionVisible)  
				continue;
				
			UMaterialInterface* SectionMat = Section->Material;
			check(SectionMat);

			FMeshBatch MeshBatch;
			MeshBatch.LODIndex = 0;
			MeshBatch.SegmentIndex = 0;

			MeshBatch.VertexFactory = &Section->VertexFactory;
			MeshBatch.Type = PT_TriangleList;

			MeshBatch.LODIndex = 0;
			MeshBatch.SegmentIndex = 0;

			MeshBatch.bDitheredLODTransition = !IsMovable() && Section->Material->GetRenderProxy()->GetMaterialInterface()->IsDitheredLODTransition();
			MeshBatch.bWireframe = 0;
			MeshBatch.CastShadow = 0;
			MeshBatch.bUseForDepthPass = 0;
			MeshBatch.bUseAsOccluder = 0;
			MeshBatch.bUseForMaterial = 0;
			MeshBatch.bRenderToVirtualTexture = 1;
			MeshBatch.MaterialRenderProxy = Section->Material->GetRenderProxy();
			// NOTE: Change this to represent your required RVT texture channels if you need more than BaseColor
			MeshBatch.RuntimeVirtualTextureMaterialType = (int32)ERuntimeVirtualTextureMaterialType::BaseColor;
			MeshBatch.ReverseCulling = IsLocalToWorldDeterminantNegative();

			MeshBatch.DepthPriorityGroup = SDPG_World;
			MeshBatch.bCanApplyViewModeOverrides = 0;

			MeshBatch.Elements.Empty(1);
			FMeshBatchElement BatchElement;
			BatchElement.IndexBuffer = &Section->IndexBuffer;
			BatchElement.FirstIndex = 0;
			BatchElement.NumPrimitives = Section->IndexBuffer.Indices.Num() / 3;
			BatchElement.MinVertexIndex = 0;
			BatchElement.MaxVertexIndex = Section->VertexBuffers.PositionVertexBuffer.GetNumVertices() - 1;
			MeshBatch.Elements.Add(BatchElement);

			PDI->DrawMesh(MeshBatch, FLT_MAX);
		}
	}
}

Notice how it only does something during a call which involves RVTs, and if you track this function it only gets called when the render state is marked invalid (when you change the geometry). So it’s fast enough to be used with an RVT, in contrast to actually moving objects.

If you’re using a UE source build you can just hack this function in, otherwise you can do what I did and make a custom version of UProceduralMeshComponent by duplicating it & renaming things. From my experience it’s good to be able to tweak it anyway since PMC is quite generic and there’s plenty of optimisations & improvements you can make quite simply by knowing your particular needs.

Update: UE 5.6

As of UE 5.6 the FPrimitiveSceneProxy class added a new property bSupportsRuntimeVirtualTexture which needs to be set in order for your procedural mesh to render into an RVT. Add this to the constructor to fix that:

#if ENGINE_MINOR_VERSION >= 6
		// UE 5.6 added this, required for rendering to RVT
		bSupportsRuntimeVirtualTexture = true;
#endif

Fin

Phew, this was a long one. But I haven’t seen much information about using RVTs for this sort of purpose online; they tend to be very landscape-biased. I’m grateful to Daniel on Bluesky for suggesting I look into RVTs when I was struggling with the other approaches, otherwise I wouldn’t have even thought of it as a solution to this problem.

Hope someone finds this useful!