This article provides a brief overview of UE's shader compilation, loading, cooking, and serialization mechanisms, highlighting the key logical components. For brevity, the process described below focuses on the MeshMaterialShader, CookByTheBook, and ShaderCodeLibrary pipeline by default. This is the most common pipeline we work with and has the greatest performance impact. Other shader types, CookOnTheFly, and InlineCode mechanisms follow similar patterns and are essentially subsets of the process described here.

This discussion is based on UE 4.18. If you're aware of significant changes in newer versions, please feel free to share them in the comments.

Instruction: if in a hurry, you can just read the Overview Section and the intruduction of other Sections. The rest of each section is for reference use.

Overview

At the application level, the engine uses materials to determine how to render models. At the hardware level, the GPU uses shaders to determine how to render meshes. UE's material system is a generalized blueprint system, while shaders are highly platform and hardware-specific. The material compilation and cook mechanism bridges these two systems.

General Structure

During Material Cooking (Editor or Commandlet)

  1. Run CookCommandlet: CookOnTheFlyServer triggers the cook process for all referenced assets, primarily including material cooking
  2. When an individual material cook begins: Based on the target platform's FeatureLevel and MaterialQualityLevel, it triggers compilation of one or more ShaderMaps
  3. ShaderMap compilation: Translates material blueprints into HLSL shader code. This shader code is high-level and must be further compiled to target platform code
  4. Shader compilation: Iterates through mesh types that can use this material, compiling all supported shader variants for each type to generate a series of Shader Code in formats like OpenGL, DX, Metal, Vulkan, etc.
  5. After individual material compilation: Saves the ShaderMap to DDC
  6. After individual material cooking: Serializes the ShaderMap body into the material asset and saves all compiled shaders to ShaderCodeLibrary
  7. After all materials are cooked: CookOnTheFlyServer serializes the ShaderCodeLibrary to disk
Cooking Process
Click for Full-Res Image

During Game Loading (GameThread)

  1. Engine initialization: Reads and loads SharedShaderLibrary
  2. During material deserialization: Deserializes all ShaderMaps
  3. During material loading: Registers one or more ShaderMaps (depending on MaterialQualityLevel and device configuration)
  4. During material loading: For the ShaderMap corresponding to the current MaterialQualityLevel, loads all referenced shaders from ShaderCodeLibrary
  5. Shader loading: This step is hardware-dependent. OpenGL must submit shaders to the driver for compilation, while Metal can directly use pre-compiled shaders
  6. Loading complete: Synchronizes the current ShaderMap to the render thread
Runtime Loading Process
Click for Full Res Image

During Rendering (RenderThread)

  1. When building DrawPolicy: The RenderPass determines which ShaderType to use, then calls the material's GetShader function
  2. GetShader: The material retrieves the MeshMaterialShaderMap from MaterialShaderMap using the VertexFactory index, then obtains the final Shader using the ShaderType index
  3. Submit rendering: Submits rendering commands along with Shader resource indices to RHIThread
  4. RHI-related processing: At this point, Shader Code has been compiled for specific hardware, but different platforms require additional processing before submission, such as glLinkProgram. This article doesn't cover these details.

Cook Framework

Different Modes

There are two cook modes: CookOnTheFly and CookByTheBook. When packaging, we use CookByTheBook mode, which collects all resources through static analysis of resource dependencies based on a set of required assets, then performs offline cooking in one batch.

(Note: Despite having two modes, both use the UCookOnTheFlyServer class, which can be confusing when reading the source code)

The cook process entry point is UCookCommandlet::CookByTheBook, called through CookCommandlet. Material cooking has three modes:

  1. Inline: Compiled shaders are stored within material assets. This can lead to redundancy when there are many assets with extensive inheritance. Generally not used for mobile platforms
  2. Shared: Stores shaders centrally in an external ShaderCodeLibrary, improving shader reuse, compression rates, and reducing package size. This is the mainstream approach
  3. Native: Building on mode 2, pre-compiles shaders into hardware-specific intermediate formats, significantly reducing runtime compilation time. Currently only used for Metal

(Note 2: During loading, these three modes only differ in the final serialization stage and are all referred to as inline code, which can be misleading when reading the source code)

ShaderCodeLibrary Structure

  • FShaderCodeLibrary is the core interface
  • FShaderCodeLibraryImpl implements FShaderCodeLibrary
  • During cooking, FEditorShaderCodeArchive handles platform-specific serialization
  • At runtime, FShaderCodeArchive handles platform-specific deserialization
  • The number of FEditorShaderCodeArchive instances depends on the number of target platforms, stored in the EditorShaderCodeArchive array for AddShaderCode
  • The number of FShaderCodeArchive instances depends on the current platform, stored in the ShaderCodeArchiveStack array for RequestShaderCode

Shader Cook Process

Entry: UCookCommandlet::CookByTheBook

  • UCookOnTheFlyServer::StartCookByTheBook Initialize and build task queue
    • InitShaderCodeLibrary
      • FShaderCodeLibrary::InitForCooking
        • FShaderCodeLibraryImpl::FShaderCodeLibraryImpl Initialize ShaderCodeLibrary (static instance)
    • OpenShaderCodeLibrary(LibraryName)
      • FShaderCodeLibrary::OpenLibrary(LibraryName)
        • FShaderCodeLibraryImpl::OpenLibrary(LibraryName)
          • For each target platform, call FEditorShaderCodeArchive::OpenLibrary(LibraryName) Prepare for subsequent shader saving
    • Build task queue CookRequests
  • Main Loop
    • UCookOnTheFlyServer::TickCookOnTheSide Continuously retrieve materials from CookRequests queue and trigger compilation
      • UMaterial::BeginCacheForCookedPlatformData (See "Material Compilation" section below)
        • FMaterial::BeginCompileShaderMap Add shader compilation tasks to GShaderCompilingManager queue for async compilation (See "Compilation Trigger" section below)
          • This process checks DDC; if found, directly retrieves shader code from DDC
      • Async compilation using HLSLCC to call platform-specific backends (See "Compilation Progress" section below)
      • FShaderCompilingManager::ProcessAsyncResults Check GShaderCompilingManager async compilation results (See "Compilation Complete" section below)
        • FShaderCompilingManager::ProcessCompiledShaderMaps Add compiled shaders to DDC
      • When material cooking is complete, calls UMaterial::Serialize through SaveCookedPackages (See "Material Serialization" section below)
        • FShaderResource::SerializeShaderCode Add compiled shaders to ShaderCodeLibrary
  • After all cooking is complete, call UCookOnTheFlyServer::SaveShaderCodeLibrary (See "ShaderLibrary Serialization Process" section below)
    • FShaderCodeLibrary::SaveShaderCodeLibrary Serialize ShaderCodeLibrary
    • FShaderCodeLibrary::PackageNativeShaderLibrary Compile ShaderCodeLibrary to native format (iOS only)
      • FShaderCodeLibraryImpl::PackageNativeShaderLibrary
        • FEditorShaderCodeArchive::PackageNativeShaderLibrary
          • IShaderFormatArchive::Finalize Final serialization to Package

Material Cook Process

This occurs in two phases: compilation and serialization.

During the compilation phase, CookServer's TickCookOnTheSide continuously triggers material cooking, which in turn triggers shader compilation.

The serialization phase also occurs within TickCookOnTheSide, where completed materials are retrieved for serialization.

ShaderMap serialization happens in both phases: first to DDC, then to the material Package. When DDC exists, compilation is skipped.

Note that during the first serialization, Shader Code goes to DDC; during the second serialization, Shader Code goes to ShaderCodeLibrary.

Material and Shader Structure

  • UMaterial and UMaterialInstance are material assets, loaded and used on the Game Thread
  • FMaterial is the render thread material representation, used on the Render Thread
  • FMaterialResource wraps FMaterial, providing serialization, initialization, and other functionality
  • One UMaterial can contain multiple FMaterialResource instances, managed using two-level indexing: Quality Level and Feature Level
  • One FMaterial corresponds to one FMaterialShaderMap
  • FMaterialShaderMap uses two-level indexing to manage Shaders: Vertex Factory Type and Shader Type
  • One Shader corresponds to one FShaderResource, which wraps FShader
  • FShaderResource references and initializes RHI resources for use in rendering commands

Material Compilation

Entry: UMaterial::BeginCacheForCookedPlatformData

  • CacheResourceShadersForCooking
    • Iterate through required QualityLevels, allocate FMaterialResource
    • CacheShadersForResources
      • Iterate through FMaterialResource instances, check if cooking is needed (based on FeatureLevel, analysis of QualitySwitchNodes used by the material, and whether the material has QualityOverride)
        • FMaterialResource::CacheShaders
          • FMaterial::CacheShaders
            • FMaterialShaderMap::LoadFromDerivedDataCache
              • If found, deserialize from DDC
            • If not in DDC, call FMaterial::BeginCompileShaderMap to start compilation (See "Compilation Trigger" section below)
  • After compilation completes, wait for CookOnTheFlyServer to call ProcessAsyncResults
    • FShaderCompilingManager::ProcessCompiledShaderMaps (See "Compilation Complete" section below)
      • Store cached FMaterialResource instances in CachedMaterialResourcesForCooking

Material Serialization

Entry: UMaterial::Serialize

  • SerializeInlineShaderMaps
    • Iterate through each FMaterialResource in CachedMaterialResourcesForCooking
      • FMaterialShaderMap::Serialize
        • Iterate through SortedMeshShaderMaps (each corresponding to a VF type)
          • FMeshMaterialShaderMap::SerializeInline
            • Iterate through all shaders
              • SerializeShaderForSaving
                • FShaderResource::SerializeShaderCode
                  • FShaderCodeLibrary::AddShaderCode Save shader code to shader code library
                    • FShaderCodeLibrary finds the corresponding EditorShaderCodeArchive and adds to it

Shader Compilation Process

Except for GlobalShaders, most shader compilation is asynchronous. The editor or CookCommandlet first triggers shader compilation. GShaderCompilingManager distributes compilation tasks to workers and continuously serializes completed ShaderMaps to DDC. Shader compilation converts materials to target platform code through the following process:

  1. Translate material blueprint to HLSL
  2. Set material parameters (macro definitions)
  3. Set VertexFactory parameters (macro definitions)
  4. Set Shader parameters (macro definitions)
  5. Compile HLSL to target platform shader format

Compilation Trigger

  • FMaterial::BeginCompileShaderMap
    • Create a new FMaterialShaderMap
    • FHLSLMaterialTranslator::Translate - Translate material to HLSL
    • FHLSLMaterialTranslator::GetMaterialEnvironment Get material feature settings -> Input.SharedEnvironment
    • FMaterialShaderMap::Compile - Compile HLSL to platform-specific shader
      • FMaterial::SetupMaterialEnvironment Get material rendering settings -> Input.SharedEnvironment
      • Iterate through all VertexFactories, get corresponding MeshShaderMap
        • FMeshMaterialShaderMap::BeginCompile
          • Iterate through all ShaderTypes, use ShouldCacheMeshShader to determine if compilation is needed
            • FMeshMaterialShaderType::BeginCompileShader
              • Create a NewJob
              • FVertexFactoryType::ModifyCompilationEnvironment Get VF settings -> Input.Environment
              • FMeshMaterialShaderType::SetupCompileEnvironment Get Shader macros -> Input.Environment
              • GlobalBeginCompileShader
                • Build NewJob->Input
                • Continue building Input->Environment
                • NewJobs.Add(NewJob)
      • Iterate through Mesh-independent Shaders (same process as above)
      • Iterate through Pipeline Shaders (same process as above)
      • GShaderCompilingManager->AddJobs(NewJobs) Add to global compilation queue for async process compilation

Compilation Progress

Using multi-process OpenGL Shader compilation as an example:

  • ShaderCompileWorker::ProcessCompilationJob
    • FShaderFormatGLSL::CompileShader
      • FOpenGLFrontend::CompileShader
        • SetupPerVersionCompilationEnvironment
        • Set compiler-related macros
        • PreprocessShader Preprocessing: header replacement, comment removal
        • FHlslCrossCompilerContext::Init Initialize compiler
        • FHlslCrossCompilerContext::Run Call HLSLCC for compilation
          • RunFrontend Lexical analysis, syntax analysis (generate AST), semantic analysis (generate HIR)
          • RunBackend Generate Main function, code optimization
            • Key functions here are implemented by FOpenGLBackend
          • FOpenGLBackend::GenerateCode Generate GLSL code
        • BuildShaderOutput
          • Generate FShaderCompilerOutput based on output code, including parameter map, sample count, final code, hash value, compilation errors, etc.
          • Generate FOpenGLCodeHeader containing shader type, name, parameter bindings, UniformBuffer mappings, and other runtime dependencies
          • Serialize FOpenGLCodeHeader to final ShaderCode output. At runtime, the Header is deserialized first, then the shader body is read and compiled

When compiling iOS shaders in Native mode, the main difference is that the compilation produces Metal platform hardware-agnostic intermediate representation (IR) rather than text. Specifically, it's saved to a MetalLib file. Since MetalLib files are native binaries that the engine cannot read, and the engine needs shader reflection information during render submission, UE saves this Header separately in an additional MetalMap file.

Compilation Complete

Entry: FShaderCompilingManager::FinishCompilation or FShaderCompilingManager::ProcessAsyncResults

  • FShaderCompilingManager::ProcessCompiledShaderMaps
    • Iterate through FShaderMapFinalizeResults, find corresponding FMaterialShaderMap and FMaterial
      • FMaterialShaderMap::ProcessCompilationResults Process entire ShaderMap compilation results
        • ProcessCompilationResultsForSingleJob
          • Find corresponding VF's FMeshMaterialShaderMap
          • FMeshMaterialShaderType::FinishCompileShader Process single shader compilation result
            • FShaderResource::FindOrCreateShaderResource
              • FShaderResource::FShaderResource
                • FShaderResource::CompressCode
                  • FShaderResource.Code = FShaderCompilerOutput.Code Binary Shader Code finally enters ShaderResource
          • Add newly generated Shader to FMeshMaterialShaderMap
        • InitOrderedMeshShaderMaps Re-sort FMeshMaterialShaderMap
        • SaveToDerivedDataCache Serialize compilation results to DDC
          • FMaterialShaderMap::Serialize Same as "Material Serialization" section above, but saves to memory instead of Package
          • Save serialized results to DDC (including shader code)

ShaderLibrary Serialization Process

Key classes:

FEditorShaderCodeArchive: Used during cooking to save shader information

FShaderCodeArchive: Used at runtime to read shader information

IShaderFormatArchive: Native mode only, used at runtime and during cooking, platform-specific class handling actual serialization and deserialization

Relationships between these classes:

During cooking, shaders enter FEditorShaderCodeArchive through FShaderCodeLibrary

After cooking ends, FEditorShaderCodeArchive serializes all shaders to library files. For iOS, it compiles shaders to metallib through IShaderFormatArchive

At runtime, FShaderCodeLibrary reads multiple FShaderCodeArchive instances to load shaders

Overall Process

Multi-process Cook

Additional Processing (Non-Native Mode)

  • Strip
  • Compression

Compile Native Library (Native Mode)

  • First, during shader compilation, XCode compiles shaders to binary IR
  • Create IShaderFormatArchive
  • Iterate through all shaders in FEditorShaderCodeArchive
    • Strip
    • Add shader to IShaderFormatArchive
      • Save compiled binary IR from shader to intermediate folder and generate ShaderID
      • Add ShaderID to shader list
      • Add ShaderHash to ShaderMap
  • Finalize
    • Call XCode to link all binary IR in the intermediate folder

Runtime (Material Assets)

Each material asset loads one or more ShaderMaps based on MaterialLevel and FeatureLevel during deserialization, then registers all shaders for indexing and requests shader loading from ShaderCodeLibrary.

Loading ShaderCodeLibrary

  • LaunchEngineLoop
    • FShaderCodeLibrary::InitForRuntime - Initialize GlobalShaderLibrary
    • FShaderCodeLibrary::OpenLibrary - Initialize project ShaderLibrary
      • FShaderCodeLibraryImpl::OpenShaderCode
        • For iOS/Mac, call RHICreateShaderLibrary to create Library
        • Otherwise, instantiate FShaderCodeArchive to create Library
        • Add new Library to ShaderCodeArchiveStack

Reading Materials

  • UMaterial::Serialize
    • SerializeInlineShaderMaps
      • FMaterialResource::SerializeInlineShaderMap Deserialize ShaderMap
        • FMaterialShaderMap::Serialize
          • Iterate through supported VF types, generate FMeshMaterialShaderMap
          • InitOrderedMeshShaderMaps Build OrderedMeshShaderMaps
          • Iterate through each FMeshMaterialShaderMap type
            • FMeshMaterialShaderMap::SerializeInline
              • Iterate through Shaders
                • FShader::SerializeShaderForLoad
                  • FShaderResource::SerializeShaderCode
                    • Find a valid FShaderCodeArchive from ShaderCodeArchiveStack
                    • FShaderCodeLibrary::RequestShaderCode Request ShaderCodeLibrary to load shader (completes before PostLoad)
                • SerializedShaders::Add Store deserialized shaders for loading
        • GameThreadShaderMap = RenderingThreadShaderMap = FMaterialShaderMap
      • LoadedMaterialResources.Add(FMaterialResource)

Loading Materials

  • UMaterial::PostLoad
    • UMaterial::ProcessSerializedInlineShaderMaps
      • Iterate through all LoadedMaterialResources
        • FMaterial::RegisterInlineShaderMap
          • FMaterialShaderMap::RegisterSerializedShaders
            • Iterate through OrderedMeshShaderMaps
              • Various shader registration operations
            • Discard unused VF
            • Discard unused MeshShaderMaps
      • Discard unused Quality Levels
      • Store all MaterialResources in UMaterial::MaterialResources or UMaterialInstance::StaticPermutationMaterialResources
        • Two-level indexing by Quality Level and FeatureLevel
      • FMaterialResource::SetInlineShaderMap Mark this shadermap as loaded from cooked resources
      • UMaterial::CacheResourceShadersForRendering (See "RHI Initialization Trigger" section below)

Runtime (Rendering Resources)

After ShaderMap serialization completes, it's not immediately sent to the driver. Only after CacheResourceShadersForRendering is called will the ShaderMap for the current active Quality Level enter the loading process. When actually using shaders, they're retrieved directly from the loaded ShaderMap.

RHI Initialization Trigger

  • UMaterial::CacheResourceShadersForRendering
    • Iterate through MaterialResources for current ActiveQualityLevel and FeatureLevel
      • UMaterial::CacheShadersForResources
        • FMaterialResource::CacheShaders
          • FMaterialShaderMap::Register
            • Iterate through all Shaders
              • FShader::BeginInitializeResources Hand off to render thread for initialization

RHI Initialization

Using VertexShader as an example:

  • FShaderResource::InitRHI()
    • FShaderCache::CreateVertexShader
      • FShaderCodeLibrary::CreateVertexShader
        • FShaderCodeLibraryImpl::CreateVertexShader
          • FindShaderLibrary Find shader's library using hash
          • [Native Metal] If Native mode
            • Call RHICreateCodeArchive
              • Directly retrieve pre-compiled shader from ShaderLibrary (called Function in Metal)
          • [Shared] Use runtime shader compilation
            • FShaderCodeArchive::CreateVertexShader
              • LookupShaderCode Retrieve compressed shader code from ShaderCodeLibrary
              • UncompressCode - Decompress
              • Call platform-specific GDynamicRHI->RHICreateVertexShader
                • [OpenGL] CompileOpenGLShader
                  • Read FOpenGLCodeHeader, focusing on parameter bindings and UB mapping relationships
                  • GLSLToDeviceCompatibleGLSL Process ShaderCode again based on device and platform for compatibility
                  • For devices supporting BinaryProgramCache (virtually all do), shader compilation is deferred to link time and re-compressed

Usage During Rendering

Using MobileBasePass as an example:

  • ProcessMobileBasePassMesh
    • FDrawBasePassDynamicMeshAction::Process
      • Build DrawingPolicy
        • GetMobileBasePassShaders Get Shader
          • FMaterial::GetShader(ShaderType, VertexFactoryType)
            • FMaterialShaderMap::GetMeshShaderMap(VertexFactoryType) (RenderingThreadShaderMap.OrderedMeshShaderMaps[VF])
              • FMeshMaterialShaderMap::GetShader(ShaderType)
                • Get FRHIShader - for NativeMetal it's an MTLFunction, otherwise uncompiled (compressed) source code
      • CommitGraphicsPipelineState
        • SetGraphicsPipelineState
          • GetAndOrCreateGraphicsPipelineState
          • RHISetGraphicsPipelineState
            • [OpenGL] Calls glCompileShader and glLinkProgram, maintains BinaryProgramCache
            • [Metal] Builds PSO
      • DrawingPolicy.SetMeshRenderState
      • DrawingPolicy.DrawMesh