Populating shader cache I bought the game yesterday, have spent almost 10 hours trying to get it working right now - my PC meets the requirements to run this game, and plays other games that are more demanding than this one. In this Cemu 1.9.0 Guide I attempt to show you exactly what a Shader Cache does in relation to Cemu Emulator, and how you can easily build one yourself. Hi the shader cache location seems to have changed in the latest drivers and I can't find it, I use it for a few games copying the shader cache files and then putting them back in when I next want to launch the games making them stutter less. DirectX Shader Cache. DirectX Shader Cache contains the files that are created by the graphics system. These files can be used to speed up application load time and improve responsiveness. If you delete them, they will be re-generated as needed. But, if you believe the DirectX Shader Cache is corrupt or too large, you can delete it. Mar 25, 2018.
If the animations on your mobile app appear to be janky,but only on the first run, you can warm up theshader captured in the Skia Shader Language (SkSL) for asignificant improvement.
What is shader compilation jank?
If an app has janky animations during the first run,and later becomes smooth for the same animation,then it’s very likely due to shader compilation jank.
More technically, a shader is a piece of code that runs ona GPU (graphics processing unit).When a shader is first used, it needs to be compiled on the device.The compilation could cost up to a few hundred millisecondswhereas a smooth frame needs to be drawn within 16 millisecondsfor a 60 fps (frame-per-second) display.Therefore, a compilation could cause tens of frames to be missed,and drop the fps from 60 to 6. This is compilation jank.After the compilation is complete, the animation should be smooth.
Definitive evidence for the presence of shader compilation jankis to see
GrGLProgramBuilder::finalize
in the tracing.See the following screenshot for an example timeline tracing.What do we mean by “first run”?
On Android, “first run” means that the user might seejank the first time opening the app after a freshinstallation. Subsequent runs should be fine.
On iOS, “first run” means that the user might seejank when an animation first occurs every timethe user opens the app from scratch.
How to use SkSL warmup
As of release 1.20, Flutter provides command line tools for app developers tocollect shaders that might be needed for end-users in the SkSL(Skia Shader Language) format. The SkSL shaders can then bepackaged into the app, and get warmed up (pre-compiled)when an end-user first opens the app, thereby reducing the compilationjank in later animations. Use the following instructions to collectand package the SkSL shaders:
- Run the app with
--cache-sksl
turned on to capture shaders in SkSL: - Play with the app to trigger as many animations as needed; particularly those with compilation jank.
- Press
M
at the command line offlutter run
to write the captured SkSL shaders into a file named something likeflutter_01.sksl.json
. - Build the app with SkSL warm-up using the following, as appropriate:Android:iOS:If it’s built for a driver test like
test_driver/app.dart
, make sure to also specify--target=test_driver/app.dart
(e.g.,flutter build ios --bundle-sksl-path flutter_01.sksl.json --target=test_driver/app.dart
). - Test the newly built app.
Alternatively, you can write some integration tests toautomate the first three steps using a single command.For example:
With such integration tests, you can easily and reliably get thenew SkSLs when the app code changes, or when Flutter upgrades.Such tests can also be used to verify the performance changebefore and after the SkSL warm-up. Even better, you can putthose tests into a CI (continuous integration) system so theSkSLs are generated and tested automatically over the lifetime of an app.
Take the original version of Flutter Gallery as an example.The CI system is set up to generate SkSLs for every Flutter commit,and verifies the performance, in the
transitions_perf_test.dart
test.For more details, see the flutter_gallery_sksl_warmup__transition_perf
and flutter_gallery_sksl_warmup_ios32__transition_perf
tasks.The worst frame rasterization time is a nice metric from such integration tests to indicate the severity of shader compilation jank. For instance, the steps above reduce Flutter gallery’s shader compilation jank and speeds up its worst frame rasterization time on a Moto G4 from ~90 ms to ~40 ms. On iPhone 4s it’s reduced from from ~300 ms to ~80 ms. That leads to the visual difference as illustrated in the beginning of this article.
Limitations and considerations
- Why not just compile or warm up all possible shaders?
If there are only a limited number of possible shaders,then Flutter could do a warm-up and compile all of thembefore-hand to avoid such jank.However, for the best overall performance,the Skia GPU backend used by Flutter can dynamically generateshaders based on a lot of parameters at runtime(for example, draws, device models, and driver versions).Due to all possible combinations of those parameters,the number of possible shaders is exponential.In short, Flutter uses programs (app, Flutter, and Skia code)to generate some other programs (shaders). The number of possibleshader programs that Flutter can generate is simply too large. - Can SkSLs captured from one device help shader compilation jankon another device?
Theoretically, there’s no guarantee that the SkSLs from one devicewould help on another device (but they also won’t cause any troublesif SkSLs aren’t compatible across devices).Practically, as shown in the table on this SkSL-based warmup issue,SkSLs work surprisingly welleven if 1) SkSLs are captured from iOS and then applied to Android devices,or 2) SkSLs are captured from emulators and then applied to real mobiledevices. As the Flutter team has only a limited number of devices in the lab,we currently don’t have enough data to provide a big picture of cross-devicehelpfulness. We’d love you to provide us more data points to see how itworks in the wild. (FrameTiming
can be used to compute the worst framerasterization time in release mode; the worst frame rasterization time isa good indicator on how severe the shader compilation jank is.) - SkSL warm-up doesn’t help newer iPhones using Metal.
Flutter recently migrated from OpenGL to Metal for all newer iOSdevices. However, Skia currently only implements the SkSL warm-up forOpenGL. So the SkSL warm-up would only speed up older iOS devicesby default. If you find shader compilation jank to be an issuefor your app on newer iPhones, please let us know by filing aGitHub issue. In the longer term, we have a plan to use test-basedshader warm-up to mitigate this. If there’s an urgent need for fixingshader compilation jank on newer iPhones, please leave feedback onIssue 61045, and we can help you turn on OpenGL for your app.
If you have questions on SkSL shader warm-up,please comment on Issue 60313 and Issue 53607.If you have general shader warm-up questions,please refer to Issue 32170.
DDC stands for Derived Data Cache.
Many Unreal Engine Assets require additional 'derived data' before they can be used. (A simple example would be a Material that has a shader). Before the Material can be rendered, the shader must be compiled for the platform the editor is running on.
Because the derived data is large, and at times may need regenerating, it is not checked into source control. Instead it's kept in the Derived Data Cache (DDC).
Where the DDC Is Stored
Depending on how your project and system are configured, there can be several DDC caches in a hierarchy that ranges from fast to slow. When assessing derived data, your system will do the following to determine how quickly it can access it:
- When a piece of derived data is needed, the fastest cache is checked first, then the next fastest, and so on, until the data is found.
- When the data is found, it is copied into the fastest local cache so that it is quicker to access next time.
- If the data is not found, then it is first generated, then asynchronously copied into the caches so it is available for you (and potentially your team) in the future.
Content stored in the DDC is disposable, and can be regenerated at any time using the data stored in the .uasset file. Storing these derived formats externally makes it possible to easily add or change the formats used by the engine without needing to modify the source Asset file.
Types of DDCs
Typically, you will have at least the following two caches:
- Boot DDC: Startup files that are loaded into memory to increase boot times.
- For Example: ProjectDir/DerivedDataCache/Boot.ddc
- Local DDC: Derived data for your projects. Permute 2 2 5 13.
- For Example: EngineDir/DerivedDataCache
However, your project may also be set up to use a DDC Pak or Shared DDC.
DDC Pak
If you download Unreal Engine from the Epic Games Store, the engine will come with a DDC Pak (.ddp). The DDC Pak contains derived data for all engine content, so you can start working without compiling shaders and so on. Similarly, some samples are shipped with a DDC Pak for the same reason.
- Engine DDC Pak
- For Example: EngineDir/DerivedDataCache/Compressed.ddp
- Project DDC Pak
- For Example: ProjectDir/DerivedDataCache/Compressed.ddp
Shared DDC
One of our strongest recommendations for teams in the same location is that they set up a shared DDC. This is a network drive that all team members and build machines can read/write to. This then writes off the cost of creating any required DDC data across the entire team. For example, when an artist edits a shader, the DDC data will be written straight to the DDC share.
- Shared DDC: Will be either a network or mapped drive.
An example of a shared drive would be: epicgames.netrootDDC-Global-Fortnite
Using a Shared DDC
A studios should use a shared DDC that all users in a particular location can access. This way, only one person needs to build the derived Asset format(s) and they will be automatically available to all other users. There will occasionally be stalls when Assets need to be processed, but the results are stored and shared. Even on a fairly small team, sharing Asset processing work in this way will eliminate most processing time.
It is not advised to copy an entire DDC across the internet, or back up your DDC, or restore a DDC from a backup. While not necessarily harmful, this would be a waste of time—it takes longer to transfer the amount of data stored in the DDC than it does to generate it from scratch locally. If you have a large project and want to distribute pre-built DDC data, you should generate a DDC Pak.
Setting Up Shared DDC
As you may see in
BaseEngine.ini,
the editor is already set up to use a Shared DDC that can be enabled one of three ways. Here are the options from most to least recommended.
- Add an override (as pictured below) in the DefaultEngine.ini for your project that sets the path to a valid location for your team.
[DerivedDataBackendGraph] Shared=(Type=FileSystem, ReadOnly=false, Clean=false, Flush=false, DeleteUnused=true, UnusedFileAge=10, FoldersToClean=10, MaxFileChecksPerSec=1, ConsiderSlowAt=70, PromptIfMissing=false, Path=YourCompanyServerDDC, EnvPathOverride=UE-SharedDataCachePath, EditorOverrideSetting=SharedDerivedDataCache)
- Set an environment variable (as pictured below) of
UE-SharedDataCachePath
(On Mac/LinuxUE_SharedDataCachePath
) to the folder to use. - In the editor, set the
SharedDerivedDataCache
variable.
Click image for full size.
You don't have to do anything special to YourCompanyServerDDC; as long as it is writable, the engine will create the structure it needs.
Disabling a Shared DDC
Developers working remotely with a project that has a shared DDC configured may experience poor performance due to the fact that it takes longer to access DDC data than it does to generate it. To temporarily disable a shared DCC, use one of the methods below:
- Pass -ddc=noshared on the command line.
- Set the environment variable to a local drive hard drive:
UE-SharedDataCachePath=None
- On Mac:
UE_SharedDataCachePath=None
Building Derived Data
The user who imports the Asset is the one who builds the derived data since they will most likely be using and testing that Asset in the Engine. However, there may be times when a new Asset needs to be processed. This happens automatically on an as-needed basis and should not result in much of an impact when running on fast hardware, though there may be occasional stalls.
You can fill your DDCs at any time by running the following command:
What Is A Directx Shader Cache
Epic Games does this on a nightly basis to ensure that the DDC is always primed, but it is not necessary as the general automatic caching feature should suffice.
Mounting Your DDC Folder
For the DDC to be used over your internal network, the folder that contains the DDC must be mounted to a network drive. If you are unfamiliar with how to do this, see Map A Network Drive on Windows.
Distributing with a DDC
Cooking is the preferred method for packaging games since cooked builds do not need or use a DDC. However, DDCs can be packaged for distribution if the need arises.
Cooking is the preferred method for packaging games since cooked builds do not need or use a DDC. However, DDCs can be packaged for distribution if the need arises.
To package a DDC:
- Run UE4Editor.exe from the UE4/Engine/Binaries/Win64 directory, passing the arguments shown below:
UE4Editor.exe ProjectName -run=DerivedDataCache -fill -DDC=CreatePak
- This creates a DDC.ddp file in the UE4ProjectNameDerivedDataCache directory.
- The engine automatically detects and uses the .ddp file.
DDC Settings
Where are DDC settings kept?
DDC settings are kept in
DefaultEngine.ini
under the [DerivedDataBackendGraph]
section. You can view the defaults in BaseEngine.ini.
Common Questions
Q: Can I have multiple DDC settings?
Yes! Just create a new [YourDDCSettings] entry in
DefaultEngine.ini
and then run the editor with -ddc=YourDDCSettings.
At Epic, we do this for three reasons:
- For teams in offices that want to use their own DDC instead of the global one.
- When creating DDC Paks we use one of several options that control what goes into the .pak file, such as
[CreateInstalledEnginePak].
- For people working from home who don't want to use a Shared DDC due to slower internet or not having VPN enabled.
You'll see that
BaseEngine.ini
contains several DDC entries by default, including NoShared, which is used for #3 above (-ddc=NoShared
). Q: I am short on disk space. Can I move the Local DDC elsewhere?
Yes! Just set an environment variable of
UE-LocalDataCachePath
to a path of your choice. For example: UE-LocalDataCachePath=d:DDC.
- At the command prompt, type
'setx UE-LocalDataCachePath d:DDC'.
- Restart Unreal Engine, along with any application you launch it from, such as Epic Launcher, UGS, or Visual Studio.
This can be used when you are short of space on your primary drive, or when you are working on multiple branches and wish to ensure the data isn't duplicatednote
Q: I have a slow network connection. Can I turn off the Shared DDC?
Yes! There are two options, depending on whether you want this to be temporary or permanent:
- If you set the environment variable for a DDC to
None
it will disable it. In this case, you would setUE-SharedDataCachePath=None.
- Launch the editor with
-ddc=noshared.
Q: Can I change the Shared DDC path?
Yes! You can change it by setting the
UE-SharedDataCachePath
mentioned above. However, you should never set
UE-SharedDataCachePath
to a local path! Doing this means you would have both Local and Shared caches on your machine—twice the disk space for zero gain! Q: How can I diagnose DDC issues?
If you think your editor is not correctly reading DDC data, then search your logfile for
LogDerivedDataCache.
Here you can see:
- There's no project or Engine Pak cache, which is what would be expected as I compiled this build myself from Perforce.
- My local data cache path is mapped to d:DDC and is writable.
- I am using the Epic shared cache, which is writable.
You can also run with
-logcmds='LogDerivedDataCache Verbose'
to turn on verbose logging. Q: What is the S3 DDC?
The Fortnite DDC is extremely large due to the amount of content we have. When Covid19 struck, we quickly realized that it was very inefficient for developers to pull all this data from a network share over the limited bandwidth of a VPN to the office, so instead we distribute the most commonly used data via S3.
Setting up the engine to use the S3DDC is extremely easy, but you will need to set up automation that generates the DDC content and uploads it to S3.
Q: What is IDDCUtilsModuleInterface?
This interface allows you to use ? Name arguments in your DDC settings that can then be resolved dynamically at runtime based on platform or domain.
Unfortunately, we don't ship an example of this, but implementing it is straightforward:
Click image for full size.
Q: How do I create a DDC Pak?
Creating a project-specific DDC:
UE4Editor.exe ProjectName -run=DerivedDataCache -fill -DDC=CreatePak
This commandlet will create a DDC for all content in the project. Alternatively, you can simply supply -DDC=CreatePak while running automation or even a user session to generate a more targeted set of content.
What Is Directx Shader File
Creating an Engine DDC:
What Is Directx Shader Cache
UE4Editor.exe -run=DerivedDataCache -fill -DDC=CreatePak