Skyrim Memory Patch - fixing ILS, uGrids CTD, freezes
Posted: 16 Jan 2014, 21:20
Disclaimer
This patch will require your first born and a pint of your blood.
What bad can happen when Skyrim uses more memory without tripping over itself?
Fried Hardware maybe? If anything bad does happen then it is your problem since I already got your firstborn and I do not care. This information is provided as is. I am sharing it because I am hoping a large part of the community will benefit from it. Take it or leave it.
Be sensible about memory use and modding done right in general - refer to S.T.E.P. and Skyrim Revisited. If you want to go beyond the default limits of the game you need to have a stable foundation. Modding requires dedication and patience and you have to read a lot.
The information below helps my(!) Skyrim with higher ugrids and many mods. For the love of the holy cow make sure to only use well documented and tested ini settings or spend some time testing them thoroughly yourself. If you can not properly verify cause and effect all you do is wasting time. No need to mess with unknown, untested, unproven settings or mindlessly copy and paste made up things that have no effect.
tl;dr just give me the setting
For additional memory add this to the .ini
[NotPlacebo].
GiveFirstBornToSheson=1
Yes. Really.
Updates
2014, February 9 2014, January 29
- Added MemoryBlocksLog 1.2 SKSE plugin for simple logging. No functional changes. Public release on Nexus
- Removed all old versions of MemoryBlocksLog. Download latest version from Nexus
- Added MemoryBlocksLog 1.1 SKSE plugin for simple logging
- Added MemoryBlocksLog 1.0 SKSE plugin for simple logging
- SKSE says it is better to use Visual Studio 2008 than 2010
- Added How much memory to pre-allocate section
- Added Recommended section and note about precache killer
- Added a link to more detailed post from ZerOxShadows how to compile
- Added a note about the need of Stable uGridsToLoad in requirements
- Added hint to run updates after installing a 4 year old program from MS
It has been a while since the first memory patch that fixed 4GB LAA that made it into the official game. Thanks to Boris we have the second memory patch ENBoost taking care of textures not wasting main memory anymore. In this naming tradition this third memory patch will allow Skyrim to use that available space right from the start to fix ILS (infinite loading screen) without side effects and other memory related CTD (crashes to desktop) or freezes. After many hours of peeking and poking it turns out there is a simple fix and one has to wonder why this is not something that is implemented as an option or parameter for the PC version of the game.
How it works
When tesv.exe is started it allocates two 256MB blocks of memory. When the first block gets full, the engine will allocate more blocks. This can cause the known troubles. Thankfully, by telling the engine to request a bigger block from the start it magically makes use of it without any further ado. This isn't the case with the second block. Thankfully again, the second block does not fill up as quickly and once it is full the engine does not trip over itself when allocating more blocks.
To make the engine allocate a larger block of memory it needs to be patched. This needs to happen very early in the process. This can not be done by a SKSE plugin because they are executed much later. However, SKSE loader already patches tesv.exe before the game starts and since SKSE sources are provided adding the patch is straight forward.
Because of the nature of this patch I will only provide instructions how to do this.
I contacted Boris about this and he is doing his own tests and investigations. If this can be part of ENB I do not know. Best not to waste his time with silly questions
Will this help your ILS, CTD or freezing?
You have a 64bit OS and lots of main memory. You have decent hardware. You already use ENBoost and do not run out of VRAM. You went through all your mods, used tes5edit to clean them and use BOSS and what not. You searched and read lots of discussions and guides and spend hours over hours fixing and optimizing everything.
If you use safety load this fix will most likely help you to replace it and not have any side effects.
You can get VMMap and check if the first memory block is fully used. It is a yellow private data block that at first has 2 childs. Once all its memory is commited it only has one child and both the size and committed column showing the same number 262,144K.
This is not problematic in itself, but this condition raises the chance for ILS, CTD and freezing and this fix will most likely help avoiding any of that.
If you have "random" CTD you need to try to make them reproducible.
For example, typically the committed number grows when crossing cell borders but maybe also shrink when busy cells are unloaded. If it reaches 262,144K troubles may start. Use tb for toggleborder in skyrim console to show cell borders and if you freeze or CTD right on a yellow line this fix will most likely help.
SKSE can write minidumps that can be analyzed in what part of the program the game crashed and why. Even if this doesn't mean anything to you, you can check if the crash is the same every time. If this crash data would be supplied whenever people talk about CTD it could help tremendously in finding related discussions to a particularly crash condition and over time eliminate a lot of guess work.
If you happen to come across
Code: Select all
FAULTING_IP: TESV+bd832
004bd832 0f298660910000 movaps xmmword ptr [esi+9160h],xmm0
Basically, if you ever see that tesv.exe committed all of the memory block with the single child it is very likely this fix will help. It may even improve performance when zoning and also give you a homecooked meal once a day.
Magic
Here is a well documented patch that you can put into SKSE steam_loader main.cpp at the end of
Code: Select all
void InstallHook(void * retaddr, UInt32 hookSrc) Code: Select all
FlushInstructionCache(GetCurrentProcess(), NULL, 0);I will happily update anything if someone more knowledgable provides feedback about it.
Code: Select all
UInt32 enableMemPatches = 0;
if(GetConfigOption_UInt32("NotPlacebo", "GiveFirstBornToSheson", &enableMemPatches))
{
if(enableMemPatches)
{
_MESSAGE("Sheson took your first born in exchange for more memory");
//These patch addresses only apply to tesv.exe 1.9.32.0.8
//do not use this on any other version
//1st block, request 512MB (0x0300) instead of 256MB (0x0200)
//if too small cause of ILS or CTD when to much cell+buffer data does not fit anymore and engine allocates more memory.
//tesv.exe:00687E73 sbb edi, edi
//tesv.exe:00687E75 push 0
//tesv.exe:00687E77 and edi, 0x0FFFFE00
//tesv.exe:00687E7D push 0x158
//tesv.exe:00687E82 mov ecx, offset 0x01B418B0
//tesv.exe:00687E87 add edi, 0x200 <-----
//tesv.exe:00687E8D call 0x00A48D60
//tesv.exe:00687E92 test eax, eax
//tesv.exe:00687E94 jz short 0x00687EB1
//tesv.exe:00687E96 push 1
//tesv.exe:00687E98 push 0
//tesv.exe:00687E9A push offset 0x010CD3C8
//tesv.exe:00687E9F push 0x5500000
//tesv.exe:00687EA4 shl edi, 0x14
//tesv.exe:00687EA7 push edi
//tesv.exe:00687EA8 mov ecx, eax
//tesv.exe:00687EAA call 0x00A4DCF0
SafeWrite32(0x00687e87+2, 0x00000300);
//2nd block, request default 256MB (0x10000000), typically seems big enough
//changing this value is not recommended because side effects are not yet properly tested
//when 1st buffer is set over 512MB and there are a lot of mods this needs to be raised as well or tesv.exe crashes at start
//however there are no commits over 256MB, hard coded max limits later in the code
//tesv.exe:00A4E6B1 mov ebp, dword ptr 0x0106B268
//tesv.exe:00A4E6B7 push 4
//tesv.exe:00A4E6B9 push 0x2000
//tesv.exe:00A4E6BE push 0x10000000 <-----
//tesv.exe:00A4E6C3 mov [ebx+0x8D08]
//tesv.exe:00A4E6C9 mov [ebx+0x8D0C]
//tesv.exe:00A4E6CF mov [ebx+0x8D10]
//tesv.exe:00A4E6D5 push edi
//tesv.exe:00A4E6D6 mov [ebx+0x8D14]
//tesv.exe:00A4E6DC call ebp
//SafeWrite32(0x00a4e6be+1, 0x10000000);
//change movaps to movups (slower on old CPUs) in case memory is not 16byte aligned,
//there are most likely more, but when 1st buffer large enough this is not problem anymore
//this one is executed once per new cell load
//tesv.exe:004BD82F xorps xmm0, xmm0
//tesv.exe:004BD832 movaps xmmword ptr [esi+0x9160], xmm0
SafeWrite8(0x004bd832+1, 0x11);
}
}
For the love of the holy cow, please do not ask me for help with this. Check out this post from ZerOxShadows for a bit more detail on the process.
- Get SKSE sources, doh
- Get a free Visual Studio 2008, get service packs and run windows updates to make sure this has all the updates
- Open SKSE src/skse/skse.sln
- add patch to steam_loader/main.cpp
- in top menu change dropdown from Debug to Release
- in left pane right click steam_loader, build
- you will find new skse_steam_loader.dll in src/ske/Release
- if Skyrim does not start just put original skse_steam_loader.dll back
- 64bit OS - 32bit may work but untested, probably should use /userva switch, but if you are that desperate good luck
- a decent amount of RAM - more than 4GB
- a suitable graphics card with decent amount of VRAM - depending on texture sizes
- ENBoost - need to free up main memory
- Stable uGridsToLoad - if you want to test with higher uGrids. You may not really need it, but it fixes a recursion bug that could potentially cause CTD with default uGrids as well.
- You may still need a precache killer to load all the assets. Remember this fix is not making the game use more memory it just tells it how much memory for a certain block to anticipate.
Remember to add this to skse.ini
[NotPlacebo]
GiveFirstBornToSheson=1
If you look at the code you can see how this becomes a real setting
Check skse_steam_loader.log for the message about your offspring.
Compare VMMap
Before: two 256MB allocations.
After: one 512MB and one 256MB allocation.
Disable Safety Load and test if ILS are fixed. Test with any old saves you have, travel back and forth Solstheim.
You can still use Safety Load, but if you need it, it basically means the 1st memory block is not large enough for your setup.
Do speed runs, you should already know about tgm and player.setav speedmult
Use Skyrim Performance Monitor to keep an eye on VRAM
How much memory to pre-allocate
There is no need to pre-allocate a much larger block than the game will actually need. In fact doing so can have a negative impact, because other parts of the game need memory as well and the more is pre-allocated to this specific block the less is available to the other processes.
Remember, we are not telling the game to use more memory, we are telling it to pre-allocate the first block in one big continuous chunk instead of the default small chunk and later using additional smaller chunks somewhere else.
If you look at this screenshot of VMMap you can see that from the 524,288K of the first block 94,464K are not used. It is not much in this example, but it also means there is no use pre-allocating any more than this.
When testing for the highest needed value of your setup, you need to find the busy spots with lots of cell data. Windhelm Harbour comes to mind, the Rift or Solitude Harbor. It is typically the exterior spots where you had ILS. You can lower your value with a buffer in mind.
Any Megabyte size should work, but only experience will show if there needs to be a multiple of something.
Only raise the second block if the game doesn't start. It is a workaround at the moment and should only really be needed when testing limits with high uGrids or many mods.
Download MemoryBlocksLog. This is a SKSE plugin that writes the sizes of the two memory blocks to a logfile whenever more memory is committed to them. Check the readme.txt in the zip for more info.
Important Notes - Pay Attention
It is only recommended to raise the first block to 512MB and leave the second block alone. Like with all new mods this needs to be tested. Extensively.
This fix should be suitable for a lot of setups, provided there is enough main memory, a decent graphics card with plenty of VRAM and some common modding sense.
This will simply raise the limit of when the engine will trip over itself again when overloaded. If VMMap shows the same number in both size and committed column the engine is in trouble.
Raising uGrids can have effects on quests and game play, depending on installed mods.
It is your responsibility to find out what is safe for your setup.
Desired Feedback
If this fixes something I didn't describe yet.
If this fix works for you on exotic hardware or 32bit OS.
Please be detailed and thorough in any report.
Skyrim.ini
For the love of the holy cow, please get rid of all the nonsense settings. For starters you can check the tesv.exe for its content with notepad.
All settings are there to read. saveini does not dump everything it seems or there are strings that are not used.
Whatever wierd things you copied and pasted in your futile attempts to make higher uGrids stable, get rid of them now.
[Grass]
iGrassCellRadius=X
A little known setting, make this half uGrids if you want to raise fGrassStartFadeDistance in prefs.ini
Legal?
Use this info for whatever you like. This will stop working if for some strange reason a new version of tesv.exe comes out. Fat chance.
Please link back to this original post on ENB forum instead of copy and pasting.
If you use this in your loader, patch, mod or whatever I would be honoured if you keep the setting name. As an alternative make a dancing holy cow animation for one of MMOxReview videos. Please give credit if you use this.
Do not repost MemoryBlocksLog anywhere please.
The Sky is the Limit
Yeah, so I can set 768MB for both and raise uGrids to 17/19 with 100s mods and textures packs etc. The only practical use is pretty? screenshots. Papyrus is pretty much dead at this point. Test what is possible if you like. You understand the consequences. It will fry your hardware, mess up your save game and you will never be able to have kids again.
It is your responsibility to find out what is safe for your setup.
Rhetorical
Can we filter out the crap from the Internet please?
Can you please stop mindlessly repeating stuff that you read somewhere else as fact and thus adding to the crap on the Internet?
So SKSE can write minidumps and nobody uses that information to share and discuss?
You do not have a certain ini and need to ask about it?
So much time and resources wasted to answer the same basic questions over and over again. RTFM, RTFA?
If you need help with anything explained here I can not help you.
Please help yourself or patiently wait until someone spoon feeds you.
Did I mention that it is your responsibility to find out what is safe for your setup.
Credits
Thanks to Boris for his continuous work on ENB, ENBoost and having a Forum
Thanks to the SKSE team for SKSE and its sources
Thanks to who explains and document things for other to learn from it
Thanks to who can help themselves with the information provided.
Nord Cattle - HD cows
Holy Cow I fixed Skyrim - Sheson
==============================================================
