Skyrim Memory Patch - fixing ILS, uGrids CTD, freezes

[keithinhanoi]

@sheson

Thanks a bunch for creating the memory block logging SKSE plugin! You are awesome.

It's clearly much more elegant and slicker than my .bat hack job I made to Skyrim VM usage snapshots in VMMap, and probably affects performance far less.

BTW - I noticed in the included read me the instructions are repeated twice. Also - I LOVE your included screenshot, and the your new tag line, Holy cow, I fixed Skyrim.

Finally, I have some very TECHNICAL questions about the patch method:

Could you please explain a bit more about the third part of your patch - changing movaps (MOVe Aligned Packed Single-precision floating-point values) to movups (MOVe Unaligned Packed Single-precision floating-point values)?

I've been reading up on the need for data alignment, including this explanation from IBM that's nice (though PowerPC-centric). However it's not entirely clear in the case of in Skyrim whether CTDs due to unaligned data are directly related to which CPU you have, or if there are other factors.

Anyhow, it appears that with that last change in the patch, you're trying to address the original problem of Skyrim crapping out when it needs to increase the memblock 1 allocation, but it's not clear what you mean with the comment of "there are most likely more". Do you mean there are other instances of movaps being used in TESV.exe?

Also, are you aware of a way to know whether a change to movups is absolutely necessary on a particular system, given that there's some potential performance hit in changing to movups (as aligned data access is more efficient despite having to "throw" away some unneeded bits)?


@Tuoni

My .bat file solution has now been superceded by shesons memblock logging SKSE plugin, so I highly recommend using that instead.


@pangallosr

The red text from ENB will only display if you have UsePatchSpeedhackWithoutGraphics=true in enblocal.ini, located in the main skyrim directory.

This setting turns off nearly all of the graphical post-processing features of ENB, though, so if you're using ENB to enhance the graphics you will want to leave it set to false, but then the settings under the [MEMORY] heading need to be correct for your particular setup in order for ENB's memory managment enhancement to work properly.

There's a STEP wiki guide on this, here. Try clicking on the Quickstart tab, and reading through that for an overview.

[sheson]

@keithinhanoi

I silently updated the zip with a fixed readme.

That specific movaps was the first hint something is really wrong when skyrim allocates more memory past the original 256MB. It is part of a group of commands that set a new cell buffer to zero. (fun fact: one of skyrim crashes can be caused by a command that sets 128bit to zero) It is the first command I patched to fix exterior buffer related CTDs, but it also is a dead give away something being amiss between additional allocations and whatever other code in skyrim expects. It prompted me to look at the initial allocation itself... and in the end just avoid the whole issue altogether.

If I remember correctly what I read - somewhere in a userpost on the Intel website - the whole thing that movups taking more cycles on old CPUs it really means a few generation back when MMX instruction set was a new thing. I just skimmed at it, since the change from protection fault to working is more important.

That particular command is executed once per cell load. So not that often actually. Tops, it may run as many times whatever the ugrids is when crossing a yellow border since a whole row of cells is loaded, but only for cells that are not in the buffer.

I know there are more movaps, but with increased first block none of them is doing unaligned reads/writes in my further tests atm I can make them CTD but only in circumstances where the engine is in extreme stress. Something else will break a ms later anyhow. I would leave them so we know of any more weird memory allocations and to safe a couple cycles just in case.

So with my patch of first block, there is no need to change that movaps as long as the block is large enough. You could use it as a defined CTD to know first block is full and exterior buffer needs more space. I doubt you can measure the performance impact of one command either way, but give it a try.

NINJA EDIT: Having just now spelled this all out, it kind of explains why some users report performance increase using the patch. If your Skyrim used additionally allocated memory for first block stuff it wasn't aligned. Generally speaking, non aligned memory makes things slower. Also with patch less memory management needs to be done. So there

[keithinhanoi]

Intriguing.

So from what you're saying, I understand that -

• The 1st memory block affected by your patch is used for Skyrim's exterior cell buffer allocation.
• The last part of the patch is just to help when the initial exterior cell buffer allocation is full, and needs to be increased.
• When you cross a cell border, buffered data for the unneeded ones is zeroed out and data for the new ones is loaded.
• Even though increasing the 1st memory block allocation avoids crashes associated with the commands to automatically increase its size, there may still be other situations where memory address block allocations need to be increased that could cause crashes due to the use of the movaps function on unaligned data.

Is that correct?

If all that's right, then it seems you may want to talk to Queued, the author of the d3dx9 library variant of your patch, SSME, because he has quite a lot of experience with identifying and avoiding the causes of memory exception errors in Fallout NV with his New Vegas Anti Crash (NVAC) patch.

Perhaps he's aware of other situations where there's a chance movaps will be used on unaligned data in FNV that would also apply to Skyrim. Also, his approach is interesting because he encourages users to submit errors from their logs so that they can be added to the list that NVAC can "catch".

Your explanation has also helped me to see very clearly why, for example, before using your patch I would get crashes along the road between Riverwood and the Guardian Stones. Using the tb (toggle borders) console command I could see there are at least three spots along the path there where you may happen to step on the corner borders of 4 cells, which means you'd potentially have double or even triple the normal number of cell buffer clearing and loading (compared to crossing from the side of one cell into the next.) Crash-o-matic!

I appreciate you taking the time to explain this (not to mention all of the countless hours figuring it out in the first place!)

[thalixte]

@sheson

Thks a lot for your memory log skse plugin. I will check it as soon as possible.

I have tested SSME, and it works as intended, no problems so far. Better method than build a modified skse_loader.exe, as it does not interfere with other skse libs, and respect skse copyright rules, until a new version of SKSE is avalaible.

[keithinhanoi]

@sheson

Finally got a chance to try out your new MemoryBlocksLog SKSE plugin.

If it's not too tricky or a performance hit, I'd like to suggest a change to make it easier to read.

I think instead of logging a change to either block on a separate line, such as this:

Code: Select all

logging of blocks enabled
logging all values
Block1 Max	512MB
Block2 Max	256MB
Block2	8
Block1	3
Block1	0
Block1	3
Block1	7
Block1	0
Block1	1
Block2	9
Block2	10
Block2	11
Block1	2
Block1	3
Block1	4

...if there's any change (up or both up/down, depending on the .ini setting) the values of both blocks should be written on one line, like this:

Code: Select all

logging of blocks enabled
logging all values
Blk1 Max  512MB
Blk2 Max  256MB

Blk1   Blk2
8      8
3      8
0      8
3      8
7      8
0      8
1      8
1      9
1      10
1      11
2      11
3      11
4      11

Also, a second idea:

I don't know if it's less of a performance hit, but perhaps there could be a setting for a mode where both values are logged every second. The trade off is the cycles spent checking if the usage of either block has changed versus a log file write every second.

Anyhow, I feel pretty strongly that my first suggestion would make the output easier to scan through and find the highest values with a column of numbers for each block.

If not possible, clearly what you've come up will still do the trick.

Thanks again!

[Nearox]

Even though increasing the 1st memory block allocation avoids crashes associated with the commands to automatically increase its size, there may still be other situations where memory address block allocations need to be increased that could cause crashes due to the use of the movaps function on unaligned data.
he countless hours figuring it out in the first place!)

Could that be the reason some other people and myself are still experiencing an occasional crash? I am crashing much less than before but no matter which mod setup I try (using different profiles for STEP, SRLE as well as vanilla + USPs + default INIs) the game still crashes regardless of which ugrids setting is chosen. Seeing as I run a vanilla profile as well, I can basically rule out crashes due to an erroneous script and/or mesh.

[Eu4orick]

I'm naive on too much technical babble here so just some needed advise pretty please?

Which of the mem patches is most recommended until SKSE produce there own? I have one of Uriel24's I believe running my system just fine as far as I can tell, stable ish yet very slow ugrids 15 up to 5750x1080 multi monitor.

Then there is that new Nexus SSME mod just popped up over this morning.

Does it really matter which to use at the moment or is this newer one more stable or anything please?

[Nearox]

For more end-user related stuff you may also want to check the thread over at STEP http://forum.step-project.com/showthread.php?tid=4646

[ApolloBez]

Hey Guys.. Just found this place and signed up thx to people singing praises to "sheson" and this CTD fix, (so hope you forgive my ignorance), but I'm a recent PS Console convert, (in-fact, sold a PS4 prior to X-mas that prompted me to buy a gaming PC while waiting on it's release), "AND" am a bit older, so slower.. lol.

Anyway, I probably got a 100+hrs in the past 2 weeks getting up to speed with all the mod tools and terms a "glutton" like me needs to use all them, (been running max but still not satisfied), and making a lot of progress en-lieu of all the bad/mis-information out there, so I'm REALLY glad to have stumbled onto you guys who very apparently "KNOW WHAT THE HECK YOUR TALKING ABOUT"!!

To it already!!

O.K.

I've followed ZerOxshadows instructions to the letter, down to getting the, " ==Build: 2 succeeded, 1 failed, 0 up-to-date, 0 skipped ==" confirmation. The problem starts when I look for the "skse_steam_loader.dll" in "skse_1_06_16\src\skse\Release". It's not there, it's back in "skse_1_06_16" and un-changed.

So I still replace the ".dll" version in Skyrim's directory, make a "skse.ini" put in the appropriate text with the "memblock" additions, pop it in "Steam\steamapps\common\Skyrim\Data\SKSE", check the log.... and realize I still have to feed my kid!! :?

I've repeated the process several times and even tried the "re-build" opt in the steam loader right-click menu. Also, not that, I guess, this should matter, but the skse files I manipulated are just on my desk-top, (where I put all down loadable stuff to start with).

I believe this fix can help me as I'm now only getting CTD's when opening menu's around major cities, and, (like many folks I'm sure), really wouldn't mind playing for awhile, so any in-site that helps would be worth my other kid as well... lol **NOT KIDDING!!** (they just cause headaches and cost money these days), lol.

Anyway, thx all

[sheson]

Intriguing.
Is that correct?

Yes. But first block is not exclusive to exterior cell buffer.
Skyrim has many different memory block allocations for all kinds of different things. I have yet to observe reproducible trouble with them when they are used/expanded etc - in a game setting that makes sense. What I mean is, of course it will crash at some point with too high uGrids, but I am not going to investigate since there is no point or sensible way to fix it.

I just left the movups it in there for people in the know. I think it is wiser to fix the underlying problem than to fix the movaps themselves. We are extremely lucky this fix is this simply because the whole initial memory management code in the game is stable. If this wouldn't have worked as well as it does we would have to look at the additional allocation routines, but you know I am just as lazy as the next guy. My point is, you don't want to mask a protection fault but rather fix its origin.

Also, my game is stable now. Any other weird thing happening I can now trace down to mods or messing with settings in inis.

Now I know who the guys are to contact directly with any new findings. I had no clue couple weeks ago. Simply documenting and posting all findings in public still seems a good way too.