[FO4] Replicating retro shader?

[FiftyTifty]

Ages ago, a user on Reddit made this image by reducing it's palette to 16 colours, and then downscaling it:



Now that looks kickass, almost like an old-timey DOS game pre-render.

How would I go about creating this effect? I assume sort of LUT in enbeffect.fx, then downscaling in enbeffectpostpass.fx. But how to do that, I have no idea.

[ENBSeries]

There is dos game shader for Deus Ex HR, you can use it for basis
http://enbdev.com/effect_en.htm

[FiftyTifty]

I took a look at the postpass .fx file, and it's a bit too complex for me; I'm using KingEric's barebones one for the SMAA effect (viewtopic.php?f=7&t=4721), so I'm not sure how to add it without breaking everything.

Is it possible to use ENBEffect.fx for this shader?

[ENBSeries]

Everything is possible, but i'm busy with Fallout 4 modding now to make anythings like this inside non mine code (i don't like to touch code of other people).

[Marty McFly]

I can do that for you, if you give me a little time, not at home right now. You're using the standard SMAA postpass.fx from kingeric with no other changes?

[FiftyTifty]

Aye, I am. I would like to learn how to do it myself, but I'd need an example to show how it's done. No idea what I was doing when I tried to do it.

[Marty McFly]

There's more to the trick than just downscaling and degrading color range. The image looks as if the author uses a custom palette and also dithered the image with bayer I guess to reduce the banding. I can set you something up rather quickly, but as I don't like retro effects at all, you have to tell me if it does what you expect because to me, it kinda looks all same.

EDIT: Ok, so I found the respective reddit thread. Seems like the guy did this:

Stuck the image in Aseprite, then indexed the pallete to about 16 colors. After that I just lowered the size of the image and voila, 'pixel art', even if it looks a bit processed.

So it's not easy at all to replicate this. Maybe if you'd go the LUT way and do the same but then you'd need to do exactly what he did. I just semi-ported Boris' retro fx.

https://pastebin.com/LSGieS2g

use with the rest of kingeric's smaa files. I cannot test atm how it looks ingame as I don't own F4 and my SSE crashes with ENB when the game has finished loading - which makes absolutely NO sense but oh well..

[FiftyTifty]

Unfortunately, it doesn't run:

It thinks that there are SMAA variables being used elsewhere, and there is no Sampler0 variable.

[Marty McFly]

WTF I swear I fixed that. There's also another bug I definitely fixed as it definitely ran. Seems like I didn't copy the file properly and had an old version in the clipboard... Oh well. You can get the sampler0 from the standard file, just put it anywhere above the pixel shader.

And in the pixel shader itself, replace origcolor with color.

And replace the line that declares texcoord with

float2 texcoord = floor(IN.txcoord0.xy * screenresRescaled) / screenresRescaled;

No clue why I sent you the wrong one as I had it definitely running.. The warnings you see there are always there but ENB suppresses them, as long as there is no error that aborts compilation, so I assume the original SMAA effect is buggy but not buggy enough to break it.

[FiftyTifty]

Oh man, it looks so good. Now I need to figure out how to make LUTs, to get the colours right.

All I could find was this super expensive software that generates it from two source images. Is there a more, eh, inexpensive way of making them?