Porting NLA's enbeffect.fx To New Vegas?

[FiftyTifty]

Hullo lads.

Spent the last half hour cramming in NLA's enbeffect.fx shader into New Vegas, and...It's odd. No errors are thrown, so ENB is quite happy in that regard; there's no red text at the top-left of the screen.

However, the shader window in the ENB GUI doesn't have an ENBEFFECT.FX section. It is missing entirely; it only lists ENBBLOOM.FX, ENBEFFECTPREPASS.FX, ENBSUNSPRITE.FX and ENBLENS.FX.

It works fine when using other ENBs, as well as using the default enbeffect.fx file. Utterly bizarre, I tell ya.


With all of my collected inexperience, the odds are I'm completely wrong, but I've three ideas as to what could be the issue:

1) ENB doesn't like New Vegas using many GUI entries
2) ENB for New Vegas compiles into Shader Model 2.0; it's run out of instructions
3) ENB for New Vegas compiles into Shader Model 2.0; it doesn't support some of the functions used by NLA, that Skyrim's Shader Model 3.0 supports.


Since the shader is pretty long, I'll put it up on Pastebin. I've added #ifdef and #define tags, so you can collapse the GUI entries in Notepad++; makes things much easier to read.

Ported enbeffect.fx: http://pastebin.com/W2wcZ2Mz

Original NLA enbeffect.fx: http://pastebin.com/jfCnyruv


So, yeah. Any idea why this is happening? Bloody bizarre, I tell you.

[mindflux]

There's a mistake somewhere in the GUI definitions.

[ENBSeries]

Shader model 2_0 is not a problem, because ENBSeries use shader model defined by external shaders always, no matter what game tries to do. In current case it's 3_0

Code: Select all

VertexShader  = compile vs_3_0 VS_Quad();
PixelShader  = compile ps_3_0 PS_C1DAE3F7();

As i remember, parameters from annotations do not appear when shader failed to compile at all or variables are not properly declared, which is not your case. Try to remove preprocessor definitions like NLA_TONEMAPPING_CONTROLS at all, may be somehow compiler fails with them. It's a good practice to comment (or "hide" by temporal definition) all the code and start activate parts of it till problem found. You may always disable entire pixel shader just to check if parameters are visible in GUI or they are the culprit.

[FiftyTifty]

There's a mistake somewhere in the GUI definitions.

Turns out I was missing three GUI variables; they were for the adaption shader. No change, though. Had to compare every single damn variable. Took me an hour and a half.

Shader model 2_0 is not a problem, because ENBSeries use shader model defined by external shaders always, no matter what game tries to do. In current case it's 3_0

Code: Select all

VertexShader  = compile vs_3_0 VS_Quad();
PixelShader  = compile ps_3_0 PS_C1DAE3F7();

...

You may always disable entire pixel shader just to check if parameters are visible in GUI or they are the culprit.

Huh. I thought ENB compiled to the game's default shader model. Colour me impressed. Is it possible to compile for shader model 4.0 or 5.0? Or is that D3D 11 only?


When commenting out the pixel shader, is this the proper way to do it? Just to make sure.

Code: Select all

VS_OUTPUT_POST VS_Quad(VS_INPUT_POST IN)
{
	VS_OUTPUT_POST OUT;

	OUT.vpos=float4(IN.pos.x,IN.pos.y,IN.pos.z,1.0);

	OUT.txcoord0.xy=IN.txcoord0.xy;

	return OUT;
}



//falloutnv shader specific externals, do not modify
float4	_c1 : register(c1);
float4	_c2 : register(c2);
float4	_c19 : register(c19);
float4	_c20 : register(c20);
float4	_c22 : register(c22);
/*
float4 PS_C1DAE3F7(VS_OUTPUT_POST IN, float2 vPos : VPOS) : COLOR
{

Code: Select all

#endif //E_CC_PROCEDURAL

	_oC0.w=1.0;
	_oC0.xyz=color.xyz;
	return _oC0;
}
*/ <-- End of Pixel Shader comment


//switch between vanilla and mine post processing
technique Shader_C1DAE3F7 <string UIName="ENBSeries";>

Edit: Commenting out the pixel shader code didn't work. What's odd, is when I pasted over the variables to the default enbeffect.fx, they show up just fine.

Bloody what?

Going to copy over the pixel shader code piece by piece. See how it breaks the file. This is gonna be a while.

[FiftyTifty]

'Scuse the multi-post. Want to keep things easier to quote.

I've found the first culprit in the code. The shader window bugs out because of this line:

Code: Select all

color = pow(color, Gamma);

Now, I've only pasted the first part of the shader code, as I'm doing this step by step. Here's the change in the shader, denoted by //begin Paste and //end Paste

Code: Select all

#if (POSTPROCESS==3)
	
		//begin Paste
	float Gamma=lerp(lerp(GammaNight, GammaDay, ENightDayFactor), GammaInterior, EInteriorFactor);
	float RedFilter=lerp(lerp(RedFilterNight, RedFilterDay, ENightDayFactor), RedFilterInterior, EInteriorFactor);
	float GreenFilter=lerp(lerp(GreenFilterNight, GreenFilterDay, ENightDayFactor), GreenFilterInterior, EInteriorFactor);
	float BlueFilter=lerp(lerp(BlueFilterNight, BlueFilterDay, ENightDayFactor), BlueFilterInterior, EInteriorFactor);
	
	float DesatR=lerp(lerp(DesatRNight, DesatRDay, ENightDayFactor), DesatRInterior, EInteriorFactor);
	float DesatG=lerp(lerp(DesatGNight, DesatGDay, ENightDayFactor), DesatGInterior, EInteriorFactor);
	float DesatB=lerp(lerp(DesatBNight, DesatBDay, ENightDayFactor), DesatBInterior, EInteriorFactor);
	
	float AdaptationMin=lerp(lerp(AdaptationMinNight, AdaptationMinDay, ENightDayFactor), AdaptationMinInterior, EInteriorFactor);
	float AdaptationMax=lerp(lerp(AdaptationMaxNight, AdaptationMaxDay, ENightDayFactor), AdaptationMaxInterior, EInteriorFactor);
	
  float Saturation=lerp(lerp(SaturationNight, SaturationDay, ENightDayFactor), SaturationInterior, EInteriorFactor);
	float ToneMappingCurve=lerp(lerp(ToneMappingCurveNight, ToneMappingCurveDay, ENightDayFactor), ToneMappingCurveInterior, EInteriorFactor);
	float ToneMappingOversaturation=lerp(lerp(ToneMappingOversaturationNight, ToneMappingOversaturationDay, ENightDayFactor), ToneMappingOversaturationInterior, EInteriorFactor);
	float IntensityContrast=lerp(lerp(IntensityContrastNight, IntensityContrastDay, ENightDayFactor), IntensityContrastInterior, EInteriorFactor);
	float Brightness=lerp(lerp(BrightnessNight, BrightnessDay, ENightDayFactor), BrightnessInterior, EInteriorFactor);
	float BrightnessCurve=lerp(lerp(BrightnessCurveNight, BrightnessCurveDay, ENightDayFactor), BrightnessCurveInterior, EInteriorFactor);
	float BrightnessMultiplier=lerp(lerp(BrightnessMultiplierNight, BrightnessMultiplierDay, ENightDayFactor), BrightnessMultiplierInterior, EInteriorFactor);
	float BrightnessToneMappingCurve=lerp(lerp(BrightnessToneMappingCurveNight, BrightnessToneMappingCurveDay, ENightDayFactor), BrightnessToneMappingCurveInterior, EInteriorFactor);
	
	
	float greyscale = dot(color.xyz, float3(0.3, 0.59, 0.11));
	color.r = lerp(greyscale, color.r, DesatR);
	color.g = lerp(greyscale, color.g, DesatG);
	color.b = lerp(greyscale, color.b, DesatB);	
  
	
	//color = pow(color, Gamma);
	
	color.r = pow(color.r, RedFilter);
	color.g = pow(color.g, GreenFilter);
	color.b = pow(color.b, BlueFilter);
	
	
	//end Paste

	grayadaptation=max(grayadaptation, 0.0);
	grayadaptation=min(grayadaptation, 50.0);
	color.xyz=color.xyz/(grayadaptation*EAdaptationMaxV3+EAdaptationMinV3);//*tempF1.x

	float	lumamax=EToneMappingOversaturationV3;
	color.xyz=(color.xyz * (1.0 + color.xyz/lumamax))/(color.xyz + EToneMappingCurveV3);

#endif

With that Gamma line commented, ENB works. Un-comment it, and the shader window disappears again. Going to muck around, see if I can figure out why that line bugs it out.

Got no idea why, but eh.

[roxahris]

Compatible shader model is determined (limited) by the game's DirectX version (not ENBseries) - so, nope, you can't use SM4 or 5 in NV.

[ENBSeries]

Try to set color.xyz=pow(color, Gamma), may be it won't compile because alpha is not set.

[FiftyTifty]

Compatible shader model is determined (limited) by the game's DirectX version (not ENBseries) - so, nope, you can't use SM4 or 5 in NV.

Grah. There goes me sudden "I must be the smartest man in the world!" moment.

Try to set color.xyz=pow(color, Gamma), may be it won't compile because alpha is not set.

God damn Boris, you're smart. Had to delete the word-wall I was going to post, because you figured that out.

My reaction was to do it for each channel, e.g,

Code: Select all

color.r = pow(color, gamma),

which cocked up the colours because I didn't specify the channel in

Code: Select all

(color, gamma).

Alright. Time to continue on with this beast.


Edit: That seems to have done the trick. Blooded schoolbuses and disabled cicadas, that was the only problem. Thanks a bunch lads.

Now I just gotta re-fuckulate the settings. Even with just changing the shader's brightness setting to 1.00, the game looks way better than vanilla.

[FiftyTifty]

New post, new problem.

It turns out, that my ported enbeffect.fx somehow breaks Ambient Occlusion and Detailed Shadows; whether they are enabled or not, they have no effect. As in, they might as well not exist.

So for the past hour and a half, I've been buggering around with presets, uninstalling drivers, installing new ones, etc. And somehow, it's due to my changes to enbeffect.fx; using the default enb files, the effects work fine and as intended. With the ported shader? Nada.

Here's the pastebin of the shader. http://pastebin.com/uTSZ6jxd

Edit: I forgot to mention I included prod80's depth bloom enbbloom.fx. Only involved a couple lines of code in enbeffect.fx, and it's not the source of the problem (reverted to default bloom.fx and bloom code in enbeffect.fx, didn't fix the issue). 'Ere it is. http://pastebin.com/E97Gnt1v

Guess I'll have to comment out the code, bit by bit, again. Ergh. Any of you lads got any ideas on how such a thing could happen?


Edit2: Urgh. Brain hurt.

Commented out all the pixel shader code. Does nothing.
Commented out all the pixel shader code & commented out all the added GUI controls. AO and DS works, but ghosts a previous frame over the screen.

Yeah, the despair's setting in; I've nae clue how to approach this.


Edit3: Oh what the hell. Commented out the default POSTPROCESS GUI variables, sans POSTPROCESS3's, and now the effects work.


Edit4: Restarted New Vegas, due to the ghosting effect appearing again (figured it was due to constant toggling on/off AO and DS, then clicking Apply Changes*, and now it isn't working.

ashahdfhjfdh

This is so damn frustrating. I'll zip and attach me current progress; I'm going to take a break from getting this baby to work, me nerves can't take much more.

[ENBSeries]

that's why i rarely touch code made by someone else.
Well, to break internal effects you "need" to set device render state which is not handled inside the mod. Handled are culling, aplha blending, z-buffer toggling and few more (i don't see that in the shaders you provided). Another case is for shaders which don't have internal backup or their defaults by the game, so applying not working shader simply does nothing and you see render target of some previously drawed shader, internal or external. It could be compiler bug too, because i frequently having issues, it may fail to compile simplest non buggy things or generate wrong code, that's why i'm trying to develop code while game is running and test any progress. Try to replace files with original versions to find which produce issue and then again, disable the code of it partially.