Do you mean the jagged edges that makes up the hair shadow on the character face?

Well increase the quality settings in these effects inside enbseries.ini

[SHADOW]

ShadowCastersFix=true
UseBilateralShadowFilter=true
ShadowQualityFix=true
DetailedShadowQuality=-1 ----
ShadowFilterQuality=0 ----
ShadowBlurRange=0.65 - This is my setting and I use this with a iBlurDeferredShadowMask=1 located in the skyrimprefs.ini. This only blurs the shadows nothing else.


If you are using SSAO.

[SSAO_SSIL]

SamplingQuality=-1
SamplingPrecision=-1
SizeScale=0.7
SourceTexturesScale=0.7
FilterQuality=-1

Change Scale values to whatever your PC can handle as they are the big FPS hitters that and SamplingRange.

Also in the skyrimprefs.ini and skyrim.ini change the shadow resolution to whatever you're PC can handle I have all my resolutions to 4096

skyrimprefs.ini:

[Display]
iShadowMapResolutionSecondary=4096
iShadowMapResolutionPrimary=4096
iShadowMapResolution=4096

skyrim.ini:

[Display]
iShadowMapResolutionPrimary=4096

If the list of changes done in the skyrimprefs.ini are accurate you missed one resolution command to change

iShadowMapResolution=4096.

As Mindflux said it has nothing to do with AA it is all in your shadow settings both in the skyrim .ini files and enbseries.ini file.

The issue has nothing to do with AA.

You sure you're editing correct SkyrimPrefs.ini, do the changes make any difference? As a test, try increasing iBlurDeferredShadowMask past 3, the issue should definitely vanish.