Low poly artifacts in specular highlight for toon shader

Hey guys! I'm trying to do multi-variant shader with multiple ways of doing a toon shader. I'm going now for a BOTW kind of toon shader and I followed this tutorial, however it only supports directional lights and I would like it to support all kinds of lights. Important to note that the tutorial uses vertex and fragment shaders and I'm using surface shader with a custom lighting function.

As such I started working with the Unity's surface shader with a custom lighting function, but when I tried to add the specular part of the light it seems to snap to geometry of the surface, it kind of gets clipped by the triangles of the mesh I think and I'm not quite sure why. In the gif, you can see what I mean.

My lighting function:

 fixed4 LightingToon(SurfaceOutput s, fixed3 lightDir, half3 viewDir, fixed atten) {
             //First calculate the dot product of the lightDir and the surface normal
             half NdotL = saturate(dot(s.Normal, lightDir));
             float3 halfVector = normalize(lightDir + viewDir);
             half NdotH = saturate(dot(s.Normal, halfVector));
             float lightIntensity;
 
             #if _TOONMODE_RAMP
                 //Remap NdotL to the value on the ramp map
                 half uvNdotL = NdotL * 0.5 + 0.5;
                 lightIntensity = tex2D(_RampTex, fixed2(uvNdotL, 0.5));
             #endif
 
             #if _TOONMODE_SHADINGLVL
                 //Snap the color
                 lightIntensity = floor(NdotL * _CelShadingLevels) / (_CelShadingLevels - 0.5);
             #endif
 
             #if _TOONMODE_TWOTONE
                 lightIntensity = smoothstep(0, 0.01, NdotL);
             #endif
 
             half4 color;
             float specularIntensity = pow(NdotH, _Glossiness);
             float specularIntensitySmooth = smoothstep(0.005, 0.01, specularIntensity);
             float4 specular = specularIntensitySmooth * _SpecularColor;
 
             color.rgb = (s.Albedo * lightIntensity * _LightColor0.rgb + specular) * atten;
             color.a = s.Alpha;
 
             return color;
         }

my surface function:

 void surf (Input IN, inout SurfaceOutput o) {
                 // Albedo comes from a texture tinted by color
                 fixed4 c = tex2D (_MainTex, IN.uv_MainTex) * _Color;
                 o.Albedo = c.rgb;
                 o.Alpha = c.a;
 }

and my pragma declarations:

 #pragma surface surf Toon fullforwardshadows
 #pragma shader_feature _TOONMODE_RAMP _TOONMODE_SHADINGLVL _TOONMODE_TWOTONE

The desired effect is on the right and the and current effect is on the left. Am I missing something in the way the lighting function is converted to vertex / fragment shaders? What is happening that makes the specular highlight low poly in on my sphere but not on the one of the tutorial?