Layering multiple Normal maps into one without blending in a Surface shader script

I'm remaking an older Unity project right now and I'm rewriting a shader I made for it, the main point of which being that it combines multiple textures into one and applies them to the Albedo layer. This time around I'm trying to have the same system apply to the Normal/Bump map as well, but it seems like I'm doing something wrong every time I try to make it work.

My "layers" consist of a regular foundation normal and a couple more images with transparency that are meant to go over the bottom layer and overlap each other. To my understanding, the combined texture should be in a format, suitable to use with UnpackNormal and then to be applied to the material.

I'm fairly new to shader scripting and I'm not sure how normal maps are read by Unity exactly. I've tried several ways of making it work, including different ways of blending, but I couldn't figure out how to make them work with more than two "layers" and with overlapping. I tried using the same method I used for layering the textures, but the type required for a normal map by the UnpackNormal function seems to be one with 4 parameters, rather than one with 3, like the ones I used in this.

I'm going to include an example of my surf function in the aforementioned state just for reference on how I've managed the texture layering and generally just for reference I suppose:

         float4 baseTex = tex2D(_BaseTex, IN.uv_BaseTex);
         float4 eyeTex = tex2D(_EyeTex, IN.uv_BaseTex);
         float4 accentTex = tex2D(_AccentTex, IN.uv_BaseTex);

         float4 normal = tex2D(_NormalMap, IN.uv_NormalMap);

         float4 baseNormal = tex2D(_BaseNormal, IN.uv_NormalMap);
         float4 eyeNormal = tex2D(_EyeNormal, IN.uv_NormalMap);
         float4 accentNormal = tex2D(_AccentNormal, IN.uv_NormalMap);

         half3 baseTexVisible = baseTex.rgb * (1 - eyeTex.a) * (1 - accentTex.a) * _BaseColor;
         half3 eyeTexVisible = eyeTex.rgb * eyeTex.a * _EyeColor;
         half3 accentTexVisible = accentTex.rgb * accentTex.a * _AccentColor;

         half3 normalVisible = normal.rgb * (1 - baseNormal.a) * (1 - eyeNormal.a) * (1 - accentNormal.a);
         half3 baseNormalVisible = baseNormal.rgb  * (1 - eyeNormal.a) * (1 - accentNormal.a);
         half3 eyeNormalVisible = eyeNormal.rgb * eyeNormal.a;
         half3 accentNormalVisible = accentNormal.rgb * accentNormal.a;

         float3 finalNormal = normalVisible + baseNormalVisible + eyeNormalVisible + accentNormalVisible;

         float3 finalTexture = baseTexVisible + eyeTexVisible + accentTexVisible;

         fixed4 c = tex2D(_BaseTex, IN.uv_BaseTex);
         o.Albedo = finalTexture;
         o.Metallic = _Metallic;
         o.Smoothness = _Glossiness;
         o.Alpha = c.a;

         o.Normal = UnpackNormal(float4(finalNormal.r,finalNormal.g,finalNormal.b,baseNormal.r * baseNormal.g * baseNormal.b));
         o.Normal.y = o.Normal.y * _NormalLevel;
         //o.Normal.z = finalNormal.b;

Any help would be greatly appreciated.