PNG vs TGA format for alpha to get right specular shading

If I export my outputs from Substance Designer in PNG format with RGBA made with the channel-shuffle filter, in Photoshop it shows only RGB channels but nevertheless PNG specifications say that it might be suitable to get alpha for any purpose. A typical RGBA PNG 1024*1024 texture is 4-500 KB. Small. If I export my outputs from Substance Designer in TGA format with RGBA made with the channel-shuffle filter, in Photoshop it shows to have all four RGBA channels. A typical such 1024*1024 texture is 3-5 MB. Large. To reduce it I have to use Unity import compression, which is not always the best choice. Now, the point is: if I use those two different files in Unity with two equal specular shaders (say bumped specular), the specular result is perfect with TGA, while it's very questionable with PNG. Is there any trick to correct the PNG use for specular? Either I simply must use TGA or any other "full" four channels format?