Why would Baked GI decrease my performance?

I am building a mobile game with a fairly small level and a single directional light to represent the sun. I've just started to learn about GI and have realised that despite having both the Baked GI and Precomputed GI checkboxes in the Lighting window checked it probably wasn't doing anything because nothing in my game was marked with static lightmaps. In spite of this my game has been running at a solid 60FPS and looking decent enough with just the realtime lighting, although there was some noticeable shadow pop in.

Anyway I switched all my level geometry to static lightmaps and checked Baked GI only and ran a bake. It took about 30 minutes and says it has generated 45 directional lightmaps at 1024x1024px and 90.0MB.

Running it on my mobile and my performance has dropped significantly to around 25-30FPS. For the most part shadows do look better (there is no pop in) but there are also some splotchy yellow artifacts and some noticeable lines between connected pieces of floor geometry. Also the shadows from my dynamic player object appears way out of proportion to the rest of the level shadows because they are so much darker than anything else.

Everything I have read says that Baked GI is the way to go for mobile especially so am I just configuring something wrong? My guess would be the increased texture memory from holding the lightmaps might be the cause of the problem but I'm not really sure how to verify this? My build size has only increased a little - from 35MB up to 42MB so not sure where the 90MB of lightmaps have gone?

Here are my settings: