How to change 4.6 Button Image from script

I'm trying to make a Button with the new Unity UI that can be toggled programmatically between two different images when clicked (it's a zoom in/out button with two different magnify icons).

btw the button background highlights via Sprite Swap; I suspect that is related to some of the issues described below.

UI.Toggle does not work because although it will overlay a graphic when selected it does not toggle between two different images. I.e. instead of turning off image A and showing B, it just adds B. Also Toggle is not a great solution for me because there are other cases where I need to switch between > 2 different button graphics so it does not scale.

Instead I've been trying to make UI.Button do this. My first approach was to replace the Image with a different Sprite:

 // Initialized earlier
 UnityEngine.Sprite zoomInButtonSprite, zoomOutButtonSprite;
 
 // From zoom button OnClick()
 IsZoomedIn = ! IsZoomedIn;
 UnityEngine.UI.Image zoomButtonImage = buttonGameObject.GetComponent<UnityEngine.UI.Image>();
 zoomButtonImage.sprite = IsZoomedIn ? zoomOutButtonSprite : zoomInButtonSprite;

However the code above does not work at all, in fact it has bizarre bugs. The first time the sprite is changed the button background vanishes, then after that I get both sprites overlaid (should be impossible) etc. I suspect because I'm setting the sprite in OnClick there is a weird interaction happening with the button Sprite Swap logic.

Anyone suggestions?