3rd Person Camera: Keeping it out of objects etc (example)?

Hello,

I'm working on a 3rd person top-down(ish) adventure. The camera looks down at the player, but from time to time goes behind or into other objects like walls or houses. I'd like to avoid this, so I run a line cast from the camera to the player and if something is blocking, the camera moves it's offset (it can rotate freely around the player).

However this doesn't work when the camera ends up inside a house or other object, as the image attached shows.

Does anyone know of another way to ensure that the camera doesn't go inside houses or other position where the player would be hidden?

[Top shows the scene with the yellow line from the camera to the player. Bottom is the game view, and as you can see the camera is in the house.]

Thanks!!