How do I fit the bounds of custom gizmos in the Scene View?

I have a script that draws gizmos and it's working just fine. The problem is that in the Scene View, if I select my object and press F to fit it, it does not take into account the additional handles I am drawing. Is there a way I can set the bounds or otherwise get Unity to see the full bounds of the controls I am creating? These objects btw do not have renderers on them.