CombineMeshes unexplainable draw calls

I have a scene with a bunch of cubes and every cube has a collider on each (since I need to know which tile is being collided with).

Obviously I get insane amounts of draw calls. So I do the following:

1. Find all cubes that share textures.

2. Combine these meshes with Mesh.CombineMeshes (have tried StaticBatchingUtility.Combine as well) into a new mesh.

3. Remove the Mesh from the MeshFilters of the cubes and the MeshRenderer as well. I do this because I want to keep the MeshColliders for the individual cubes, so I don't want to actually Destroy the objects.

The problem is that when I run this the new mesh works perfectly, it is all in one draw call and if I toggle the GameObject the draw calls are reduced by 1. The problem however is with the GameObjects which now only have Colliders (the ones I removed the Mesh etc from) on them, which represent the cube collisions. Each of these GameObjects are still causing draw calls for some unexplained reason. The only way I can reduce the draw calls is by disabling the entire GameObject but then the Colliders won't work. So I'm a bit stumped.

Any advice would be appreciated.