NavMeshAgent and NavMeshObstacle together, unit jumping a small distance on toggle

I am making an RTS style game where I use navmeshagents and navmeshobstacles on the same unit. I do this in order to cause units to walk around attacking units. It all works as expected, except for when I turn the obstacle off and the agent on, it causes the agent to jump a small distance to the side. Pictures explaining what I mean: This happens in the same frame that the obstacle and agent are toggled, in the first picture, the unit is attacking and is an obstacle, then the second frame is when the unit is ordered, the agent jumps to the side. I am guessing this is caused by the navmesh not being "uncarved" when the agent is active. How could I solve this?