Troublesome collision detection when changing rigidbody.velocity directly

Setup:

Car is a plane with a rigidbody and box collider.

Pickup item is a capsule with a capsule trigger collider

The car has a script which adds relative force (using Forcemode.acceleration) and then runs the code below in FixedUpdate() to kill the sideways velocity if the car is not drifting.

Issue:

Code works fine, except for this annoying slightly rare bug where if you head towards a pickup without drifting you will pickup an item 1 unit before you hit it and sometimes you'll pick up everything in a line in front of you, even if it's far away.

I believe it is the following code that creates the issue, because I could imagine that the physics engine does some magic regarding adding forces before it gets to kill the velocity.

 Vector3 localVelocity = transform.InverseTransformDirection(rigidbody.velocity);
 localVelocity = new Vector3(localVelocity.x * drift, 0, localVelocity.z);
 rigidbody.velocity = transform.TransformDirection(localVelocity);

I've totally run out of ideas, I do not have a clue how to fix this. Please, if you are willing and able, shine some light upon the issue could you? :)

I'm sorry if my explanation is difficult to understand, if it's not understandable in the slightest, please tell me and I'll attempt to reword it.

Thanks in advance.