Setting transform.position makes object appear in wrong place sometimes. Possible conflict with colliders/rigidbody?

Hi all, been trying to findwhat is going on here but no luck so far, and searching through a lot of Answers/forums. Hopefully someone can shed some light.

I'm making a simulator where AI Agents wander around a landscape, picking up Food objects (just spheres with a trigger and script attached, spawned at random locations on the navmesh). The Agents also have the ability to 'throw' Food objects to each other (i.e. they Spawn a Food object at their location, and add velocity to it towards it's recipient.)

My problem: sometimes, (like maybe once every 30ish times) the Food object they 'throw' does not Spawn in the agent, instead it appears in a random location on the navmesh, with seemingly the correct velocity.

The Food objects are instantiated and pooled at the Start of the simulation, and enabled as needed by a Food Manager, like so:

FoodManager.cs:

     for(int i = 0; i < foodPool; i++)
     {
         GameObject food = Instantiate(prefabAgent);
         food.transform.parent = gameObject.transform; 
         foodScript = food.GetComponent<Food>();
         foodScript.foodManager = this;
         food.SetActive(false);
         foodList.Add(food);
     }

And spawned like so (also in FoodManager.cs):

 public GameObject spawnFood(Vector3 position)
 {
     //use first inactive object in pool
     for (int i = 0; i < foodList.Count; i++)
     {
         if (!foodList[i].activeInHierarchy)
         {
             foodList[i].transform.position = position;
             foodList[i].SetActive(true);
             spawnedFoodObjects.Add(foodList[i]);
             allFood.Add(foodList[i]);
             //break;
             return foodList[i];
         }
     }
     Debug.Log("No inactive food found in pool, FoodManager.spawnFood() returning null");
     return null;

 }

The agents are Tossing the food with this method (most of this is setting up the values in the Food script, the relevant call is at the start. The added Vector is just to make the food 'float' a little off the terrain, to match the randomly distributed food.) In StatePatternAgent.cs:

 public GameObject TossFood(GameObject recipient)
 {
     GameObject foodObject = foodManager.spawnFood(transform.position + new Vector3(0,0.3f,0));
     //Collider[] collidersAtSpawn = Physics.OverlapSphere(position, 1);

         Food foodScript = foodObject.GetComponent<Food>();
     foodScript.value = Mathf.Clamp((hunger - stash)/(100 / (altruism+1)), 5, 25); //placeholder to avoid divide by 0, change
     foodScript.recipient = recipient.GetComponent<StatePatternAgent>();
     if (foodScript.value > (hunger - foodScript.recipient.hunger) / 2) //cap the value to be maximum the difference between the 2 agents
         foodScript.value = (hunger - foodScript.value) / 2;        
     foodScript.tosser = this;
     foodScript.tosserName = agentName;

     Vector3 recipientLocation = recipient.transform.position;
     Vector3 direction = recipientLocation - this.transform.position;
     foodObject.GetComponent<Rigidbody>().velocity = (direction + recipient.GetComponent<NavMeshAgent>().velocity) * 2;
     recipient.GetComponent<StatePatternAgent>().wanderState.chosenFood = foodObject; //try make the recieving Agent go get this one
     foodManager.allFood.Add(foodObject);
     return foodObject;
 }

(I can't seem to upload images from inspector right now like I would want, i'll try again later.) The Food object is a simple sphere, with sphere collider and rigidbody attached. The Agent has a sphere collider, rigidbody and NavMesh Agent.

I am sure this problem is somehow creeping in in the Vector3 paramter passed to the spawnFood method in FoodManager. Thigs I have tried so far: setting all the prefab transforms to 0,0,0 and resetting their scales to 1. Tried not parenting the Food objects to the FoodManager transform.

Also tried Spawning the Food objects outside the Agent directly, as I heard that trying to move objects inside another collider could have this problem. This did not seem to work, but maybe a more complex check to spawn the Food somewhere not occupied by anything else? What confuses me though is if this is the issue, it should happen ALL the time, as at the moment the Food object is being moved into the throwing Agent , enabled, and then having velocity added.

Can anybody tell me what is happening here? If I bite the bullet and add a fairly complex check to make sure the Fod is enabled somewhere unoccupied nearby the throwing Agent, will this solve my woes, or is something else going on here?

Thanks for any help and please let me know if I can clarify anything, I'm quite new to this and I'm sure my code/explanation leaves a lot to be desired.