Imported objects don't react to script (default cube does)

Hi y'all,

I encountered a strange problem with imported objects from Blender. I placed the .blend file in the 'assets' folder so unity recognizes them, I put a box collider and a rigidbody on them, but somehow I can't drop the objects after I've picked them up. This only happens with the Blender models, if i try it with a default cube it works perfectly. The rotation and scale values are all set to 0 (rotation) or 1 (scale)

Here's the pick up code: (I work with Google Daydream btw)

 using UnityEngine;
 using UnityEngine.EventSystems;
 using System.Collections;
 
 public class Pickupable : MonoBehaviour, IPointerClickHandler {
     private bool isPickedUp = false;
     private float distanceFromPointer;        
     public float movespeed = 0.02f;          
 
 
     public void OnPointerClick(PointerEventData pointerData) {                
         isPickedUp = !isPickedUp;
         distanceFromPointer = pointerData.pointerCurrentRaycast.distance;    
     }
 
     void Update() {
         if (!isPickedUp) {        
             return;
         }
 
         if (GvrPointerManager.Pointer == null) {    
             return;
         }
 
         Transform pointerTransform = GvrPointerManager.Pointer.PointerTransform;
         Vector3 followPosition = pointerTransform.position + (pointerTransform.forward * distanceFromPointer);
         transform.position = followPosition;
 
         if (GvrController.IsTouching) {                        
             Vector2 touchPos = GvrController.TouchPos;      
             //Debug.Log (GvrController.TouchPos.y);            
 
             if (GvrController.TouchPos.y <= 0.25f) {                            //if the users' thumb is on the upper side of the touchpad            
                 distanceFromPointer = distanceFromPointer + movespeed;        //move the object AWAY from the user
             } else if (GvrController.TouchPos.y > 0.75f) {                    //if the users' thub is on the lower side of the touchpad
                 distanceFromPointer = distanceFromPointer - movespeed;        //move the object TOWARDS the user
             }
         }
     }
 }

Using the trackpad I 'move' the object further or closer to be able to drop and pick up the objects where-ever I want. A strange thing to notice though is sometimes i click the touchpad to pick up an object and if i don't move my thumb and click again it drops perfectly, but if i move my thumb (for example to bring it closer) and press the touchpad again it won't drop.

Anybody have a clue what's going on?

*note, i'm making a medical game so the objects i'm 'picking up' are rather small (10cm - 30cm) (syringes, thermometer etc.)

EDIT; I uploaded the wrong image, the scale and rotation are truly 0 and 1 respectively