Raycast to Detect Collider Issue

I am working on a prototype as I'm learning Unity that has focus on the player being able to grab objects and move them around the game space. To detect player input, I am using a raycast, which was a method I read about online. However, I am running into an issue where the raycast isn't hitting the collider in the right place for one game object, and I'm not sure why.

Expected Behavior: The player moves their mouse along the game world and the raycast detects when a movable object is moved over. This is done by converting the Input.mousePosition to a ray and then getting Hitinfo for the ray to check which collider was hit.

Observed Behavior: The required mouse location for the GameObject's collider varies depending on its location in screenspace (usually requiring a significant difference). But this isn't consistent with all objects, in fact it only affects the one object I want the player to be able to move.

The yellow/red line in the image is a debugger I made to see the actual ray being generated and when it was detecting a hit. Yellow is the ray and the red line is the line from origin to hitpoint. If I hover over the red marble (which can move) or the two other objects on the scene, the hit corresponds to the collider correctly. However, the blue ramp piece (currently selected in the above image) requires the mouse to be considerably lower to actually grab the collider (as shown on the right).

     private void Update()
     {
         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
         Debug.DrawRay(ray.origin, ray.direction * 25, Color.yellow);
         Debug.Log(ray.origin);
         RaycastHit hit;
 
         if (Physics.Raycast(ray, out hit)) { Debug.DrawLine(ray.origin, hit.point, Color.red); }
     }
 
     private bool IsMouseOver()
     {
         Ray ray = Camera.main.ScreenPointToRay(Input.mousePosition);
         RaycastHit hit;
 
         if (Physics.Raycast(ray, out hit))
         {
             if (hit.collider == gameObject.GetComponent<Collider>())
             {
                 return true;
             }
             else { return false; }
         }
         else { return false; }
     }

IsMouseOver() is called during LateUpdate if certain conditions are met. I know MouseOver() is a function I could use, but when I was trying to grab the object through transform.position, it wasn't working. This script is part of an "Object Controller" script that will be assigned to any object the player can move (i.e. not a level obstacle which will be static).

If you have a different suggestion on how to accomplish the result I'm going for, I would appreciate the tips. However, I would also like to understand why this isn't behaving the way I expect it to.