2D Raycasts Only Work in Certain Directions

I am trying to make a script where a 1x1 tile sends out Raycasts in the 4 cardinal directions to detect whether or not it is next to any other of these 1x1 tiles. I am performing this using the following function:

    private bool FindTiles(Vector2 direction)
         {
             RaycastHit2D hit = Physics2D.Raycast(transform.position, direction, tileLength,
                 1 << LayerMask.NameToLayer("Ground"));
             bool result = hit.collider != null ? true : false;
             return result;
         }

I am then calling this function 4 times, once for each direction:

 top = FindTiles(Vector2.up);
 bottom = FindTiles(Vector2.down);
 left = FindTiles(Vector2.left);
 right = FindTiles(Vector2.right);

However, the function only ever returns true when the Raycasts are sent down or left (both negative directions). The function never returns true with the other 2 directions, even when there are tiles above or to the right of an individual tile.

I have no idea why this is the case. When I used Debug.DrawLine, I was able to see the lines in all 4 directions. I also have "Queries Start in Colliders" unchecked in the Physics2D settings, so the fact that the Raycasts start in the center of each tile shouldn't be an issue.