Unity Raycast2D results are not as expected?

I'm currently working on a practice project which involves stacking of blocks. It is sort of similar to the game of stackers (the arcade game), except my version uses free-form movement instead of grid-based movement.

I have 3 prefabs for the blocks: SingleBlock, DoubleBlock, TripleBlock.

I structured each of them like this: the parent is an empty gameobject with my MovementScript that moves them left/right, while the children are the block sprite with a BoxCollider2D.

The MovementScript is attached to the empty game object(the parent) so that the block set moves uniformly left/right, for what it's worth.

For the actual stacking logic, I'm using Raycast2D to detect if there is a block below. But the problem is the results I get is unexpected.

Here is a snippet of my current code:

 foreach(Transform t in currentBlockSet.transform)
             {
                 // get all the children of this blockset. to do this, we use the transform beause it is IEnumerable
                 GameObject block = t.gameObject;
                 RaycastHit2D hit = Physics2D.Raycast(block.transform.position, Vector3.down, rayCastLength); // 0.5f raycast length
                 //Debug.DrawRay(block.transform.position, Vector3.down * rayCastLength);
                 if(hit.collider != null)
                 {
                     // this means there is a block below, we hit something
                     Debug.Log("True");
 
 
                 }
                 else
                 {
                     Debug.Log("False");
                 }
 
             }

This code is called each time the player stops the current blockset that is moving by the way.

The problem is I always get true in my logs even though I purposely didn't align the block set properly. I never get false even if I'm way off with my alignment. Why is this so?

I do like to mention that there's nothing else in the scene. It is just the blocks, so there can't be another object to collide with.

Is there something wrong with the logic or how I'm using Raycast2D?

Appreciate any help.