Line of Sight and Raycasting

I'm working on a small tabletop strategy game port. For it to work properly a couple of things are afforded as liberties:
1. The 'units' are cylinders, so line of sight to and from them can be taken from any point in the cylinder.
2. The accuracy is irrelevant, since it will be calculated with a percentage or roll based on the unit's stats.

My current problem is raycasting, I'm testing to see if I can use it as a method for determining Line of Sight, but my current code is behaving oddly, so I can't test it properly. I might be missing something basic.
My problems are:
1. Raycasting from one unit to another always turns up as failing (false), but when I test the raycast against a basic object like a cube, sphere, or even a plain cylinder, it works mostly fine. 2. When I raycast through terrain, the ray registers a hit, and ignores the terrain completely. (same as image 2, but through a wall)

Is this the best way of doing line-of-sight tracking for this type of game or do I have other choices?

Is something wrong with this code?`if (Input.GetKey (KeyCode.F)) {

         RaycastHit hit;

         if(Physics.Raycast(transform.position, transform.position - target.transform.position, out hit, 100)){

             Debug.Log("Hit:" + hit.transform.gameObject.tag);
             Debug.DrawRay(transform.position, (target.transform.position - transform.position) * 100, Color.green);
             Debug.Log(hit);
         }else{
             Debug.DrawRay(transform.position, (target.transform.position - transform.position) * 100, Color.red);
         }
     }`