Recursive Raycast Spatial Search

When the player clicks on an object, I raycast to that object, find what it hit, then run a search. This Search function, which is attached to the object itself, runs the following code in several directions around itself:

function Rayaround (dir, color, score) {
    var visitnum = -1;
    var hit : RaycastHit;
    if (Physics.Raycast (transform.position, dir * 0.1, hit)) {
        visitnum = hit.collider.gameObject.GetComponent(Tile).num;
        if (hit.collider.name == color && !index.Contains(visitnum)) {
            index.Add(visitnum, true);
            score++;
            score += hit.collider.GetComponent(Tile).Search(color, score);
            Destroy(hit.collider.gameObject);
        }
    }
    return score;
}

Some clarification: This code is called Tile and attached to the colliders I'm hitting, index is static and shared amongst all Tile objects, and num / visitnum are just unique numbers given to each object when they're first created.

So, the major issue here is that score gets really huge when I run this. Like, two billion huge. Huge enough to overflow a signed integer (great idea for a 'yo mama' joke, btw). There's about a thousand objects here, but this should only be searching through a dozen of those, if not less.

I'm not entirely sure why that number gets so large. The number is not incremented anyplace else. The program takes a noticeably long time to run through this piece of code.

I'm being careful not to delete things that search until only after they've searched, and I've created a hashtable of objects I've visited before so they don't get visited again... It feels like I'm doing all the right things, yet I'm getting nowhere towards solving this problem. Thoughts?

  Debug.Log("score is " + score);