Destroy children in scroll list

Hi, I have a leaderboard using a scroll list for high scores and also a scroll list for invitable friends. This is the layout in the scene.

ListInvite and ListLeaderBoard are the scroll list containers.

Once the Leaderboard and Inviteboard are displayed I use this code to destroy the children that have been created from listItemPrefab and listItemLeaderBoard.

 foreach (Transform child in listLeaderboardContainer.transform) {
                 GameObject.Destroy (child.gameObject);
             }
             foreach (Transform child in listInviteContainer.transform) {
                 GameObject.Destroy (child.gameObject);
             }

But the next time I display the Leaderboard I get this error:

 MissingReferenceException: The object of type 'Image' has been destroyed but you are still trying to access it.

I should also add that when the leaderboard is created it uses this method to create each item.

 //Method to add item to the custom invitable dynamically scrollable list
         void CreateListItemInvite (string id, string fName, string url = "")
         {
             ListItemInvite tempItem = Instantiate (itemInvitePref) as ListItemInvite;
             tempItem.fId = id;
             tempItem.picUrl = url;
             tempItem.txtName.text = fName;
             tempItem.transform.SetParent (listInviteContainer.transform, false);
             listInvites.Add (tempItem);
         }
         //Method to all items to the leaderboard dynamically scrollable list
         void CreateListItemLeaderboard (string id, string fName, string fScore = "")
         {
             ListItemLeaderboard tempItem = Instantiate (itemLeaderPref) as ListItemLeaderboard;
             tempItem.fId = id;
             tempItem.txtName.text = fName;
             tempItem.txtScore.text = fScore;
             tempItem.transform.SetParent (listLeaderboardContainer.transform, false);
             listLeaderboard.Add (tempItem);
         }

So I do not understand why I get the error as the items including the reference to an image are created by prefab before Display.

Maybe another set of eyes can figure out what I am doing wrong here.

John