Do unused audio sources hurt performance?

To save time, I've added an audio source component (with no AudioClip attached) to a few different prefab that will get used probably 30-50 times in each scene. How can I tell how this affects performance once I start adding the audio clips associated to instances of the prefab?

Would it be more performant to disable them in the prefab and enable them as needed? Not every instance of the prefab will need an AudioClip attached, but most will, which is why I just went ahead and added it to the prefab - it saves a lot of time in setting up my 3D Sound settings curves since they'll all be variations of the custom curve I've set up.