How can I make a wheel spin more freely using joints in Unity?

I am making a simulation physics based skateboard game in Unity3D. I want wheels that will roll freely based on the physics applied to them without any animation or drive forces. I achieved this through connecting the wheels to the board with a configurable joint. My wheel colliders are custom compound colliders and not using Unity's wheel colliders as they don't seem to be able to do what I'm looking for.

The problem is that my make shift wheels roll very slow. It behaves as though there is a large amount of friction or some other force preventing the wheels from rolling freely. The wheels actually slow down to a certain speed when rolling down hill which is the opposite behavior I want. I've tried turning the target angular velocity way up but I'm pretty sure this does nothing unless I'm applying drive forces to it, which I'm not. I just want the wheels to spin on their own as fast as they need to.

The only thing that seems to have any effect is weight ratio of the two connected bodies. For example, if I increase the mass of the board in relation to the wheels it will move faster, but the joints also fail to prevent the wheels from moving in unwanted ways. They become loose and jitter whenever any collision occurs even though I have all movement locked except the one axis I want to rotate around.

I've played with pretty much every setting and removing every kind of resistance I can think of on the configurable joint and rigidbody but there must be something I haven't thought of or I'm missing. I can't find much if anything about the friction of joints or reasons why they wouldn't spin with no limits so not sure how to tackle this one. If anyone can think of a possible solution I would appreciate it.