How do you draw a GL quad facing the camera?

So I have an empty object with a script that will draw GL.QUADS in OnRenderObject. I draw multiple quads somewhat spread around the center of the object. So far so good.

What I am struggling is to make the quads face the camera at all time. I cannot just make the base empty object a billboard as the position of the quads in 3d space is important relative to the rest of the world.

I also tried to have a prefab for each quad with their own transform and all. Unfortunately, performance is critical in my project and drawing all the quads in one loop is far more efficient than having a bunch of seperate quads.

So far, the way I do it is I calculate the position of the quads in 3D space, and then draw a quad on the x,y plane around the position. I am assuming I can use matrices in some way to draw on the x,y plane and then rotate that plane to match the one of the camera.

Maybe I am just doing it wrong. I am open to any other way if there are some.