Create hollow sphere with objects bouncing around inside

Sadly It's not possible to make a physics object act as a hollow sphere.

You could make a hollow sphere within your favourite 3D modelling program and split it in multiple parts that have less than 255 polygons each.

I don't know if you need to move/rotate the hollow sphere, but if it's static you could use a bunch of box colliders and align them to represent a sphere. This will probably be faster than convex mesh colliders, at least if you don't need to move/rotate all those box colliders every frame.

Note that the 'walls' of your hollow sphere should have a certain width, to prevent fast moving atoms from penetrating it.

There is an analytic way of calculating a collision between a point and a boundary of the sphere.

Let the center of your sphere be C (Vector3) and let the radius of the sphere be R (float).

A (spherical) atom with position P will be inside the sphere is

 (C - P).sqrMagnitude <= R*R;

An inward facing normal vector at that point on the sphere is

 C - P // No need to normalize because we will be using this to define a plane for reflection

To bounce the atom back once it collides, all you have to do is reflect its velocity using the plane defined by the above normal vector

 rigidbody.velocity = Vector3.Reflect(rigidbody.velocity, C-P);

Here is a pseudocode example that will make atoms bounce inside a sphere (attach to each atom):

 public Vector3 C; // Sphere center
 public float R; // Sphere radius
 public Rigidbody atomRigidbody; // Rigidbody of the atom

 void FixedUpdate() {
     if ((C-P).sqrMagnitude > R * R) {
         atomRigidbody.velocity = Vector3.Reflect(atomRigidbody.velocity, C - P);

         // Put the atom slightly inside the sphere so that it doesn't collide right after
         atomRigidbody.position = C + (P-C).normalized * R * 0.999f;
     }
 }

Note that this method knows nothing about the shape of the atoms, it only cares about the position of their centers.