What are some practical ways performance-wise to get a set of vertices in an area?

Well "practical" always depends. You haven't given any information on what is your input information and you also didn't clarify a lot of things.

Some of those things are:

• is that "area" specifed in worldspace or in screen space?

• if it's in worldspace, do you think of a selection volume (like a sphere / max distance around / from a point)? Or do you think about a max distance across the possibly curved surface?

• Does your selection start at one of the vertices or do you want to freely select any point on the surface of the mesh?

• If it's supposed to be a selection volume, are you only interested in those vertices that "face" a certain direction? At least in your image you don't seem to be interested in those on the backside of your sphere that are also inside the "area".

• What aboug potential gaps in the mesh? Can the selection jump over holes? Think about selecting the top of a "U" shape. Do you want to select some vertices from the left and some from the right side, even they are not connected directly?

Apart from all of this, what you need in most cases (especially if the mesh has many vertices) is neighboring information. So an efficient way to query which vertices are next to others. This can be precalculated and saved alongside the mesh. The only alternative would be to iterate through all vertices all the time to find the ones that are close to your desired start.

To determine which vertices are connected to which others you can just use the list of triangles. So for each vertex index you can essentially remember a list of indices that are directly connected to the one in question. Keep in mind that meshes can have UV seams or generally duplicated vertices (like a cube in order to specify a normal for each face). So when creating the neigboring information you probably want to account for duplicate vertices (i.e. vertices that have the same position but differ in other vertex attributes such as normal, uv, tangent or vertex color).

Since the question is extremely vague I won't go into any more detail here as this would be a waste of time.