Building a prism (adding thickness) from non-convex mesh

I have an array of Vector2 points that defina an edge of a polygon. I would like to generate a mesh collider in a shape of thah polygon at runtime. For that I am using Unity Triangulator. It works fine, represents the desired shape accurately, but the mesh I get is flat. Therefore the collider based on that mesh is non-convex.

I want to use this collider as a trigger to detect if anything enters this zone. To do that I need the collider to be set as Convex. However, when I set a Convex checkbox to true, I get a box collider as a result:

What I would like to achieve is to have the same shape as the non-convex collider, but with non-zero thickness in Y axis.

How do I approach this?