How to return from Unity Player to previous running Android Activity?

I have a Unity Player embedded in a Native Android Application. I can start the Unity Player without a problem but now I want to go back to my previous activity with the back button. I already tried Application.Quit from Unity side and finish() from android side, both closes the whole application. I also tried to start a new Activity from the Unity context but this doesn't work either, I think this is because I work with fragments. My Activity setup is as follows:

MainActivity where several fragments are embedded. Two of them contain Lists from which I can start a new Activity which shows me detail views of the List elements. The detail views are also fragments in which you can swipe left and right to previous/next element (similar to gmail app). From within the detail view, you can start Unity to have some augmentations with the qualcomm vuforia framework.

up to this point everything works, but I can not go back from the UnityPlayer to the details view.

Does anyone have any idea how to achieve this?

Thanks!