My girlfriend was a swinger, should this bother me?

I love my girl and she means the world to me... We live together and she has been very open with me in regards to her past swinger lifestyle with her ex boyfriend. Have been with her for just over a year now. Here's the problem. I try to not dwell upon her having been in my opinion treated like a sex slave by her ex boyfriend who she would swing with and not to mention that he would find men to come to their house and have sex with her as he watched. I cant hold her past against her and tell myself that her past experiences are why she is magnificent when we make love. Her past is her past and made her to be the woman that I fell in love with. However, just about every other weekend she has to meet her ex to drop off and pick up her two kids for visitation. I cant help but wonder and at times even become enraged thinking to myself what she may be up too during this pick up/drop off. It makes me wonder if she will be faithful to me having been in the swinger game. She assures me that I am the only one for her and that her past is her past. Even wants me to drive with her for the pick up/drop off but I refuse because I doubt I will be able to control myself if I ever get the opportunity to meet her ex face to face due to the way he treated her. Anyone else ever have this problem?