My man wants us to go to a swingers club!?

i dont mind going as i wouldnt mind seeing wat it is like,problem here is i said i do not want to sleep with any one man or women but i dont mind watching him enjoy himself now he is kinda complaining and ask why not!i told him i dont want to.am i wrong for this?