What is it called when you sleep with a married person?

Adulterer (or adulteress) can be used for anyone who has a sexual relationship with a person who is married to another, but again does not always imply the specific social situational that mistress does.