I don't think you can draw a line and say sex offences are special, the criteria should be violence in and of itself. Whether that is a serial killer or someone who gets into a brawl to enforce their will, there is no distinction. I think the real problem is that we as a society DO make a distinction, and see violence as no big deal sometimes, when in fact it reflects an underlying mentality. I wouldn't say 1 strike, but there should be a rapidly escalating judicial response as these incidents re-occurr. And that includes things like DV and fighting during sports events.
Violence is never OK, no matter what the circumstances or reason, and people who make a habit of it, no matter how trivial the offence, need to be removed from society or at least placed under very tight supervision.