Courts get around difficult to prove things all the time. That's just life, sometimes things are hard to prove. This does not mean we give up and stop trying. Since you're hunan beings that will inevitably err, is it necessary to err on the side of allowance in all these cases? If so, this will prohibit you from enforcing the rule, in which case it should be removed to avoid the creation of false expectations and getting your community pissed off at you for misrepresentation of your intentions.
If you want to try, how about the usage of logical fallacies? It is virtually impossible to effectively troll without utilizing especially strawman arguments, UM did that all the time. Since they are rooted in logic they are reliably identifiable.
I'll also note that part of rule 4 is low-effort comments, that's another reasonable, if subjective, metric. Though I genuinely would simply remove that as a rule, since I'm getting the sense you intended it more as a guideline, as rule 5 seems to be. Perhaps the sidebar could have a "rules" section and a "guidelines" section?
What I'm really curious about now is your guys' vision and goals for this community. Is it a free-wheeling, largely free speech zone where we should have a good time? It is a serious space for serious discussions of serious topics? You understand it cannot be both, each type of content drives the people that like the other one away. It'd be like a restaurant trying to be a posh, upscale place but only selling cheap hotdogs. It won't work well, in any competitive environment that would fail, it has to pick one goal.
Whichever it is, I would recommend you reassess how the rules are structured. The way things are right now, you are creating expectations and they are not being met. This creates a sense of disappointment in the user base, and it can be easily remedied by simply managing customer expectations better. Reformat the rules to represent what you are both capable of and willing to do, and then stick to them. This way people can understand what they are getting when they come here.