hate crimes are controversial but they have a clear definition;
"a criminal offense against a person or property motivated in whole or in part by an offender’s bias against a race, religion, disability, sexual orientation, ethnicity, gender, or gender identity.”
the difficulty arises in proving motive, in many crimes it can be hard to clearly distinguish motive, in hate crimes it often becomes very hard if not impossible.
for example - if my neighbor is sleeping with my wife and i decide to kill him, and when i do so i write things at the scene to imply it was a hate based crime, is that a hate based crime? no? or wait, maybe i really did kill him partially because of some hate based reason... maybe if a different person slept with my wife i would not have killed them, or would i?
that is why hate crimes are often not charged when it seems like they should be, and when they are charged they often fail, because it is very complex to prove.
the "element of hate" is not proven by the words said, it is proven by establishing motive.
i dont know many people that would claim that when a hate based crime can be proven that it shouldnt be punished more harshly.