Some people aren’t going to like you. Most times, cowards are content to gossip and quietly disparage your name in your absence. Word may or may not get back to you about this.
If you have a high profile, people may feel bold enough to confront you directly (or anonymously, if you “live” online). The purpose of the confrontation is to get a reaction, and it only feeds the beast to even acknowledge the attempts.
It seems a bit foolish to gratify those seeking to upset you by acknowledging their efforts, so I’d avoid feeding into it altogether. If this means that you have to keep a poker face until a kickboxing class where you blow off steam, then so be it.