Because they "nothing" you. You are neither liked, nor disliked. You are merely present.
"You have enemies? Why, it is the story of every man who has done a great deed or created a new idea. It is the cloud which thunders around everything that shines. Fame must have enemies, as light must have gnats. Do not bother yourself about it; disdain. Keep your mind serene as you keep your life clear."