The mishna tells us that just as there is an issur of ona'ah to cheat someone through money, there is also an issur to cause them pain verbally. The mordechai (306) says that based on the gemara that considers ona'as devarim to be "not returnable", it would follow that one receives malkus for violating ona'as devarim. The obvious difficulty with this is that ona'as devarim is a la'av she'ein bo ma'aseh which one does not receive malkus for violating. The sefer Hachinuch explicitly argues with this mordechai and says that since it is a la'av she'ein bo ma'aseh there isn't any punishment of malkus.
The nimukei yosef explains based on the gemara 59a that only עם שאתך בתורה ובמצוות are inlcuded in ona'as devarim, that the issur of ona'as devarim doesn't apply when one speaks harshly about one who is not a ya'arei shamayim. Although the gemara excludes one who is a rasha from this issur (meaning that it is mutar to offend him), it seems a little strange why the nimukei yosef raises the bar so high, and insists that there isn't a violation to speak against someone who is not a ya'arei shamayim.
We find that the gemara darshens in Baba Metzia 48b and 62a - ונשיא בעמך לא תאור, בעושה מעשה עמך, to the exclusion of a rasha. We also find in the Hagahos Maimon (deios 6:1) on the mitzvah of לא תשנא את אחיך בלבבך וכו' ואהבת לרעך כמוך that we darshen - דוקא שהוא רעך בתורה ובמצות אבל אדם רשע שאינו מקבל תוכחה מצוה לשנאותו. The source that one can hate a rasha is in pesachim 113b, that one can hate someone who does aveiros. We also find in the rambam (rotzeiach end of perek 4) who darshens - לא תעמוד על דם רעך, ואין זה ריעך and learns from here that shepherds who are considered theives are not included in this mitzvah to save them.
All these sources that use the terms "amcha", "rei'acha" or "achicha" seem to exclude only real wicked people. But the nimukei yosef seems to understand that the term "amisecha" in the context of ona'ah excludes anyone who is not a ya'arei shamayim. The term implies a higher standard than the other terms.
For more discussion of the issur on'ah see the link that i posted on the side to an audio shiur that i gave last night.