I found that the ketzos hachoshen (182:2) cites this nodeh b'yehuda, and explains that it depends. If the husband appointed the shliach to divorce her against her will, then he is a shliach for a d'var aveira and the consequence is as the nodeh b'yehuda says. But if the husband appointed the shliach without any mention of divorcing her against her will, and the shliach decided on his own to divorce her against her will, the aveira that is being done is incidental to the shlichus so that the gett is still binding. It is the equivalent of one appointing a shliach to do an act b'heter, and the shliach on his own decides to do it b'issur, the shlichus is binding.
It seems clear that the ketzos would agree that if a kohen made someone a shliach to marry a woman for him without specifying who, and the shliach on his own chose a gerusha, that would qualify as "ein shliach l'dvar aveira" just as if the kohen would have explicitly told him to be mekadesh a gerusha. We would not say in this case that the aveira of the shliach was done on his own and is incidental to the shlichus. The distinction is pashut - since there is no heter way for the kohen to be married to this gerusha, the issur is inherent to the shlichus and therefore we say ein shliach l'dvar aveira. But in the case of violating the cherem, there is a heter way for the husband to be divorced from this woman without violating the cherem, therefore even if the shliach violates it, so long as that wasn't part of what he was appointed to do, the act would be binding.
2. The ketzos (105:1) explains that ein shliach l'dvar aveira only applies when there is an issur on the sender, but if it is permitted for the sender and only an issur on the shliach, the he can be a shliach. Based on this he disagrees with the Pnei Yehoshua in Kesubos who says that one who tries to be zocheh for a ba'al chov, when it is a detriment to others, doesn't work because it is a shliach l'dvar aveira. The Ketzos argues that since there is no issur on the person the shliach is being zocheh for i.e. mishalei'ach, he would not be considered a shliach for an aveira.