The Shulchan Aruch (even ha'ezer 20:1) writes that one who does chibuk v'nishuk with any of the arayos and enjoys the kiruv bassar, receives malkus. This is the language of the Rambam who holds that the issur to touch any of the arayos is an issur d'oraysa provided that there is hana'ah from the kiruv ba'sar. The Beish Shmuel (1) says that the Ramban holds that it is only an issur d'rabonon. The Beis Shmuel continues that the Ramaban writes in a teshuva that even when the touching isn't derech chiba (which seems to mean that there is not hana'ah of kiruv bassar), the same machlokes would apply - Rambam would hold it's d'oraysa and ramban will hold that it is d'rabonon. Although the Shach (y.d. 195) holds that even according to the Rambam the issur d'oraysa is only derech chiba, the Beis Shmuel disagrees and holds that according to the Rambam there is an issur d'oraysa even not derech chiba. The achronim (sidrei tahara, kreisi u'pleisi) reject the Beis Shmuel and hold that even according to the Rambam it is only an issur torah when done derech chiba. R. Moshe (Igros Moshe 1:56) mentions that some are lenient to shake hands with women since it is assumed not to be derech chiba, based on the shach who understands that even according to the Rambam it would only be an issur d'rabbonon and perhaps mutar to avoid embarrassment. However, R. Moshe falls short of actually being matir shaking hands with women.
Tosafos quotes a Yerushalmi in the context of the waiving that implies that there is not even a real issur d'rabonon when one touches an ervah in a non-chiba way. Tosafos holds that the standard method of tenufa with the kohein placing his hands "under" the hands of the owner, doesn't necessitate touching because the owner holds the upper part of the vessel and the kohein holds it from the bottom. Tosafos then cites a yerushalmi that implies that the kohein literally puts his hand underneath the hand of the owner. In the context of Minchas Sotah the yerushalmi asks how the korban mincha is waved - ואין הדבר מכוער - isn't it repulsive? The yerushalmi answers that it is done using a cloth, which is rejected because it would create a chatzitza. The yerushalmi suggests that it is done by an older kohein who doesn't have a strong yetzer hora, and then concludes that even if done by a young kohein it is permitted since there is no yetzer horah at that time. Now, if we assume like the beis shmuel's understanding of the Rambam that touching even without hana'ah (not derech chiba) would be an issur d'oraysa, why does the yerushalmi ask "isn't it repulsive?", it should ask using a stronger language, isn't it an issur d'oraysa. Furthermore, how can we permit an older kohein to do it? Where do we find that an older kohein who doesn't have a yetzer horah has less of an issur d'oraysa. It seems clear that there is no issur d'oraysa, and seems that there isn't even a real issur d'rabonon, just that it is repulsive to do an act in the beis hamikdash that would be an issur if one had intent for pleasure.
However, according to Tosafos conclusion that perhaps the Yerushalmi is only addressing inadvertent touching because the kohein is not deliberately touching her (he is merely putting his hands under the vessel and will sometimes accidentally touch her), there would be no proof from the yerushalmi. Perhaps the yerushalmi is only saying that inadvertent touching would be מכוער, but intentional touching even not derech chiba could be an issur d'oraysa.