The gemara says that both by Parah Aduma and Egla Arufa, there is a distinction between a yoke and other things. A yoke passuls even when it is "not the time of avoda", whereas other work only passuls "at the time of avoda". The Chazon Ish (E.H. 146) quotes that Rash in Para that the definition of "not the time of avoda" is when the person doesn't want it. Meaning, the requirement that the work must be consented by the owner, only applies to other type of work, but by a yoke it passuls even if it the owner doesn't want it to have been done.
But, the chazon ish points out that rashi implies that even for a yoke we require the owner to consent in order for it to passul. Rashi defines "not at the time of work" to mean that the person is not interested in making it work now, he is just using the animal as a place to put something down that he no longer wants to hold on to. The difference between a yoke and other work is that for a yoke it will be passul even if he is not intending to work with the animal, but it is only where he is happy that the work was done (so that he doesn't have to continue holding the yoke in his own hand).