The gemara concludes that when one is already wearing bigdei kehuna to do the avoda, they can continue to wear it even after the time of the avoda - מפני שבגדי כהונה נתנו ליהנות בהן. But, the braisa says that the kohein wearing bigdei kehuna must remain in the beis hamikdash and not go out to the "medinah". Rashi explians that it would be a problem of treating the bigdei kehuna as non-kodesh garments. The language of Rashi implies that the nature of this prohibition would only be d'rabonon. Tosafos points out that just as one cannot go out l'mdinah wearing the bigdei kehuna, it would also be forbidden to put on the bigdei kehuna when one is not putting them on for the purpose of doing the avoda. The leniency is only to continue wearing them once they are on, but not to actually put them on. Tosafos understands this from the gemara in kiddushin which justifies the practice of wearing them after the time of the avoda as לא ניתנה תורה למלאכי השרת which implies that since it is unavoidable, it is permitted, but to put them on is certainly avoidable and not permitted (According to Tosafos perhaps one cannot wear them for much time after the avodah either because that is also easily avoidable).
The Mishna gives an option of reading the parsha either in the bigdei lavan that the kohein gadol was wearing until now, or to change into special clothing that was made for this purpose. The Tosafos Yeshanim points out that it does not offer the option of changing into the bigdei zahav to read the parasha. Perhaps the rationale is based on what Tosafos points out that he cannot lichatchila put on bigdei kehuna when unnecessary.
Tosafos adds an additional chiddush. Tosafos explains that the tzitz was an exception to the other bigdei kehuna because there is an explicit pasuk permitting it to be worn "always" - והיה על מצחו תמיד. This implies that the issur of putting on bigdei kehuna unnecessarily is not merely a d'rabonon violation (as rashi explains for walking out l'medina), but rather a Torah violation which is only waived by the Torah itself for the tzitz. Perhaps Tosafos will hold that just as placing on the bigdei kehuna not at the time of avoda is a Torah violation, the continuing to wear too long after the avoda where there is no justification of לא ניתנה תורה למלאכי השרת, would also be a Torah violation. And perhaps Tosafos will hold that even walking out l'medinah would be a Torah violation, against Rashi who writes - שנוהג בה דרך חול, which implies that it is only a drabonon violation.
There is another point that is apparent from Rashi at the beginning of th sugya. The entire discussion is whether one is allowed to benefit from bigdei kehuna שלא בשעת עבודה, but it seems to take for granted that it is perfectly permissible to benefit from them and enjoy their warmth while doing the avoda. There doesn't seem to be any requirement to avoid the intention of pleasure that is being derived by the clothing that he is wearing while doing the avoda. Although when it comes to the concept of מצות לאו ליהנות ניתנו, the Ran (and most rishonim) holds that one cannot derive a hana'as ha'guf that is simultaneous with the mitzvah, this seems to be different. The bigdei Kehuna seem to be similar to a succah, where although there is an issur d'oraysa to benefit from a succah (learned from chagiga - succah 9a), one can enjoy the shade and protection of the succah while sitting in it for the mitzvah. The rationale is that the definition of succah is to be a protection from the elements, as the definition of clothes is to provide warmth, therefore we assume that these benefits that are simultaneous with the mitzvah are permitted.