The Gemara says that during Yom Tov the gezeirah on the clothing of am ha'aretz being tamei didn't apply (therefore the gezeira of kusim being bo'alei niddos was necessary). The reason it didn't apply is because in the pasuk by pilegesh b'givah it says ויאסוף כל איש ישראל אל העיר כאיש אחד חברים, and we darshen that during a time of gathering we regard everyone as a chaver, not an am ha'aretz. Since Yom Tov is a time of gathering, everyone is elevated to a status of chaver.
The Maharatz Chiyus points out that a similar drasha appears twice in yerushalmi, once in baba kama and once in chagiga. Both darshen the pasuk of כעיר שחוברה לה יחדיו, meaning that Yerushalayim is a city that brings unity. In Baba Kama the drasha is - עיר שמחברת ישראל זה לזה, a city that attaches Jews one to the other. But in Chagiga it says - שהכל נעשו חברים בשעת הרגל, everyone is considered a chaver on Yom Tov. In baba kama the focus is on achdus - unity, whereas in chagiga the focus is on yom tov. The Maharatz Chiyus has a beautiful explanation based on the Rambam in Moreh Nevuchim (43:3). The Rambam explains that the purpose of Yom Tov is to create achdus among the Jewish people -
אבל ימים טובים המה כולם לשמחה ולקבוץ שיש בהם הנאה שבני אדם צריכים עליהם, ויש מהם תועלת בענין האהבה וכו' והתאהב בני אדם וחברתם קצתם לקצתם
One of the main causes of division between Jews is their not relying on each other for kashrus and therefore being unable to eat at each other's homes. This division was even greater in the times of the Talmud when there were classes created based on one's willingness to accept certain rules of tu'mah and ta'hara. The animosity that existed between Talmidei Chachamim and Amei Ha'aretz as apparent from R. Akiva's statement in Pesachim 48b [when i was an am ha'aretz i wanted to bite talmidei chachamim like a donkey that breaks bones in its bite], is exacerbated by their inability to dine together. Being that the purpose of Aliya L'regel specifically, and Yom Tov in general was to create unity between people, Chazal felt that it was necessary to break down the barriers of division and allow the talmidei chachamim to eat together with the amei ha'aretz. Therefore, the concept of Yerushalayim being a city that brings unity, and the fact that this occurs on the regel is understandable. To allow aliya l'regel to fulfill its function, chazal broke down the barriers that divide the classes.
The Meshech Chochma in Parshas Emor (23:21) elaborates on the same idea. He explains that some mitzvos are meant to create a connection between Jews and Hashem, others are there to establish a connection within the Jewish community. That is why we permit אוכל נפש type activity on Yom Tov, since the whole purpose of Yom Tov is to allow people to eat in each other's homes and join together. With this he conceptualizes the halachic concept of one who bakes much more bread than they need for Yom Tov on Yom Tov is not violating a Torah prohibition - הואיל וחזי לאורחים, since it is fit for guests. The entire concept of permitting cooking and baking on Y.T. is to enable guests to visit.