The gemara tries to figure out the case where one inadvertently violates avoda zara and is obligated to bring a korban for the violation. The gemara says if he is in a church but thinks that its a shul (maybe he thinks the guy in front on the cross is Moshe Rabbeinu) - הרי לבו לשמים, his heart is to heaven. Meaning, that is not a violation of avoda zara at all and he doesn't need to bring a korban. The Ritva points out that even if it wouldn't be לבו לשמים, so long as it wasn't לבו לעבודה זרה, he wouldn't be obligated to bring a korban. The only reason the gemara goes to the extreme of לבו לשמים is because when a person is in shul, it is likely that his heart is to Hashem.
Rashi also seems to be bothered by why we need to say that in his mind he is both inside a shul and לבו לשמים. The Ritva seemed to understand that the thinking it being a shul is essential, but the לבו לשמים is not necessary, so long as his heart isn't to avoda zara. Rashi on the other hand seems to say that the כסבור בית הכנסת הוא, the thinking of a shul is not necessary. Rashi comments - ומה חיוב יש כאן, אפילו ידע שהוא בית עבודה זרה והשתחוה בו לשמים אין חיוב כאן. Rashi holds that since his heart is to Hashem, who cares where he is standing. Even if he is in front of an avoda zara and well aware of where he is, so long as his heart is to Hashem, he is not obligated to bring any korban. From Rashi we learn that so long as someones heart is to Hashem, even if it looks like they are worshiping avoda zara, they are not chayev misah or a korban (perhaps what compels rashi is that otherwise, we have a perfect case where he is chayev a korban, he is in a church and bowing with his heart to Hashem thinking that it is permitted to do so). The Ritva comments that although the issur avoda zara may be dependent on what he has in mind, the din of יהרג ואל יעבור is certainly not dependent on what he has in mind. If by simply having לבו לשמים the halacha would be that one wouldn't have to sacrifice his life, there would never be a din of יהרג ואל יעבור because we would always tell the person that his heart should be to Hashem. Therefore, the Ritva concludes that the din יהרג ואל יעבור must not be dependent on the issur avoda zara, rather on chilul hashem. Since there is a chilul hashem we demand that he give up his life and not bow down.
Rav Moshe (Dibros #27, pg. 536) struggles with Rashi. Rav Moshe assumes that when a person knows that he is in a church and bows down in the church, even if his heart is to Hashem he is serving avoda zara and chayev. The fact that his heart is to heaven becomes irrelevant since to everyone watching it looks like he is worshiping avoda zara. Rav Moshe suggests that Rashi must be describing a situation where the people watching are confident that he believes in Hashem and his heart is to shamayim. Nevertheless, Rav Moshe says that it would at least be a d'oraysa violation of chilul hashem which would ruin his presumption of innocence thereby rendering his thoughts to merely דברים שבלב and therefore he should be chayev a חטאת in this situation. Perhaps it is talking about when he is alone and therefore believed to say that his heart was to Hashem.