The gemara says that a goy who keeps shabbos and a goy who studies Torah is chayev misah. The simple reading of the gemara seems to indicate that they actually receive capital punishment. Regarding a goy who keeps shabbos the gemara asks that it should be added in the 7 mitzvos given to Noach, to which the gemara responds that since it demands of the goy to do something active such as work, it isn't part of the list. The gemara seems to hold that for technical reasons we don't include it, but it would have the same consequence as the other mitzvos for which he gets capital punishment. Similarly, the gemara says that a goy who studies torah is in violation of either theft or adultery, both of which a goy receives capital punishment.
However, the Rambam (Melachim 10:9) holds that for both of these violations the goy doesn't receive capital punishment. The Rambam groups both the issur on a goy to keep shabbos and the issur on a goy to study Torah together. He then writes:
כללו של דבר, אין מניחין אותן לחדש דת ולעשות מצוות לעצמן מדעתן, אלא או יהיה גר צדק ויקבל כל המצוות, או יעמוד בתורתו ולא יוסיף ולא יגרע. ואם עסק בתורה או שבת או חדש דבר מכין אותו ועונשין אותו ומדיעין אותו שהוא חייב מיתה ע"ז אבל אינו נהרג
The Rambam seems to hold that these 2 things are just examples, but the same would be for other mitzvos that a goy keeps - או חידש דבר. The Rambam holds that the nature of this prohibition is not the activity of torah study or the keeping of shabbos, rather it is a fundamental issue. The 613 mitzvos are a unit and cannot be broken apart. The acceptance of some mitzvos and the rejection of others is essentially a new religion. The prohibition of a goy to accept some mitzvos and not others is essentially the start of a new religion, and therefore forbidden similar to the prohibition for jews of adding or subtracting from the mitzvos of the Torah.
What is the source of the Rambam that the goy doesn't actually receive captial punishment? The lechem mishna says that Tosafos asks on the gemaras rule that we don't find anything that is permitted for a Jew and forbidden for a goy, what about shabbos and torah? Tosafos answers that things which are mitzvos for Jews can be forbidden to goyim, it is only things which are permitted to Jews that must be permitted for goyim. The explanation for Tosafos' logic is that a goy can be stricter than a Jew, so if it is permitted for a Jew it must be permitted to a goy. But, if the nature of the prohibition on a goy is BECAUSE it is given as a special privilege to Jews, then it makes sense that it can be assur for a goy and a mitzvos for a Jew. However, the Rambam maintains that the logic of the gemara that anything permitted for a Jew should be permitted for a goy applies to shabbos and torah as well. Therefore the Rambam holds that the issur on a goy to keep shabbos and study torah cannot be a Torah prohibition, it must only be rabbinic.
The difficulty with the approach of the lechem mishna is that he understands that according to the Rambam it is only prohibited m'drabonon, which doesn't seem to be the case from the language of the Rambam or from stating the severity of the issur as a chiyuv mi'sah? It seems to me that the Rambam is bothered by Tosafos question, but answers it slightly differently. The Rambam holds that the rule of things permitted to Jews must be permitted to goyim, only applies to specific actions. Shabbos and Torah are merely examples of something a goy can do to undermine the theology of the Torah by adopting certain mitzvos and rejecting others. The prohibition is not a specific action and is not limited to Shabbos and Torah, therefore doesn't undermine the rule of whatever specific actions is permitted to Jews is forbidden to goyim.