The gemara says that the term רעהו in the torah is davka, so that the din should be that whether a jew's ox gores a goy or a goy's ox gores a jew, the damager should not be liable, as the tana of our mishna holds by hekdesh (The nimukei yosef explains that we darshen "ray'eihu" by goyim as we do by hekdesh to exempt in both directions. Maharshal changes the girsa in tosafos to conform to this, but the maharsha and maharam maintain that the term "ray'eihu" only excludes hekdesh, but does not exclude goyim. Rabbeinu peretz in shita also says that goyim are included in רעהו). However, the rabbonon darshened that since the goyim were not keeping the 7 noachide mitzvos, the torah was matir their money and hold them responsible for damages they cause to a jew.
The Rambam (nizkei mamon 8:5) offers a rationale for when a jew causes damage to a goy being patur, and when a goy causes damages to a jew being chayev. When a jew damages a goy he is patur because "לפי שאין הגוים מחייבין את האדם על בהמתו שהזיקה, והרי אנו דנין להם כדיניהן". When a goy damages a jew he is chayev because "קנס הוא זה לגוים לפי שאינן זהירין במצות, אינן מסלקין הזיקן, ואם לא תחייב אותן על נזקי בהמתן אין משמרין אותה ומפסידין ממון הבריות". Why does the Rambam give a rationale in both directions? If we darshen רעהו to exclude a goy, it is understandable why a jew is patur for damaging them without any logic, and if we don't darshen it to exclude a goy, it is understandable that a goy who damages a jew is chayev without offering any rationale?
The rambam clearly learns that the gemara which says that since they were not keeping the 7 mitzvos, the torah was matir their money, to explain why a goy who gores a jew is chayev to pay full. This implies that the default position is that both a jew to a goy and a goy to a jew should be patur, because רעהו is davka, and the gemara had to explain why a goy is chayev for damaging a jew. The Rambam understands that the fact that they weren't keeping the 7 mitzvos is an indicator that they were not concerned about others i.e. killing, stealing, adultery. Since they were not concerned about damages to others, they weren't watching their animals to prevent them from damaging, so we apply a penalty to encourage them to watch their animals.
Now, the only question that is left is why the Rambam has to give a rationale for a jew being patur for damaging a goy - the reason should simply be because it says רעהו? The Ra'avad argues on the Rambam's logic that it is simply not true, because when a goy's animal gores another goy, the nizuk grabs the animal to collect from? Although the Magid Mishna tries to justify the rambam's rationale, that it applied where he lived, he points out that the din would be the same even in societies where secular law dictates that people are responsible for their animals damaging. The lechem mishna claims that the Rambam's rationale is from the yerushalmi, but the question still remains why is it necessary?
The Rambam seems to darshen the טעמא דקרא, that the rationale for the torah exempting a jew for damaging a goy is because in their dinim they are not liable for damaging each other. The Rambam isn't just making this rationale up on his own (as the magid mishna indicates), rather he saw this rationale in the pasuk itself. Whenever the Torah limits a mitzvah to a ריע, such as ואהבת לרעך כמוך and לא תעמוד על דם רעך, it is indicating that since the Jews are an isolated entity from the goyim, they are obligated to treat each other with special privellages. Therefore, the Rambam writes that the רעהו in the context of damages as well, is not merely a gezeiras hakasuv, rather it is an indication that paying for damages that were caused by one's animal, is a special privellage between jews, and doesn't apply to goyim, since they don't recognize it as something worthy of being liable for.
This past shabbos, Dr. Ari Tuchman raised a very insightful question: Why do we use the term וישאלו איש מאת רעהו by the jews taking from the mitzrim, doesn't רעהו imply jew from jew? (This question would only be according to nimukei yosef and maharshal that i mentioned above, who consider the maskana of the gemara to be that רעהו excludes goyim).
I found that Rabbeinu B'chaya answers:
יראה לי שקודם מתן תורה היו כל הבריות חברים כאחד, אבל לאחר מתן תורה שהחזיר הקב"ה את התורה על כל אומה ולשון ולא קבלוה עד שקבלוה ישראל, יצאו כל האומות מן האחוה והריעות ונשאר השם הזה בעם ישראל בלבד שנקראו אחים ורעים למקום, הוא שכתוב "למען אחי ורעי", ודרשו רבותינו "לכל אבדת אחיך" אחיך ולא לגוי, וכן "לא תשיך לאחיך" אחיך ולא גוי
Based on Rabbeinu B'chaya, the status of the jews being called "רעים", to the exclusion of the goyim, occured at matan torah when the goyim refused to accept the torah, so that only the jews remained רעים of Hashem by accepting His Torah. Therefore, the status of רע which teaches that a jew is patur for damaging a goy, occured at the precise moment that the goyim rejected the torah, causing Hashem to be mafkir their money to the jews and making them liable for their ox damaging jews.