The gemara says that if one davens and then finds excrement in the place he davened, it would invalidate his tefilla. However, the gemara considers this to be a violation of זבח רשעים תועבה, indicating that he did something wrong. Therefore, Tosafos explains that the halacha of תפלתו תועבה only applies if he was negligent because it was a place that he should have expected there to be tzoah, but if it was a complete accident - אונס, this halacha would not apply. It comes out that we only consider his tefillah to be a תועבה if 2 conditions are met. 1. He actually found tzoah afterward (this is clear from the gemara which says ומצא צואה במקומו). The M.B. discusses whether it would include a place that there is almost always tzoah there, but would certainly not include a place that may or may not of had tzoah. The implication is that even if he were negligent, chazal don't render his tefillah to be a to'eiva, unless there was actual tzoah present. 2. He had to have been negligent by davening or saying shema (shulchan aruch brings this halacha in hilchos krias shema (end of siman 76). If he wasn't negligent, even though he found tzoah afterward, his tefillah wouldn't be considered a to'eiva. It comes out that the status of תועבה is a combination of factors, his negligence AND the metzius of davening in the presence of tzoah.
What does it mean to be תפלתו תועבה? Tosafos cites two approaches. Both approaches agree that the tefillah is invalid. The first approach is that he hasn't been yotzei tefillah and there is nothing he can do about it. The second approach is that he must daven again. The first approach of Tosafos is very difficult to understand. Since he wasn't yotzei tefillah, he should certainly daven again. What rationale is there to say that he doesn't daven again? It seems to me that what forces Tosafos to say this is that Tosafos understands that davening in the presence of tzoah invalidates the tefillah. Therefore, even if one were not negligent, but later found that they davened in front of tzoah, they must daven again. If so, what does the aspect of being negligent contribute to creating the status of תפלתו תועבה? Tosafos understands that regardless of him being negligent, he wasn't yotzei tefillah since the fact is that he davened in the presence of tzoah. However, if he were negligent we penalize him by no allowing him to make it up, whereas if he was a complete אנוס, we would allow him to make it up by davening again.
The second approach of Tosafos (which is the one brought l'halacha in shulchan aruch) that even when we say תפלתו תועבה we allow him to daven again to make it up, would have to hold that when he was not negligent we don't regard the tefillah to be a to'eiva, and he is actually yotzei with that tefillah. According to this approach, davening in the presence of tzoah may be an issur d'oraysa, but it is not an automatic invalidation of the tefillah (Biur Halacha 76 d.h. tzarich takes this approach that m'doraysa it is forbidden to daven, but the invalidation of the tefilla is only d'rabonon). Therefore, when he is negligent and actually davened in the presence of tzoah we penalize him by invalidating his tefillah, but if he is not negligent, we don't invalidate the tefillah.