Tosafos proves from the fact that the gemara considers pits to be included in the issur of orlah that they are considered part of the fruit. Therefore if one eats eatable pits, the bracha is בורא פרי העץ. However, the Rashba disagrees and says that just as we include the shell for orlah based on the extra word את פריו, to include even the shomer of the fruit, we include pits for orlah from the same source, but they are not actually part of the fruit and therefore the bracha is בורא פרי האדמה. Although the Shulchan Aruch 202:3 paskens like Tosafos (and the Rosh), the Tzlach says that he thinks that the halacha should be like the Rashba to make ho'adoma on fruit pits. He proves this from a mishna in Orlah 1:8 that says that although they are included in orlah, they are exempt from revai. This would only make sense if they are not really part of the fruit, but if they would be part of the fruit they should be included in revai as well. Although the tzlach himself rules in accordance with the Rashbah, he offers an answer for Tosafos. The tzlach suggests that there is the outer shell over the eatable part of the pit which is exempt from revai, but the inner eatable pit is part of the fruit and ha'eitz.