Rava Bar Rav Chanan (RBRC) had palm trees taht were causing damage to R' Yosef. When R' Yosef demanded that he cut them down, RBRC responded that he refuses to cut them down because it is forbidden to cut down a fruit tree as rav says, and is a bad "segula" as we see from r' chanina who attributed the death of his son on the premature cutting down of a fruit tree. The gemara seems to imply that even though the halacha actually requires the fruit tree to be cut down because it is causing damage to R' Yosef's vineyard, it would still be assur or at least a bad omen to cut down the tree.
However, Tosafos asks that we find in the mishna that when a tree causes damage, it should be cut down, so why was RBRC reluctant to cut it down? From the fact that Tosafos doesn't simply answer that even when the halacha permits and even requires it to be cut down, it is a bad omen and not recommended, implies that Tosafos holds that whenever the halacha permits the cutting of it, there isn't even a bad omen. The sakana issue only exists when the halacha doesn't permit it.
Tosafos answers that perhaps the mishna only permits the cutting down of non-fruit trees. Although the gemara in baba kama 92a implies that when the tree is causing damage it can be cut down, Tosafos says that may only be when the damage is excessive (it is weakening the vines so much that the flavor of the grapes is in the dates).
Based on Tosafos, we have to limit the heter that the Taz writes (y.d. 116) to cut down a fruit tree if you need the space for something more valuable, to a situation where it is a significant loss, not just a minor loss.
Another answer is offered by the Nimukei Yosef. In a situation where the tree is causing damage, there is no issur of ba'al tashchis or issue of sakana to cut it down. But, RBRC held that he wasn't obligated to cut it down, so that there would indeed be an issur of ba'al tashchis and a sakana to cut it down. He was telling R' Yosef that if he felt that the halacha requires it to be cut down then there wouldn't be an issur or sakana, so he should cut it down himself.