Thanks for this very fair reply !
I agree on the last point, the actual statistics have no real use in our discussion since we seem to be in a theoretical matter (i used to discuss this subject with people actually thinking that it would lead to under representation of cis women in winning women athletes, that's where this argument came from).
The two remaining points (differences and fairness) are kinda the same to my eyes. I'd say i do not focus on disadvantages rather than advantages, i just do not care for any of them, in my eyes they cancel each other out. Now, saying that those differences make the sport thing unfair, does not make sense to me : sport is unfair. We sometimes try to make it more fair with arbitrary categories based on gender, age, weight, etc, but in the end we cannot erase advantages. Some swimmers have genetically better lungs or bigger arms, and we do not make a special category for them, and we shouldn't : that's the point, seeing who's better. There is an unending list of differences between athletes that can lead to (dis)advantages : i think there is no sense trying to erase them all, and even if you did, trans athletes is such a small sample as you said, with such little and debated differences, that even if there were actual advantages, it would be so long down the list of advantages to erase that it should not matter anyway.
To sum up, in my opinion : trans athletes do not have advantages, and even if they had, they would be far less impacting than other advantages that we do not and should not account for.