I am not too thrilled about swimming, but the reason is that I nearly drowned as a small child and has nothing to do with gynecomastia.
Conversely, I would not be likely to go bare chested at the beach unless the girls had the same privilege. Not because of the breasts, but because it is a fairness issue. (Breasts are not sex organs)
The fact that my gynecomastia is a family trait and that I've had it since about 1949 no doubt have a lot to do with my attitudes.