That is an interesting topic that is not very discussed. After a certain point, gynecomastia can really look like female breasts. Regardless if they are on a men, they are still perceived as female breasts, and it does certainly attract masculine attention.
I saw some reports where a man that had severe gynecomastia did received constantly stares, and perceived often they often had... ''marks'' on their pants. At one moment, a married man, with wife etc. asked to ''get closer'' with his breasts, as he was feeling tempted.
Regardless if those breasts are on a man, they still being something that men want to look. Specially in situations where the female ones are covered, but the ones on guys with severe gynecomastia are not.
I would say it is wise to wear bras or some selection of clothing if you do want to avoid those situations.