Have you considered men's shapewear designed for this purpose? Underworks comes to mind:
https://www.underworks.com/men-compression-swimwear. It of course depends on how you want to present and what you are comfortable wearing, but womenswear will generally tend to accentuate/enhance particular male body styles, much like a bra would. The typical plunging necklines of such garments are also distinctly feminine.
I guess it would boil down to what you want here. If you are looking to conceal and not "raise eyebrows" per-se, my advice would be to look towards men's shapewear that is designed for swimming, or at least won't be damaged by it (i.e. a men's T-Shirt and trunks over a compression garment). If you are cool with having a more female body type and don't mind potentially drawing attention to that, wear whatever the heck you want to.
MIMAN's pictures are nice, I'd wear that top under a T-shirt while swimming with my immediate family for sure -- the arm holes are so big it looks very comfortable, but I wouldn't do it in public. The neckline and the racerback are dead giveaways that it was intended for a women, and they will definitely show under a shirt.
Whatever you're comfortable with. To hell what anyone else thinks.