Hi Bummed,
Seeing a couple of pictures of your chest here makes for a much easier comparison. Looking at your chest it would be very normal and ordinary at a nudist club. Nobody at all would notice those breasts. From your description I was picturing something far larger. They appear to be below 30% percentile or so for men. Even if they were 99+%tile most would not notice.
Somehow people expect a brass band entry the first time they enter a club walking into the pool area without any clothing and are disappointed that nobody at all notices them.
And one other thing to remember, gravity and entropy get us all in the end.
I'm an alchemist and practice a spiritual evolution of the being. And I teach this. It is about changing one's being. I firmly believe that people can change themselves. I don't believe that going through life afraid all the time of what somebody might think does a person any good.
I do understand the fear and shame and the hiding. I went through the hell of junior high school, with sizable breasts. As an adult, how much time do you spend looking at other people's chests while you are out; women and men. Quite possibly you and I and others with concerns spend more time than people without this concern. Most men are not looking at women's chests much less other men's. Everybody is far more concerned about their own body than anybody else's.
Have you had you thyroid checked, just as a thought? You mentioned having a change in metabolism. Maybe something changed with your thyroid.
So, here you are, small breasts visible from the side. And genuinely, at a nudist club they would not noticeable at all. A substantial percentage of men there will have equal or larger breasts than you, not identical, but of equal or larger size. There is no "average" body. You just can't add up all the parts and average them together. A guy with a 34 inch chest is just built totally differently than a guy with a 48 inch chest. Saying 42 inches is average then doesn't describe any body. So instead of having any hypothetical "average" persons you have a whole lot of unique and ordinary bodies at the same time. What one doesn't see often are anorexic fashion models or the most beautiful men or women in the world, whatever that means. But rather you see the same people you see mowing their lawns, or working or shopping or in church
Right now on the TV as I'm writing this was a man in a t-shirt in an ad with bigger looking breasts than you and a much bigger gut.
I started wearing t-shirts nine years ago after recovering from 20 years of chronic illness and I had bought my first new clothes in 20 years. And in 9 years of wearing an average fitting t-shirt with D or DD breasts you know what has happened? Almost nothing at all. A few guys working the registers with large breasts hunched over hiding them but they don't hide well when they are big, smiled in relief to actually see somebody larger than they are in a t-shirt and not because they have to wear the company t-shirt.
Maybe it's time for the Gynecomastia Liberation Front (has to be "front", "org" just doesn't put it out front where it needs to be. Who gives the small percentage of skinny flat as a board guys the power to be the definition of male body ideal? I find it silly, ridiculous, for a person to think that most people even could look like that much less "should".
Our body hating culture defines something that makes most men impossibly wrong (as the society does to women) so that the whole body fear and body shame piles on them, something that I have difficulty seeing as other than body dysmorphic disorder in some of the more severe cases. This is important for many to be able to achieve reasonable happiness. All those layers of hot clothing trying to hide breasts was just a different definition of misery for me. Surgery has it's own problems and uncertainties. And sadly satisfaction isn't always possible for many reasons. And there are no guarantees for any choice.
People have an highly artificial idea of what a "normal" body is. It covers a full range of all the body types including a full range of males from flat as a board to at least "H" cups and a full range of women from flat as a board to at least "H" cups. This covers the 2 SD or 3 SD majorities as well a few folks like me and Hammer and a few others. Natural female breasts are what, a B cup average, maybe C, these days? They get bigger as the bodies get fatter, like men, but not always for any specific body.
I personally don't intend to live my life in fear and shame for a body type I had no responsibility in "causing", as you also state. I didn't get to pick my mother from whom I inherited familial gynecomastia and other things. Nobody picks their parents and certainly not for biological idiosyncrasies.
You need to do what makes you happy. It's impossible to know in advance. Some are fully satisfied by surgery and able to do everything like, taking their shirt off publicly, going swimming at a public pool or beach. Concealing clothing might be satisfactory for some but are often uncomfortable and clearly unable to do all sorts of things because of shame about their chest. A few visits to a nudist club does nothing irrecoverable or terribly expensive normally. Nothing gets cut off. The only significant side effects are possible sunburn if you don't use enough sun block or stay in the shade. So it is a good feasible first try. Major side effects include having a good time and accepting your body. You can always try something else.
Personally, and I think many of the old timers here will agree with me, that just ignoring it and living our lives could produce the greatest happiness. I'm never going to be anybody's idea of great looking guy with a perfect body. It wasn't going to happen at 10, 20, 30, 40, 50, 60 or likely 70 either. So I just don't opt into that game I can't win or even play. I prefer to dazzle the ladies with good meal at least some decent conversation, and that works ever so better anyway. In 9 years of wearing t-shirts I haven't had even one rude occurrence. My nipples are always sticking through the fabric of any shirt I wear. Find what you do well and you will find that breasts have absolutely nothing to do with that. I wear a light cotton shirt or a t-shirt.
You have a wife who must at least reasonably accepts your breasts and maybe has no understanding of why you have a problem. My ex-wife never understood why I was so troubled and helped me work through it. My partner doesn't get it at all either why I would have any left over reactions. She is far more concerned about 20 pounds extra weight of her own, which is totally normal in people. She has seen me in everything and nothing so a more clingy shirt makes zero difference in her eyes.
To conclude, genuinely do what will make you happiest in the long run. "To thine own self be true". Happiness isn't running your life for fear of what somebody somewhere might think of some aspect of your body. Have fun.