My sister had brown eyes just like me, Why are my brothers eyes blue?
The answer to your question is probably similar to the answer to my question.
Some boy babies have Gynecomastia at birth which goes away in a few months. About half of all boys have some degree of the condition in early puberty. For some, it goes away in a couple of years while for others, it does not diminish. The condition arises again in older age.
All together, almost 1/3 of all men have the condition and (surprise) it is completely normal. But it can be a source of embarrassment. If it troubles you, surgery will reduce the size of your breasts. The choice is yours alone to make.