I can't tell you why it is you, but it is not only you! I know that doesn't make it any easier, but you're not alone in this!
Are you on any type of medications? There are a lot of them that will will cause gynecomastia, if not the first place to start is with your doctor, because you are right, if it hasn't gone down it 4 years it most likely will not.
I'm sorry about the bad news, but I would still welcome you to the forum.
Bob