It depends I had good results from diet and exercise, some people don't though. I was eighteen stone and was six feet 2 when I started .I lost four stone before I got down to my ideal weight. Iam a waist 32 which I boast about
It is pretty hard to tell actually if it is fat or gland from touch. Most of the time fat is soft and gland is hard but sometimes it can be the other way round and you get hard fat and soft gland.
However seeing as your doctor thinks it's fat and you said you had a weight problem in the past it might be worth carrying on and seeing if your chest gets even better. It is pretty hard actually to get the fat off your chest but if you stick with it you may see an improvement.
My advice is to add some weights in now (not too heavy plenty of reps) as getting toned up may well help from looking at your chest.Circuit training is good to for burning fat. It may help with loose skin too.
I added in circuits and weight training almost from the beginning when I got in shape and I don't have any loose skin. It took me a year before I saw a great improvement inmy chest. All I have left is a slight puffy nip on the right hand side . I lost fat from my feet , face, arms legs before it went from my stomach and chest to a huge degree . If I were you I would get to my goal weight and add in some weight training and then have another think .