Fit, active but developing breasts!

[glaister]

Dear Sirs,
I am worried and unhappy about the development of fatty deposits on my chest. I am 41 years old, but (for my age) an extremely active person, I weigh 11 1/2 stone and am 5feet 10inches tall. I ski, rock climb, mountaineer, etc... I go out at least once if not twice a week and climb a mountain, usually taking me 6-8hours, sometimes 12 hours with 1-2 thousand meters hight gain. I also walk my dogs regularly several times a week.
But, since my 40th I have started to develop fat deposits on my chest and under my arms, its wierd. Now, because I am completely obsessed by my weight I get rather depressed about it.
Fundementally I just dont get it: if I do an average of 3 hours sport a day for a week (perhaps all on 2 or 3 days) then it disappears. If I stop for a week and do nothing it comes back! I fluctuate that much. I just dont have time to do 3-4hrs of sport every day all my life, I have family, work etc...
The only 'bad' things I can suggest I do which may contribute is: drink a bit too much wine and smoke a few cigarettes after dinner, no more than 3 or 4 a day, sometimes less.
Another wierd fact: when I am in France (the Alps) for a few weeks, regardless of the level of sport I do, the fat disappears, but when back in the UK it comes back.
French food is very natural and British food, no matter how natural we think it might be, is usually packed with chemicals and all sorts of extras. Are there foodstuffs that promote fat growth on the chest?
I really do not know what to do.