Sagging and hard to get rid of weight

[juineaux]

Well, I lost a lot of weight about two years ago(went from 315 to 180, I'm 6' 5"), through tons of walking and a low calorie vegitarian diet. I've been working out in gym for the last 5 months to get rid of some of the excess skin and hopefully build some muscle because I believe I cut the initial weight too quick and ended up losing alot of muscle mass. Since then, my weight has climbed up to 220, but I see no gains in muscle and it seems like just more fat(probably cause I changed my diet again too). Either way when I was 180 or 220 the weight on my chest hasn't seemed to change, even though it doesn't feel like there is any gland. Help-