Before actual growth takes place, swelling occurs. It appears that you caught this in that early phase. It will take several months before you will know for sure.
While there are obviously many men who take Finasteride without problem, There are also people like you. This drug does have risks. Leave it alone for a while, and see what happens.