Is it mandatory for Token or earnest money to be refunded?

Dear Team, I accepted token of 1 lac for sale of my flat on the condition that the buyer has a pre sanction loan and signed on stamp paper which mentions deal figures but no specific date to end the deal! Now even after 2 months, the buyer has failed to provide any additional sum of money saying his agent cheated him and that the loan amount sanctioned was much less than wat he applied for and that I must give him time to apply for a new loan! On the basis of this deal, I have a booked a flat and now have received a demand letter from the builder saying that I will have to pay 24 percent p.a. in case I fail to pay in time. The buyer says that it is not his look out that I will suffer a loss now and is not ready to share the charges! He has indirectly threatened that in case of deal cancellation he will ensure that token if confiscated will be recovered anyhow! I am clueless wat to do! Inspite of being at the right side of law, I am the one who is suffering!