House Owner -Token Advance Issue

Hi Everyone, We gave a token advance of sum RS.12,000 to one of the big shot in Bangalore(Ramamoorthy nagar) on 11-July-2018. Due to our personal circumstances we are unable to move into home, which we intimated by 21-July-2018. All the while a tenant was staying there in the apartment for which we gave token advance. Now the owner says he will return only half the money and that too when a new tenant occupies the permises. Emotionally as well it has been a very bad experience for me as he made us wait 2 hours in front of his home to speak this with us. We don't have any document proof for this which is a big issue. But we do have call recordings for that. Any legal action could be taken against this?.. kindly advice