Dear Govind, you are using word borrowed and not gift. It appears that you have given a cash loan and not gifted money to your brother which you expect to be refunded later.
Please note that receipt of loan in cash in excess of Rs 20,000/- is in violation of section 269SS of Income Tax Act. If your brother shows this cash loan in books, he is liable to be penalised u/s 271D of the Act by way of penalty of Rs 9 Lakhs.
In future don't lend him a single penny in cash as it will again be violation and liable to be penalised by sum of identical amount.
The money given to brother so far can be shown as cash gift by you to your brother as the same will neither attaract any tax or penalty. However, by acknowledging gift, you will lose any right to recover this sum back.