Ancestral property or self acquired property in hindu law

It is ancestral property and your brother with the other legal heirs has equal rights over the property.

The property inheruited by your father from his father will be considered as his self acquired property and not ancestral property.

After inheriting it he had  transferred the property records to his name which concludes that it is his self acquired property.

Any meaningful suggestion requires that the so-called relinquishment deed be perused and issues discussed.

The facts posted suggest the properties to be self-acquired.

Admittedly the said deed was signed when the grandfather and father was alive. At that point of time 10 acres belonged to the grandfather and 5 acres to the father. There was no property vested with the brother which he could have relinquished.

Further,  who had paid the alleged consideration for the so-called relinquishment deed? Was the deed duly registered?

Our father paid the money for the release deed, which was registered in a sub-register office.

Both your father and grandfather were alive at the time the relinquishment deed was signed by the legal heir. In a way your father forward sold part of the property that he then possessed in exchange for the relinquishment. There are two things to be noted here. (1) Instead of getting a relinquishment from the legal heir, your father could have made a will excluding the legal heir from the will.(2) As your grandfather was also alive then, your father had no right to commit your grandfather's property. Though your father was the only child of your grandfather, your grandfather could have willed his property in any manner that could have excluded your father. When the relinquishment deed was made some calculations would have been made as to how much amount was paid and what would be the value of  possible share of legal heir in your father's property. If amount paid was excessive you survivors would not have agreed. So the legal heir has a share in the property that then belonged to your grandfather. The question of ancestral property or self-acquired property does not arise here.

I know similar cases in our family circles. When it comes to the rights on ancestral property, brothers, sisters and brothers-in-law, in particular, become enemies and even murders take place.  In Tamil there is a proverb "Don't care if my brother-in-law (wife's brother) dies. Both the rugs will then belong to me."