Claiming father's self owned property

No... you cannot claim the self acquired property of your father during his lifetime. Your dad is correct when he says that you have no right over it. The court will not grant you any stay even if he tries to transfer the property to the maid or anyone else. He is free transfer the property to anyone he wants. 

Is there any way for me to claim that property. Can i give a complaint to the police station stating that my father is having such a relationship? The house maid is havin 3 sons and she is not legally separated from her husband.

There is one more ancestral property which will be shared by my father and his three brothers. It is a joint property and four of them have signed in the document. Can my dad transfer that property as well?

If it is an 'ancestral property' then only you can lay your claim.

As far as complaint to the police is concerned, then police cannot and will not do anything about it. Your father is an adult and that woman is also an adult so, no crime is committed so as to warrent the interferance of the police. If the husband of the maid wants then he can file a case of adultry againt your father in the court but you cannot file that case against him.