You are the biological father of your child, hence you have equal rights over your children similar to your wife.
The wife cannot sit in her father's house only to hide the children from you or to prevent you to have an access to your children.
If you can gather courage to forcibly drag your children out of her father's house, then you can proceed and it will not be considred as a criminal offence.
Firstly her father cannot illegally confine your children in his home and deny access to them Even if he is giving complaint with the police, the police may advise him to advise his daughter to approach court.
He is a third party to your children hence you can make a complaint against him for illegal detention of your children and also for illegally preventing your access to your children as a counter complaint