validity of marriage

The alleged second marriage, presumably under Hindu rites and rituals, is invalid and illegal. Your father / step mother can challenge and get it declared void.

Whether your father is alive and staying with the lady ?

If it is registered whether your father has mentioned about his marriage (with your mother) in application and affidavit ? If so, you may file an application before Registrar of Marriage to inquire and book him (your father) for filing false affidavit.

It would be better to consult and engage a local prudent lawyer for appreciation of facts/ documents, professional advise and necessary proceeding.

It is illegal, better to consult local lawyer will be advisable

Do not underestimate your father or his present wife,  as they might have got the marriage registered after the death of your mother and might have declared the present wife only after the death of your mother.  Check-in for facts and how you are benefitted by making the marriage of a father invalid when there are chances of legalizing such marriage officially after the death of his first wife?  Even without all this, he can deal with his self acquired property as he likes through a legally valid document of transfer.

What is your objective from challenging the marriage?

As per the timeline in your query, your Mother passed away in 2011 and your father remarried a lady in 2016. Apparently, the marriage in 2016 is valid as it was performed subsequent to the passing away of the first wife. Your Father did not remarry while his first Wife ie. your Mother was alive.

Now coming to the Nominations given to the lady by your Father even before 2011.  Technically, it is possible to have a nominee who is not a relative or legal heir and that cannot be questioned. In the Nomination proforma there shall be a question to the effect "Relationship with Depositor if any". Maybe your father left that column blank. or just mentioned as none.  However, it is the duty of such Nominee to receive the money in trust from LIC or Bank, etc, and distribute such money to the legal heirs as per their share, only in the event of the death of your father. The mere fact of your father giving the nomination to a lady cannot be construed that he was married to that lady at that time. And besides, your father officially got married to that lady in 2016 and got it registered.

When the nominee is wife (maybe at the time of claim settlement), the nomination even when challenged can not prevail in a court of law.  The wife is entitled to the claim amount.

You have stated that your father legally married your step mother in the year 2016 i.e., very well after the death of your mother and got the marriage registered, therefore the marriage is legally valid and your step mother can claim all rights as a wife.

The nomination of LIC in the year 2011 has not been encashed since your father is living and there was no occasion to prove the nomination anywhere. 

If you raise this question they may cleverly cancel the nomination now and your father can re-nominate her once again after a little while later, since she is now your father's legally wedded wife. 

The proposed challenge by you about their marriage is not valid nor maintainable in law, hence better take advise from a local lawyer properly before proceeding with your thoughts about it.