According to Section 497 defines Adultery.as — "Whoever has s*xual intercourse with a person who is and whom he knows or has reason to believe to be the wife of another man, without the consent or connivance of that man, such s*xual intercourse not amounting to the offence of rape, is guilty of the offence of adultery, and shall be punished with imprisonment of either descriptttion for a term which may extend to five years, or with fine, or with both. In such case the wife shall not be punishable as an abettor. "
In your case you have not committed any s*xual intercourse with that lady, so the question of charging you for Adultery may not arise.
But, avoid/stop from interacting with that lady forthwith. Even if phone call comes from her do not attend and allow the call to end on its own.