Well do you have any proof that she was living with you ? Upcourse the only lie she can say would be marriage was not consummated because what so ever happens between two of you happens would happen in a closed room. You have to prove that both of you did lived as husband and wife and pls read clause 1) which might be very important for you since you are not neglecting her or refusing to stay with her !!!
In your case you will try to argue as per clause (iii) in which you have ot prove that other party never showed any willingness to stay with you then how can a marriage be consummated ? You should be allowed to stay with her if that's the case.
The Special Marriage Act, 1954
25. Voidable marriage. -
Any marriage solemnized under this Act shall be voidable and may be annulled by a decree of nullity if -
(i) the marriage has not been consummated owning to the willful refusal of the respondent to consummate the marriage; or
(ii) the respondent was at the time of the marriage pregnant by some person other than the petitioner; or
(iii) the consent of either party to the marriage was obtained by coercion or fraud, as defined in the India Contract Act, 1872 (IX of 1872):
Provided that, in the case specified in clause (ii), the court shall not grant a decree unless it is satisfied,-
(a) that the petitioner was at the time of the marriage ignorant of the facts alleged;
(b) that proceedings were instituted within a year from the date of the marriage; and
(c) that marital intercourse with the consent of the petitioner has not taken place since the discovery by the petitioner of the existence of the grounds for a decree:
Provided further that in the case specified in clause (iii), the court shall not grant a decree if,-
(a) proceedings have not been instituted within one year after the coercion have ceased or, as the case may be fraud had been discovered; or
(b) the petitioner has with his or her free consent lived with the other party to the marriage as husband and wife after the coercion had ceased or, as the case may be, the fraud had been discovered.