You should not have handed over the cheque.
Employee should always consult elders in the family, competent and experienced well wishers, lawyer/law firm before signing on the dotted line accepting any condition like to deposit a cheque. One should never act in haste.
Has the company mentioned in advertisement/interview call letter/offer letter, issued to you that candidate shall have to pay an amount of Rs................if the candidate does not join services after selection, and has to deposit a cheque etc towards the same, or that candidate has to sign some service agreement/ keep a security cheque-FD with the company. Did you accept the condition in writing?
This is another trend started by the companies. Some time back a pharma company was contemplating to collect instrument of payment and use it if the candidate does not join.
You may approach a competent and experienced service lawyer, with copy of all communications. You should think of some pretext under which you can claim that cheque was collected from you, issue a notice that cheque should not be presented and stop the payment of cheque. However let your lawyer draft your communications and refrain from acting in haste and without consultation. If the need be let your lawyer issue legal notice. You may not issue any communication without consulting.