Agent principal relationship

We are facing a legal problems due to misdeed done by our Agent who was responsible for sourcing loan cases for us. We are loan aggregators, hiring agents who are not working under exclusive relationship and will be paid on the basis of amount of loan that gets disbursed. We are facing legal troubles due to the fraud conducted by that person. Our relationship was purely based on the amount of business done by him and we have no direct control of him. Our argument is that agent is an independent contractor and any misdeed done by him is not binding to principal. Kindly suggest your opinion.