Under Contract Labour (R&A) Act it is normal that the Inspector inspects the premises.
All persons engaged by the contractor comes under the purview of the act. however they should have working in your premises.
As principal employer your liability is to maintain Register of Contractors, Witness and Certify the wage payment every month in the contractor's wage register, ensure that the PF/ESI is deducted and remitted. Canteen,washing,restroom facilities are wither provided by contractor and as Principal Employer you allow them to use your premises.
Other than the above, you have to take care that you are not engaging contract labour in prohibited works.