If those pavers were laid right over concrete, then your contractor was correct to tell you they don't compact them. Unless the broken paver look is what you're after.
Not to mention the challenge of just holding onto that compactor when run over a solid surface!
Silica or other sand will wash out in time, and polymeric sand has been our default for paver projects for the last 4 years or so. So I'd be in favor of that. But don't compact those pavers! If you really feel the need, go buy yourself a $15 hand tamper at the local hardware store and lightly tap those pavers to get the sand to settle. Or rent a hand roller and only fill it up partway - enough to jiggle the pavers as it rolls, settling the sand. Re-sweep, re-tamp, sweep one more time, then wet it down.