If he works well, and you're not losing work because of it, then it's no problem. Any worker who is good at what they do and cause no problems are an asset. I've had some right dodgey looking blokes working for me in the past, I never really bothered about how customers perceive them as long as I don't lose money.
From what I've read on here about employees, just be happy the bloke does what you pay him for and does it well. If you let him go, then let some of the guys on here know, I'm sure he won't have to wait long for a new job.