I guess we could ask what would you expect him to say? Something like this maybe?
'Go on my son, get out there and do your best, it's great this job'
He's just being unpleasant and trying to put you off.
So what if he dumped the work you've picked up, lucky you. Don't sweat it mate, carry on.
You will get days when you wish you did something else, but thats true in any job, I know I've had quite a few of them.