You don't have to fall out with him, do something along the lines that Tosh has suggested, chat to him in a pub and explain that you are now too busy to be able to keep helping him out, and that by doing so it is now costing you lost income.
Tell him that your work has grown to the point that there are only a few of his accounts that you would want even if he did retire, and that to be honest, you no longer need or require to buy work.
If he's a decent guy, even if he doesn't like it very much, he'll understand.
There's a chap I help out occasionally, I cover his commercial work when he goes on holiday, I don't mind too much, when he fully retires I'm hoping he'll just offer me the work, it's only a handful of accounts really, but worth £100-£140 a week, which will be a nice boost
He's ill with prostate cancer at the moment, so at some point he may have to retire a little earlier than he had planned, he's 69 now and the treatment he is undergoing tends to wipe him out, if the medication works he should be in remission by this time next year, slow process, kind of depends how debilitating the treatment is as to whether he can continue to work at the level he is currently doing.
I digress...do your best to remain on good terms, but don't be afraid to stick to your guns...
Ian