You did the right thing.
It's a business and you shouldn't take rejection personally. If you were an employee your boss would have expected this from you, and been annoyed if you hadn't even tried.
Some customers cost you money, constant complaints, bad access, can you leave it this time, and these you have to walk away from. You can never be sure you did the right thing, but you can try.