It's definitely ok to say no. Each job that involves an element of risk is exactly that, a risk. You have to weigh it up and if you think you can do it safely then fine.
I have a few that are a bit risky e.g. that involve standing on a ladder and wfping over a skylight to get a decent angle to get at three windows which I can't reach from the ground. The ladder is butted up against a wall and while I don't have the three points of contact, it still feels safe enough.
Another one involved a high 1st floor flat roof but with a drop into a stairwell so it's a long way down. Customer asked me to clean his pyramid of a skylight. I told him I would have a look but the roof was covered in moss, slippy as hell and with only about 20 inches to the edge. He was a bit surprised when I said no, but he is still a customer so can't be that bothered.