With ladders it always comes down to risk and assessing that risk and making sure that the risk is always assessed in the same way.
Some take a risk so many times that they then don't see it as taking a risk, they see it as safe as the last 100 times they done that, they didn't fall or have an accident so therefore it's safe. A lot drive their cars on that basis.
Ste - it's good that you are getting some confidence with using a ladder but don't let that fool you into complacency. Remember that fear you felt when you first cleaned windows with a ladder, don't forget that and base each using of a ladder on that original fear no matter how confident you feel.
Personally, if I'm not 100% certain about the position, the angle, the wind, the surface, and so on, I won't go up the ladder and that's final. Also, when using ladders, forget about speed and how quick you can do a job, the only point to remember is to do the job safely. Getting quicker comes naturally as you get more used to doing the job.
The advice above about wfp is sensible. Use how you're cleaning windows now as a stepping stone to upgrading to wfp.