I'm sure I've read somewhere on the HSE Working at Heights website that when working on ladders, 'being carefull', (which is what you're basically saying), is not good enough.
From my own limited experience, which consists of ONLY one fall; which gave me (and Wor Lass) a bit of a fright (she saw me slip, land on a porch roof, then fall again and hit the ground), I'd agree with HSE.
When working with ladders, especially when you're young, fit and have a lot of work on and many bills to pay; you're up and down them like a yo-yo.
I still regularly use ladders.
But when you're working fast; you cut corners; take chances and inevitably you'll have a fall.
It's what happened to me and many other members of this forum.