but without them police can tell you to stop as has happened to me many times in the past
They can advise. They have no power to prevent you from going about your lawful business, even in a "no cold calling" area.
You can help yourself by being "high profile" - I always wear my uniform, green logo'd cap, green logo'd shirt with green tie, green uniform trousers and if it's cold a green logo'd jacket. I leaflet first and have my logo on the leaflets along with a photo of me in my uniform.
There's absolutely no doubt who I am when I knock on a door