It's what remains of the aerial roots that comes with ivy growing up walls. The easiest way to get rid is to clear as much of the ivy off the wall by pulling, scraping, struggling and plain physical labour. Then using a hot air paint stripper on the small aerial root remnants. Gently, so as not to scorch the brick work. Then brush it down.