Totally agree, WFP isn't goin to scratch glass, its next to impossible with trad too.
I use stainless steel fan jets in my brush and I can't even get them to touch the glass, and even if I could it would be highly unlikely to scratch the glass as there are not any sharp or rough edges.
The bristles are nylon, the brush head is plastic, none of which is as hard as glass, so how on earth are you going to scratch glass with something so soft?
If he looks through his window and the sun is out on a cloudless day, and the sun is also low in the sky, then every flaw in that pane of glass will stand out, if this was the case then that is probably why he noticed the scratches, but it's virtually impossible for it to have been you.
Be confident and deny it totally, don't be rude to him but stand your ground, and I agree with the idea of giving him your pole & brush and asking him to try and scratch the glass!
Ian