Just the glass. Most of the house I do have wooden painted frames, most are flaking with loads of putty missing. The sills tend to be concrete, again with pitted and flaking paint. A quick wipe over isn't really a goer!
Where it's a nice upvc frame/sill, then yes I'll wipe it over but only because it makes cleaning the glass easier.
To me a "window" is just the glass. If you break somebodies window it's taken as read that you've broken the glass....not the frame as well!