well josie
i'm going with martin, i had a word with my partners son who has been using these sealants for 17 years in the double glazing industry.
there is only one answer, dig it out and start again, the trick is to do two things, first is to thoroughly clean all surfaces and i mean deep clean second apply a commercial mould product and let dry, then re apply sealant.
if it is your own place the following is best practice
use bath/ shower - spray sealant with anti mould spray - hose off with shower head, everytime sealant is rubbed down it is abraided and these abrasions become the areas the the molud collects in so if at all possile avoid scratching the sealant