I have this same problem.
I went to an RV repair shop and bought some of it by the foot... I think they called it glass lock or glazing lock... I can't remember. But it was expensive, close to $5/ft.
I think for the rest of my windows, I will apply a bead of caulking as best I can without it looking bad.
Taking out the old dried and shrunken pieces is not easy feat... be careful not to break the glass like I did on one of my windows.