I never remembered hearing the music when I woke up the next day.
Ah, but that doesn't mean it wasn't there! Or do you generally remember everything you dream?
I suppose in a way your son has a point, calling it "scary", as there are places where the music can be imagined as background to a nightmare. But I think that's rather too negative. Perhaps "haunting" would be better here; that's more neutral and doesn't have to have so negative a connotation.
Perhaps to be most receptive to this music you have to be in a certain mood, and for some reason the words "vacant" and "pensive" have just shot into my head. Like Wordsworth's daffodils "flash upon that inward eye which is the bliss of solitude", this music is for your inward ear, and it should "fill your heart with pleasure", not your mind with horror.