I think it might be because as a child, maybe posters teaching numbers and days of the week in the classroom were colored, so I associated the color of that number and day with it?
Sunday is yellow, Monday is red, Tuesday is green, Wednesday is blue, Thursday is yellow-orange, Friday is yellow, and Saturday is light blue.
1 is red, 2 is green, 3 is purple, 4 is orange, 5 is red, 6 is indigo, 7 is silver-blue, 8 is purple, 9 is green, 10 is red.
As for people, for example, I associate my mother with the color green.