Could you all go together? Make it a Birthday and Father- Son celebration.
Have a big cake for you and a small cake for them.
Also he could spend 8am 5pm with his son and 7pm till with you.
Or you could celebrate the next day.
There's nothing to be sad or selfish about.