I will take a guess. I think it's psychological. In a way it you are going "backwards" maybe you are remembering something that already happened.
Is it known why walking back thru an entrance of a doorway can oftentimes reset a forgotten memory?