There's an age factor involved, of course. Most people don't clearly remember anything much before they were around 18 months to 2 years old. My first clear memory is of the first time I saw my younger brother, when I was two. My stepdaughter knows that her biological father lived with her and her mother for the first year of her life, but actually remembers nothing before Anne and I started seeing each other when she was 18 months old - I'm the only 'Dad' she actually remembers having.
Beyond that there are memories which are impactful, and those which are ground into us by endless repetiton. How well do you remember your school hall? I can call up images of every school hall I attended daily Assemblies in without effort.
I'm often suspicious about 'buried' memries, because it's usually the bad ones that haunt me. I think we just recall what's important to us!