This isn't a plot hole. You're just writing about your lack of understanding.

What if a memory wipe doesn't eliminate memories, but locks out those memories from being accessible. But fragments of the memories break through, like a person you knew from long ago but can't place where.

Doctor Who is stronger for open minds, and not the rigid thinking you're exhibiting.