Looking Forward By Looking Back