... we have to keep on going, even if it's only us.

To give the past meaning, this is the only thing that we can do.