A growing wildfire has forced some Lytton residents to flee their homes, in a community still trying to rebuild after a devastating blaze four years ago.