A local bakery has ditched a design for its annual gingerbread project based on the new Harry Potter show.