Urban planners have deemed that the building, which stands alongside a historic church, must go.