On the ground in the English seaside town rightwing influencers say is being ‘invaded’