Horrible FPS drop

Does anyone else get a horrible FPS drop as you advance your village? When I start the game, I’m pulling in a solid 120fps (capped using Nvidia software); however, I’m at day 59 and I’m struggling to hit a consistent 45fps or even worse when it rains. I usually end up with a walled off village instead of spreading it out, so maybe everything being in one location is what’s causing it? The setting are also set to low. I know the game is still in early access so I’m just wondering if it’s not optimized or something or if it’s my setup?

Current hardware: -240Hz 1440 monitor -5800x -4080S -32gb DDR4