Question about chunks and why do they take so much processing power

Hello! So I started testing some stuff out in my game, and I've found out that, normally, when my game world is 256*256*256 and the chunks are 16 units large, the game runs fine, at max fps, but when i turn down the chunk size to 8, my fps tanks hard. Aka when i create more chunks, the game lags more. Also, when I enlarge the map to 512*512*512 and the chunks are at 16, the fps tanks aswell, but with 512*512*512 and 32 chunk size, the game runs normally. Can someone explain what is happening here? My idea is that chunks are supposed to be run once; they are initiated once when the world loads and then only when you update them. Why does having more chunks make the game lag? Its not like they are constantly doing something intensive or store more data when they are smaller... Is it something about memory, that doesnt allow the game to run fast when more than X are in the game? Thanks!