I made a simple test.
One tilemap of 512x512 tiles, chunks 32x32, when I remove one block (tileMap.Layers[0].SetTile(x, y, -1) and then tileMap.Build()), it takes 2000ms.
When I try this on a 32x32 tilemap, it takes less than 5ms to remove one block. Should I make my own "chunks" with multiple 32x32 tilemaps or did I forget something ?
Note: when profiling I see TileMapCollision.Update() takes 1950ms.