However, the cost of the collections used is actually a lot worse from our perspective. And even with compression cost of 2.6 seconds, we are still ahead. So the good news is that we can visible see that we reduced the I/O cost from 8 seconds to 3.8 seconds. Here is the cost of committing using compression (lz4): We’ll focus on the other costs beyond I/O, and profile heavily on systems with higher I/O costs.Ĭompression should work, in fact, my first thought was that we pay for less I/O with more CPU, but that is a guess without looking at the numbers. So for now I think that I’ll do the following. To test it out, I run the same test on an Amazon m3.2xlarge machine, and got the following results (writing to the C: drive): The good thing about this is that running this through the profiler shows that we are running on a high I/O machine, where the amount and speed of I/O doesn’t impact our performance much. That means that 100 – 121 pages usually are compressed to 10 – 13 pages. The LZ4 compression factor is about x10 for our data set. That was pretty disappointing, to tell you the truth, I fully expected that we’ll see some interesting improvements there. I changed how we are writing to the journal, and disabled reading from it. Note that I have done just the minimal amount of work required to test this out. Using NTFS compression slow us down considerably, while both LZ4 and Snappy resulted in greatly reduce file writes, but roughly the same performance. Sadly, we don’t really see a meaningful improvement under that scenario. That means that we have got a pretty big buffer that we can try to compress all at once. We are also usually writing multiple pages at a time for most transactions, and for the scenario we care about, writing 100 random values in a single transaction, we are usually write about a 100 pages or so anyway. And we only ever read from the journal file when we recover the database.
The major cost we have is writes to the journal file.
I decided to try and see what that would give us, and the easiest thing to do is to just enable file compression at the NTFS level. So the obvious thing to do would be to use compression. Note: I would like to point out Alex’s comment, which helped setup this post. Writing 500K random items resulted in over 2.3 GB being written. Writing 500K sequential items resulted in about 300 MB being written. So, we found out that the major cost of random writes in our tests was actually writing to disk.
0 kommentar(er)
