Last week I was checking the performance of some recently installed Nehalem-based ESX servers in our server farm and found out that poor memory sharing was happening at VM level and also that memory consumption at ESX level was much higher than before changing hardware.
This surprised me because I know that one of the main advantages of virtualization is memory pages sharing. To give a little technical background, memory page sharing is a memory management technique where the hypervisor analyses and hashes all memory in the system and stores it in a hash table. Every one hour (but this is configurable), the hashes are compared and if identical hashes are detected, a further bit-by-bit comparison is performed. Then, if the pages have the same content, just a single copy is kept and multiple VMs memory are mapped to this shared page. The standard size for this pages is 4KB.
This is what VMWare calls Transparent Page Sharing (TPS) and, as you can see from the following screenshot, this is not taking place anymore for all of my migrated VMs.
After some investigation I discovered that this uncommon behaviour is due to the fact that my new ESX servers are running on a Intel Xeon 5570 CPU, which is member of the relatively new Nehalem based Xeon 5500 processor family. This family of procs implements Extended Page Tables (EPT), which is Intel's second generation x86 virtualization technology for the Memory Management Module (MMU).
For EPT to be really effective Large Memory Pages are used (up to 2048KB instead of the usual 4KB), but using such big memory pages really makes it unlikely that two identical pages of memory are found and therefore shared between different VMs.
This is why I am not seeing any memory page sharing taking place. So, what know that TPS is gone? Well, TPS is not completely dead. It is still there sitting behind the curtains and waiting for the right memory usage condition to pop it. In fact, happily enough, VMware does not let the performance benefits of the 2048KB page size come at the price of the old TPS mechanism: under heavy memory pressure (i.e. memory commit = 100%), the ESX will accept to break down the large 2MB memory pages into smaller 4KB memory pages in order for TPS to effectively kick in and begin sharing memory.
So it looks like just that I have to get used to this new behaviour and accept it. If this new mechanism can bring better performance without forcing the old good TPS to resign, well, it is warmly welcome!
For more information about EPT and the Intel Virtualization Architecture, check the Intel website.
If you found this article useful or if you want to share your opinion about this new behaviour, do not hesitate to leave a note or a comment!