• Bug
  • Status: Resolved
  • 2 Major
  • Resolution: Fixed
  • ehcache-core
  • ljacomet
  • Reporter: shyrkov
  • October 21, 2013
  • 0
  • Watchers: 4
  • February 20, 2014
  • December 02, 2013

Attachments

Description

Hello guys,

I am having some strange behaviour with the disk overflow option.

My test case is pretty straight-forward. I am dedicating 10MB of heap to my cache and would like the rest to overflow to disk (limiting to 100G which more than enough). Putting 10 000 entries into the cache each 10K size, I have strange and somehow unpredictable results after that:

Cache size: 6050 elements Heap: 247 - 5113888 bytes Disk: 5854 - 61348241 bytes

And with each test run those numbers are somehow different, but I never have 10 000 elements on disk, which I expect would be the right behaviour. I’ve also tried to enable diskPersistent=true to check the number of entries after the “restart”. Same issue.

I went further and replaced in the pom.xml of my test project “ehcache” with “ehcache-ee” (I was evaluating BigMemory Go 4.0.4, which includes ehcache-ee-2.7.4.jar).

The results of the tests against ehcache-ee-2.7.4.jar, compared to ehcache-2.7.4.jar, are exactly the one, I expect:

Cache size: 10000 Heap: 456 - 9441024 bytes Disk: 10000 - 164000000 bytes

So, no entries are “lost”. The heap is properly limited to approx. 10M and all the rest my entries are overflowed to the disk.

What is different in the code base for ehcache vs. ehcache-ee (same version number - 2.7.4) that makes the first one behave wrongly in the provided test case? Are my expectations for ehcache perhaps wrong?

I am attaching the configuration, I’ve used.

Thank you in advance!

Kind regards Sergiy

– Sergiy Shyrkov Product Development ——————————– Jahia Solutions Group web: http://www.jahia.com ——————————— Jahia’s next-generation, open source CMS stems from a widely acknowledged vision of enterprise application convergence – web, search, document, social and portal – unified by the simplicity of web content management.

Comments

James House 2013-11-04

Louis, please eval, assign to QA to reproduce if need be.

Louis Jacomet Jacomet 2013-11-05

Hi Sergiy,

There is an implementation difference between open-source disk storage and enterprise disk storage. The first one uses heap to store the key and a pointer to the disk entry containing the value. The second one does not use any heap. So what you are seeing can be explained by this major difference. You should look at eviction statistics to assess whether or not some elements were evicted in your open-source test. If this explanation does not match what you see, please provide a reproducible test case, so that we can look further into the issue.

Sergiy Shyrkov 2013-11-07

Hello Louis,

thank you for answer! I am not sure I understand correctly the argument that the difference in technical implementation for community and enterprise distributions have completely different results in the functional point of view.

If you could take a look at the attached test case, please, and confirm my expectations are correct / wrong w.r.t. to the ehcache and ehcache-ee?

I’ve attached the Maven-based project with a single test case that: 1) Uses sample ehcache configuration that limits cache heap size to 10MB and uses localTempSwap persistence strategy to overflow the rest to the disk 2) Populates the cache with 100 elements of 1 MB size 3) Waits a couple of seconds 4) Checks the cache statistics to verify that the disk store has finally 100 entries written into the cache

I. Test case 1 I execute “mvn test” on the project. The output is:

Done populating cache with 100 element of size 1048576 bytes Cache stats: Size: 8 Heap: 2 - 2097552 bytes Disk: 8 - 8390656 bytes Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 2.99 sec «< FAILURE!

Results :

Failed tests: testDiskStore(org.jahia.sandbox.ehcache.EhcacheTest): Expected to find 100 element on disk, but found 8 expected:<100> but was:<8>

Tests run: 1, Failures: 1, Errors: 0, Skipped: 0

With every test the numbers (amount of items in heap and disk) are strangely not constant. Sometimes I have 0 elements in heap and 10 on disk, sometimes 2 / 6, sometimes 3 / 4.

II. Test case 2 I change the pom.xml in the project and change: net.sf.ehcache ehcache 2.7.4 to net.sf.ehcache ehcache-ee 2.7.4

to use the enterprise version of the Ehcache (same version 2.7.4).

I execute “mvn test -Dcom.tc.productkey.path=" on the project. The output is:

Done populating cache with 100 element of size 1048576 bytes Cache stats: Size: 100 Heap: 9 - 9438984 bytes Disk: 100 - 209716800 bytes Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.07 sec

Results :

Tests run: 1, Failures: 0, Errors: 0, Skipped: 0

And the results are always the same and always correct with each test run.

So, for me the behaviour of the community cache is inconsistent. It seems that when the memory store is reaching its limits and starts evicting elements, it also evicts those elements from disk store, which is strange.

Thank you in advance for your help!

Kind regards Sergiy

Louis Jacomet Jacomet 2013-11-29

Hello Sergiy,

I had a look at your test and can explain the result (in a simplified way): When you put an entry to your cache, it first is accounted for completely on heap. Once it has been written to disk, the entry size is adapted to no longer account for the full value but only the reference to the disk entry - thus releasing some space for other entries. By doing your puts in a tight loop, there is a lot of pressure on your heap of only 10MB, which results in a lot of evictions. This is caused by the latency of writing to disk.

If you add the eviction statistic to your test, you will see that: {{out(“Eviction “ + stats.cacheEvictedCount());}}

There is however a bug as some evictions that happen are not accounted for - thanks for helping finding that out. I will be committing a fix soon for this.

There is also a control mechanism that will help you reduce the amount of evictions. There is a per cache setting: {{diskSpoolBufferSizeMB}} This will limit the rate of puts once it sees that the queue for writing to disk is full. The default value is 30MB, which with 10MB onHeap never happens as evictions occur way before this queue is full. With only 10MB of heap, even with {{diskSpoolBufferSizeMB}} limited to 1 I still see evictions. However, having 30MB of heap and 10 for {{diskSpoolBufferSizeMB}} then I have no evictions, 100 elements on disk at the end of the test.

Hope this clarifies the issue for you.

Sergiy Shyrkov 2013-12-02

Hello Louis,

thank you for your clarifications!

Kind regards Sergiy