Bug 553573 - WindowCache's use of SoftReferences causes issues in Gerrit
Summary: WindowCache's use of SoftReferences causes issues in Gerrit
Status: NEW
Alias: None
Product: JGit
Classification: Technology
Component: JGit (show other bugs)
Version: unspecified   Edit
Hardware: PC Mac OS X
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Project Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-28 05:45 EST by Matthias Sohn CLA
Modified: 2020-05-29 15:51 EDT (History)
0 users

See Also:


Attachments
window cache size and used heap timeseries (178.43 KB, image/png)
2019-11-28 05:49 EST, Matthias Sohn CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Matthias Sohn CLA 2019-11-28 05:45:59 EST
While investigating a performance issue on a Gerrit production server we found out that the window cache never reaches its full capacity which we configured as: core.packedGitLimit=96G. We set Xmx (and Xms) to 256G.

We found out that the window cache grows in average to about 10G and then gets wiped off by the Java GC. The attached graph shows the jgit/block_cache/cache_used metric over time.

The drops in the window cache size correlate with the heap consumption peaks. Given that JGit uses SoftReference for WindowCache entries [1], it is no surprise that Java GC is allowed to collect such entries if necessary. We tried increasing the XX:SoftRefLRUPolicyMSPerMB by the factor of 10 (set it to 10,000 as the default value is 1000) but this seems to only bring small improvements.

However, we think that we should have more control over the eviction of the window cache and not just hope that Java GC will not wipe it off.

We asked admins of large Gerrit servers if they can also share their time series for the metric jgit/block_cache/cache_used and the core.packedGitLimit value and the max heap size.

[1] https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/refs/heads/stable-5.1/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/WindowCache.java#571
Comment 1 Matthias Sohn CLA 2019-11-28 05:49:33 EST
Created attachment 280808 [details]
window cache size and used heap timeseries
Comment 2 Eclipse Genie CLA 2019-11-30 18:56:39 EST
New Gerrit change created: https://git.eclipse.org/r/153601
Comment 3 Eclipse Genie CLA 2019-12-09 04:19:50 EST
Gerrit change https://git.eclipse.org/r/153601 was merged to [stable-5.1].
Commit: http://git.eclipse.org/c/jgit/jgit.git/commit/?id=42f0c7c9cb1c516603da6f89ae7072989bf4b984
Comment 4 Eclipse Genie CLA 2020-01-02 19:25:06 EST
New Gerrit change created: https://git.eclipse.org/r/155149
Comment 5 Eclipse Genie CLA 2020-01-26 13:04:07 EST
Gerrit change https://git.eclipse.org/r/155149 was merged to [stable-5.1].
Commit: http://git.eclipse.org/c/jgit/jgit.git/commit/?id=709f83d489b777c21ad5bbeeb3e8b1232b3f0ee5