Bug 515268 - Use a shared JDT index across all workspaces
Summary: Use a shared JDT index across all workspaces
Status: VERIFIED FIXED
Alias: None
Product: JDT
Classification: Eclipse Project
Component: Core (show other bugs)
Version: 4.7   Edit
Hardware: All All
: P3 enhancement (vote)
Target Milestone: 4.14 RC2   Edit
Assignee: Roland Grunberg CLA
QA Contact: Roland Grunberg CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2017-04-13 13:13 EDT by Fred Bricon CLA
Modified: 2019-12-09 13:11 EST (History)
8 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Fred Bricon CLA 2017-04-13 13:13:28 EDT
Following on bug #481796

A bit of context:

[quote] We consume JDT from Oxygen in the Java Language Server[1]. The server is currently consumed by our Java extension for Visual Studio Code[2] (and eventually Eclipse Orion, Eclipse Che, vim and others). 
Because each (multi)project we open is assigned a new workspace, all these index.dbs add up *really* fast[3].
Are there any plans on optimizing disk space eventually? maybe using a shared index or limiting index space? 

[1] https://github.com/eclipse/eclipse.jdt.ls
[2] https://marketplace.visualstudio.com/items?itemName=redhat.java
[3] https://github.com/redhat-developer/vscode-java/issues/185
[/quote]


Stephan answered:
[quote]
As for a shared index, that should be possible and would probably work quite well. Those portions of the index that reflect the JRE and any external .jars on the classpath that are common among the different workspaces would only get indexed once. So depending on the workspace, it could actually make a huge difference in both disk space and indexing time.
[/quote]

The fun part about shared indexes is of course having multiple workspaces trying to write to that index concurrently.
Comment 1 Stefan Xenos CLA 2017-04-14 11:18:32 EDT
> The fun part about shared indexes is of course having multiple
> workspaces trying to write to that index concurrently.

Permitting a shared index with mutually exclusive access is different from permitting a shared index with concurrent access. I was proposing the former, not the latter... and supporting concurrent access is way more effort than I have time for.

If this is a requirement for your use-case, we'll have to find some other way to address your disk usage problems. Could you dynamically delete old index.db files when they're not in use?
Comment 2 Roland Grunberg CLA 2019-11-08 14:03:02 EST
As some background :

Bug 356620 allows one to add a classpath entry attribute, IClasspathAttribute.INDEX_LOCATION_ATTRIBUTE_NAME to a classpath entry in order to set a pre-built index location.

However, what's being requested here is not to just set the location for some libraries but rather all libraries for which an index exists in some common location. As a result it seems better to set that in ClassPathEntry#getLibraryIndexLocation() under the IClasspathEntry.CPE_LIBRARY case. It's also worth noting that if a returned index location from getLibraryIndexLocation() doesn't exist, JDT seems to fall back to the workspace location ($WORKSPACE/.metadata/.plugins/org.eclipse.jdt.core/) in AddJarFileToIndex/AddJrtToIndex#execute() .
Comment 3 Eclipse Genie CLA 2019-11-08 14:05:26 EST
New Gerrit change created: https://git.eclipse.org/r/152349
Comment 4 Noopur Gupta CLA 2019-11-11 01:16:42 EST
Adding JDT Core members to guide here.
Comment 5 Nobody - feel free to take it CLA 2019-11-13 05:38:26 EST
Thanks Noopur. 

Manoj, Jay can you spare some minutes on this? Thanks.
Comment 6 Roland Grunberg CLA 2019-11-18 12:58:10 EST
(In reply to Sopot Cela from comment #5)
> Thanks Noopur. 
> 
> Manoj, Jay can you spare some minutes on this? Thanks.

This change should have no effect on regular JDT usage. It's basically guarded off by :

String sharedIndexLocationDir=System.getProperty(IndexManager.SHARED_INDEX_LOCATION);
if (sharedIndexLocationDir != null) {
...
...
}

and the property is not defined by default. Is there any chance to push this for RC1 given the minimal impact ?
Comment 7 Dani Megert CLA 2019-11-20 12:12:07 EST
(In reply to Sopot Cela from comment #5)
> Thanks Noopur. 
> 
> Manoj, Jay can you spare some minutes on this? Thanks.
I've reviewed that change.
Comment 8 Nobody - feel free to take it CLA 2019-11-21 02:54:52 EST
(In reply to Dani Megert from comment #7)
> (In reply to Sopot Cela from comment #5)
> > Thanks Noopur. 
> > 
> > Manoj, Jay can you spare some minutes on this? Thanks.
> I've reviewed that change.

Thanks. Roland pushed an improved proposal right after it :)
Comment 9 Manoj N Palat CLA 2019-11-21 22:42:14 EST
Have one comment/query regarding platform-dependent string based CRC gen - have updated gerrir
Comment 11 Roland Grunberg CLA 2019-12-09 13:11:40 EST
Verified for 4.14 RC2 using I20191204-1800 build.