Bug 413871 - Marketplace Performance
Summary: Marketplace Performance
Status: RESOLVED FIXED
Alias: None
Product: MPC
Classification: Technology
Component: wizard (show other bugs)
Version: unspecified   Edit
Hardware: PC Windows 7
: P3 normal (vote)
Target Milestone: 1.4.0   Edit
Assignee: Project Inbox CLA
QA Contact:
URL:
Whiteboard:
Keywords:
: 454294 457981 (view as bug list)
Depends on:
Blocks:
 
Reported: 2013-07-27 22:57 EDT by Douglas Tillman CLA
Modified: 2017-07-05 08:34 EDT (History)
6 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Douglas Tillman CLA 2013-07-27 22:57:24 EDT
The Marketplace user experience over a fast internet connection is terrible!!  I'm using the latest version of Eclipse EE.
It takes minutes for the initial catalog to load.  
Canceling the operation doesn't really cancel and often hangs.

In the era of big data this experience is just horrible - how could a tools company build such a terrible tooling experience?

The data lookups to load the marketplace should be seconds not minutes.
Fetching the listing runs endlessly with no meaningful progress updates.

I'm trying to get to the Vaadin plugin for Eclipse and want to gain back some of my lost time.  I select 'tools' from the search filter and the whole ugly search and wait process begins all over again.

I finally got the install button and the plugin download operation is in progress. 

There is nothing in the operation that tells me the size of the plugin files to be installed; no estimated download time; just a progress bar that flashes messages that mean little to the user waiting to get back to work.  The minutes of waiting for dependencies and provisioning ends only to display another progress bar for installing the software.  I've downloaded and installed programs that are hundreds of megabytes and installed them faster than adding this plugin.


Not having a reasonable experience undermines the value of the marketplace.  Please improve this functionality.
Comment 1 Douglas Tillman CLA 2013-07-27 23:04:31 EDT
Came off snarkier than I intended and there doesn't appear to be a way to edit my report.  Apologies.
Comment 2 Carsten Reckord CLA 2014-11-26 07:53:32 EST
> Canceling the operation doesn't really cancel and often hangs.

I was unable to reproduce that. Could you attach a screenshot if this happens again, and better yet, a stacktrace (see https://wiki.eclipse.org/How_to_report_a_deadlock#Getting_a_stack_trace_on_Windows)?

> In the era of big data this experience is just horrible - how could a tools
> company build such a terrible tooling experience?

I agree; it would be great to have a responsive list that fills dynamically in the background while you can already work with the first items. Unfortunately, that would require a complete rewrite of large parts of the current frontend code, which probably won't happen any time soon unfortunately...

> The data lookups to load the marketplace should be seconds not minutes.
> Fetching the listing runs endlessly with no meaningful progress updates.

From my observation, this could be related to https://bugs.eclipse.org/bugs/show_bug.cgi?id=432811. Fetching the actual data is (usually) really fast, but then the icons get loaded and the progress gets stuck if an external server is slow or not responding. Unfortunately, this isn't reflected in the progress bar currently. I'll open a separate bug for that and link to it here.

> I'm trying to get to the Vaadin plugin for Eclipse and want to gain back some of
> my lost time.  I select 'tools' from the search filter and the whole ugly search
> and wait process begins all over again.

Actually, vaadin is one of the marketplace entries that have had problems with serving their external icon resource in the past.

> I finally got the install button and the plugin download operation is in
> progress.
> 
> There is nothing in the operation that tells me the size of the plugin files to
> be installed; no estimated download time; just a progress bar that flashes
> messages that mean little to the user waiting to get back to work.  The minutes
> of waiting for dependencies and provisioning ends only to display another
> progress bar for installing the software.  I've downloaded and installed
> programs that are hundreds of megabytes and installed them faster than adding
> this plugin.

A lot of what happens during installation comes from Eclipse's P2 installation framework, which MPC is built upon. I'll have to investigate the progress reporting and check what can be done in MPC and if there's a problem with P2 progress reporting. I'll create bugs accordingly and link to them here.

I don't think that MPC has access to the download size. It would certainly be great to show size and remaining time - we'd have to check with P2 if we could accomplish that.

> Not having a reasonable experience undermines the value of the marketplace.
> Please improve this functionality.
Comment 3 Ian Skerrett CLA 2014-11-26 10:52:50 EST
Can I suggest we do some profiling of the client and the server to see where the performance problems exist.
Comment 4 Carsten Reckord CLA 2015-05-28 06:36:02 EDT
Profiling on client side showed the following bottlenecks:

1) Loading remote images takes a long time and the client waits for those to be completed
2) Updating the list contents can take up to 2 seconds, because the whole SWT UI for each entry is rebuilt from scratch
3) REST requests to the server sometimes take a while to reply

Regarding 1) I changed the strategy for downloading and locally caching images. The client no longer waits for image downloads. It downloads images concurrently and shows placeholder images until the downloads have completed. Also, the image cache is preserved until Eclipse shuts down. Previously, it was discarded when the MPC wizard was closed.

Regarding 2) there's not much to be done without large changes to the UI, some of which would need to happen in P2 Discovery UI and would affect other projects outside of MPC. Maybe we could look into that for SR1.

Regarding 3) my measurements show that sometimes it takes up to 3 seconds between establishing the HTTP connection and receiving the first byte. After that, transfer speed is very good. I assume this happens when we have a cache miss and the server actually needs to compute the reply. Subsequent requests are usually quite fast (~300ms) then. I see this for all supported REST requests, so maybe there is some room for optimization on the server. Other than that, I improved the client-side caching for query results to improve speed.
Comment 5 Carsten Reckord CLA 2015-05-28 06:39:24 EDT
One additional idea I had regarding performance of online requests: We could preload some data (marketplace discovery, maybe the 'featured' list) on Eclipse startup, and bulk-load some more ('recent', 'popular', ...) in the background when opening MPC.

That way, we can at least minimize waiting time for the user. Unfortunately, I think this conflicts with the impression count for promoted solutions, so we'd need a strategy for that, too. Maybe something we can do for SR1?
Comment 6 Carsten Reckord CLA 2015-11-24 09:26:42 EST
*** Bug 457981 has been marked as a duplicate of this bug. ***
Comment 7 Carsten Reckord CLA 2015-11-24 10:12:41 EST
We introduced some additional client-side caching and some improvements regarding UI refresh and image loading that slowed down content creation for the list in Mars. I believe Chris also made some server-side performance improvements.

I fear any more significant performance improvements, e.g. loading Marketplace entry data incrementally instead of in one big chunk, isn't possible with the current architecture. So I'll close this as fixed for now.
Comment 8 Carsten Reckord CLA 2017-01-11 12:54:07 EST
*** Bug 454294 has been marked as a duplicate of this bug. ***
Comment 9 Stefan Jonsson CLA 2017-07-05 07:01:48 EDT
I have the exact same problem. The progress bar gets stuck and when I press cancel nothing interesting happens. It becomes impossible to interact with eclipse because the progress bar becomes the main window and I can't close the progress bar. The only way to shut down eclipse is to shut down and restart the computer. When it starts I can start eclipse as normal and everything looks okay, except when I try to open market place again, the same progress bar shows up and it is still stuck at the exact same spot and it is still impossible to cancel or close it.

Since the progress bar remains stuck in the same state after restart I guess there must be some file on my hard drive containing its state. I guess that what I have to do to get access to market place again, I would have to delete/reset this file, but I have no idea which file to delete.

As a long term solution, you obviously need to fix this bug. As a short term solution, could you at least tell me how to get rid of the progress bar so I can access eclipse market place for other stuff, while installing my plugin manually?
Comment 10 Carsten Reckord CLA 2017-07-05 08:34:29 EDT
(In reply to Stefan Jonsson from comment #9)
> Since the progress bar remains stuck in the same state after restart I guess
> there must be some file on my hard drive containing its state. I guess that
> what I have to do to get access to market place again, I would have to
> delete/reset this file, but I have no idea which file to delete.

There is no saved state that could cause the progress to "resume" somehow when reopening the Marketplace, so I'm not sure what you're seeing there.

The cancellation problem actually sounds like https://bugs.eclipse.org/bugs/show_bug.cgi?id=513218, but the job shouldn't get stuck completely, and neither should it somehow resume. Could you perhaps create a new bug for this, create a couple of stack dumps for your Eclipse process when you encounter this issue and attach them there? 

See https://wiki.eclipse.org/How_to_report_a_deadlock on how to do that - and let me know if you need help with it.