Bug 522733 - [Gtk][Webkit2] Crash on close with webkitgtk 2.18
Summary: [Gtk][Webkit2] Crash on close with webkitgtk 2.18
Status: VERIFIED FIXED
Alias: None
Product: Platform
Classification: Eclipse Project
Component: SWT (show other bugs)
Version: 4.7   Edit
Hardware: PC Linux
: P3 major with 1 vote (vote)
Target Milestone: 4.7.3   Edit
Assignee: Leo Ufimtsev CLA
QA Contact:
URL: https://etherpad.openstack.org/p/Ecli...
Whiteboard:
Keywords:
: 526032 (view as bug list)
Depends on:
Blocks: Webkit2Enhancements
  Show dependency tree
 
Reported: 2017-09-25 09:29 EDT by Alexander Kurtakov CLA
Modified: 2018-04-03 12:28 EDT (History)
13 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Alexander Kurtakov CLA 2017-09-25 09:29:49 EDT
Backtrace on close is:
#0  0x00007f40313f0ecc in WTFCrash () at /lib64/libjavascriptcoregtk-4.0.so.18
#1  0x00007f4031c93fe1 in WebKit::WebCookieManagerProxy::processPoolDestroyed() () at /lib64/libwebkit2gtk-4.0.so.37
#2  0x00007f4031cd939a in WebKit::WebProcessPool::~WebProcessPool() ()
    at /lib64/libwebkit2gtk-4.0.so.37
#3  0x00007f4031cd9859 in WebKit::WebProcessPool::~WebProcessPool() ()
    at /lib64/libwebkit2gtk-4.0.so.37
#4  0x00007f4031e91ad0 in webkit_web_context_finalize(_GObject*) ()
    at /lib64/libwebkit2gtk-4.0.so.37
#5  0x00007f4049eae579 in g_object_unref () at /lib64/libgobject-2.0.so.0
#6  0x00007f40be9a4bd8 in __run_exit_handlers () at /lib64/libc.so.6
#7  0x00007f40be9a4c2a in  () at /lib64/libc.so.6
#8  0x00007f40be98a041 in __libc_start_main () at /lib64/libc.so.6
#9  0x0000563c30d926fa in _start ()
Comment 1 Leo Ufimtsev CLA 2017-09-25 14:01:08 EDT
(In reply to Alexander Kurtakov from comment #0)
> Backtrace on close is:
> #0  0x00007f40313f0ecc in WTFCrash () at
> /lib64/libjavascriptcoregtk-4.0.so.18
> #1  0x00007f4031c93fe1 in
> WebKit::WebCookieManagerProxy::processPoolDestroyed() () at
> /lib64/libwebkit2gtk-4.0.so.37
> #2  0x00007f4031cd939a in WebKit::WebProcessPool::~WebProcessPool() ()
>     at /lib64/libwebkit2gtk-4.0.so.37
> #3  0x00007f4031cd9859 in WebKit::WebProcessPool::~WebProcessPool() ()
>     at /lib64/libwebkit2gtk-4.0.so.37
> #4  0x00007f4031e91ad0 in webkit_web_context_finalize(_GObject*) ()
>     at /lib64/libwebkit2gtk-4.0.so.37
> #5  0x00007f4049eae579 in g_object_unref () at /lib64/libgobject-2.0.so.0
> #6  0x00007f40be9a4bd8 in __run_exit_handlers () at /lib64/libc.so.6
> #7  0x00007f40be9a4c2a in  () at /lib64/libc.so.6
> #8  0x00007f40be98a041 in __libc_start_main () at /lib64/libc.so.6
> #9  0x0000563c30d926fa in _start ()

Is there a java side of the stack trace? Any steps to reproduce?
Comment 2 Alexander Kurtakov CLA 2017-09-26 03:32:25 EDT
(In reply to Leo Ufimtsev from comment #1)
> (In reply to Alexander Kurtakov from comment #0)
> > Backtrace on close is:
> > #0  0x00007f40313f0ecc in WTFCrash () at
> > /lib64/libjavascriptcoregtk-4.0.so.18
> > #1  0x00007f4031c93fe1 in
> > WebKit::WebCookieManagerProxy::processPoolDestroyed() () at
> > /lib64/libwebkit2gtk-4.0.so.37
> > #2  0x00007f4031cd939a in WebKit::WebProcessPool::~WebProcessPool() ()
> >     at /lib64/libwebkit2gtk-4.0.so.37
> > #3  0x00007f4031cd9859 in WebKit::WebProcessPool::~WebProcessPool() ()
> >     at /lib64/libwebkit2gtk-4.0.so.37
> > #4  0x00007f4031e91ad0 in webkit_web_context_finalize(_GObject*) ()
> >     at /lib64/libwebkit2gtk-4.0.so.37
> > #5  0x00007f4049eae579 in g_object_unref () at /lib64/libgobject-2.0.so.0
> > #6  0x00007f40be9a4bd8 in __run_exit_handlers () at /lib64/libc.so.6
> > #7  0x00007f40be9a4c2a in  () at /lib64/libc.so.6
> > #8  0x00007f40be98a041 in __libc_start_main () at /lib64/libc.so.6
> > #9  0x0000563c30d926fa in _start ()
> 
> Is there a java side of the stack trace? Any steps to reproduce?

There is no java stack trace, it happens on close and the real bug from it is preventing IDE restart. On fully upto date Fedora 27 start any snippet involving Browser and observe the crash on close.
Comment 3 Leo Ufimtsev CLA 2017-09-26 09:26:09 EDT
(In reply to Alexander Kurtakov from comment #2)
 
> There is no java stack trace, it happens on close and the real bug from it
> is preventing IDE restart. On fully upto date Fedora 27 start any snippet
> involving Browser and observe the crash on close.

Looks very important. I will investigate now.
Comment 4 Eric Williams CLA 2017-09-26 09:55:13 EDT
I had a similar crash when working on accessibility: closing a snippet would crash every time.

If you notice in the stacktrace there is a method "webkit_web_context_finalize", which is part of the GObject destruction process. My guess would be that g_object_unref() is being called on an object that has already been unref-ed.

https://developer.gnome.org/gobject/stable/howto-gobject-destruction.html
Comment 5 Leo Ufimtsev CLA 2017-09-26 10:32:45 EDT
I can reproduce crash with 2.18 packages from Rawhide. ~Investigating.
Comment 6 Leo Ufimtsev CLA 2017-09-26 15:59:43 EDT
~Still investigating. 

Finding some irony in the update notes for Webkitgtk 2.17.*
  https://webkitgtk.org/2017/09/04/webkitgtk2.17.92-released.html
   [] Fix a web process crash when closing the WebView.

I wonder if related.. ..

It seems that there is an object_unref being applied to webView, which triggers an assertion. If I add an g_object_ref(webView) (e.g in onDispose()), then the crash doesn't occur.

Will continue to investigate.
Comment 7 Leo Ufimtsev CLA 2017-09-27 10:01:32 EDT
I've been investigating.

I've reduced Webkit.java to basically more or less only:
webkit_web_view_new()
OS.gtk_container_add (browser.handle, webView);
OS.gtk_widget_show (webView);
OS.gtk_widget_show (browser.handle);
...
gtk_widget_set_size_request()

Then when I run the snippet, after java code finishes, the JVM crashes.

If I remove "OS.gtk_container_add()" or if add a g_object_ref(webView) [as mentioned in comment above], then the crash doesn't occurs.

Theory:
The parent shell disposes Browser/Webview, and now in Webkit 2.18+ it tries to re-dispose webview when glib runs exit handler cleanups.

I will need to consult with webkitgtk+ folks to find the proper proccedure of disposing of the webview in 2.18+.
Comment 8 Leo Ufimtsev CLA 2017-09-27 18:53:42 EDT
Asking webkitgtk folks for advice:
https://bugs.webkit.org/show_bug.cgi?id=177577

This one is hard to figure out... will keep investigating.
Comment 9 Eclipse Genie CLA 2017-10-03 10:54:54 EDT
New Gerrit change created: https://git.eclipse.org/r/106141
Comment 11 Leo Ufimtsev CLA 2017-10-03 11:37:02 EDT
(In reply to Eclipse Genie from comment #9)
> New Gerrit change created: https://git.eclipse.org/r/106141

After some investigation, (on 2.18) I found that the JVM only tries to g_object_unref() the *first instance* on shutdown (In an on_exit handler). It doesn't touch other instances. 

For the time being, I've merged a workaround that g_object_ref(webview) the first webview instance, which prevents the shutdown crash and doesn't introduce memory leaks ([I believe], but haven't tested with Valgrind).
For the time being, this makes webkit2 functional again and usable for production. 

For the time being I will prioritize the general Webkit2 port. Once webkit2 is default, I will come back to revisit this issue to find a better long term solution.
Comment 12 Leo Ufimtsev CLA 2017-10-03 12:44:56 EDT
I should backport to 4.7.2 also.
Comment 13 Leo Ufimtsev CLA 2017-10-12 11:39:44 EDT
I'll backport this guy to 4.7.2 soon-ish.
Comment 14 Michael Vorburger CLA 2017-10-14 06:37:12 EDT
Is Bug 526032 a dupe of this or another problem?
Comment 15 Leo Ufimtsev CLA 2017-10-16 10:41:18 EDT
(In reply to Michael Vorburger from comment #14)
> Is Bug 526032 a dupe of this or another problem?

~Possible duplicate, but I'd like to investigate first. The stack trace appears a little simmilar (crash on deference), but it's a bit different from the one seen here.
Comment 16 Michael Vorburger CLA 2017-10-16 12:56:38 EDT
*** Bug 526032 has been marked as a duplicate of this bug. ***
Comment 17 Eclipse Genie CLA 2017-10-17 14:24:33 EDT
New Gerrit change created: https://git.eclipse.org/r/110248
Comment 19 Leo Ufimtsev CLA 2017-10-17 14:32:51 EDT
(In reply to Eclipse Genie from comment #18)
> Gerrit change https://git.eclipse.org/r/110248 was merged to
> [R4_7_maintenance].
> Commit:
> http://git.eclipse.org/c/platform/eclipse.platform.swt.git/commit/
> ?id=0ad8ea80dcb3660bf19672582e7e83b95fd182eb

Backported.

To be continued after other webkit2 components were fully ported.
Comment 20 Leo Ufimtsev CLA 2017-10-31 15:30:56 EDT
Some suggestions were posted on the linked webkitgtk bugzilla tracker.

- Maybe our use of webcookies is an issue
- maybe we should create our own webcontext 
- ... few more.

I can't seem to reply to the comment as I can't seem to log in. Will try again later.
Comment 21 John Johnson CLA 2017-11-25 05:33:36 EST
I'm experiencing this issue with Eclipse 4.7.1 as well as other JVM applications, like IntelliJ Idea.  I wonder if the changes made by upstream have something to do with this.  Also, I'm not sure what that means where the responsibility is or where the bug is that needs to be fixed. 
By the way, this is on an ArchLinux installation with WebKitGTK-2.18.3-2

extra/webkit2gtk 2.18.3-2 [installed]
    GTK+ Web content engine library
Comment 22 Leo Ufimtsev CLA 2017-11-27 11:00:33 EST
(In reply to John Johnson from comment #21)
> I'm experiencing this issue with Eclipse 4.7.1 as well as other JVM
> applications, like IntelliJ Idea.  I wonder if the changes made by upstream
> have something to do with this.  Also, I'm not sure what that means where
> the responsibility is or where the bug is that needs to be fixed. 
> By the way, this is on an ArchLinux installation with WebKitGTK-2.18.3-2
> 
> extra/webkit2gtk 2.18.3-2 [installed]
>     GTK+ Web content engine library

Thank you for this report. This is rather useful.
Would you happen to have a crash/log/dump/console print from intellij?
I don't know how intelliij works internally, but I thought they used swing/gtk2 underneath, which uses webkit1 not webkit2 afaik. 

Do you still have webkitgtk<=2.4 (webkit1) installed on your archLinux box?
Comment 23 burak sarac CLA 2017-11-29 09:41:44 EST
I confirm I can reproduce issue with gtk4, Version: Oxygen.1a Release (4.7.1a)
Build id: 20171005-1200

$yum list installed \*webkit\*
Installed Packages
webkitgtk4.x86_64                                                                                                              
webkitgtk4-jsc.x86_64                                                                                                          
webkitgtk4-plugin-process-gtk2.x86_64  

trace:   

0x7f342d3ed297 /lib64/libjavascriptcoregtk-4.0.so.18(WTFCrash+0x17) [0x7f342d3ed297]
2   0x7f342dc905b1 /lib64/libwebkit2gtk-4.0.so.37(+0x6165b1) [0x7f342dc905b1]
3   0x7f342dcd5aba /lib64/libwebkit2gtk-4.0.so.37(+0x65baba) [0x7f342dcd5aba]
4   0x7f342dcd5f79 /lib64/libwebkit2gtk-4.0.so.37(+0x65bf79) [0x7f342dcd5f79]
5   0x7f342de8e620 /lib64/libwebkit2gtk-4.0.so.37(+0x814620) [0x7f342de8e620]
6   0x7f34e87d8689 /lib64/libgobject-2.0.so.0(g_object_unref+0x1b9) [0x7f34e87d8689]
7   0x7f3585b4eb88 /lib64/libc.so.6(+0x3bb88) [0x7f3585b4eb88]
8   0x7f3585b4ebda /lib64/libc.so.6(+0x3bbda) [0x7f3585b4ebda]
9   0x7f3584fc6ba7 /home/burak/java/jdk-9.0.1_linux-x64_bin/jdk-9.0.1/lib/server/libjvm.so(+0x838ba7) [0x7f3584fc6ba7]
10  0x7f35854eb019 /home/burak/java/jdk-9.0.1_linux-x64_bin/jdk-9.0.1/lib/server/libjvm.so(+0xd5d019) [0x7f35854eb019]
11  0x7f35854e8d63 /home/burak/java/jdk-9.0.1_linux-x64_bin/jdk-9.0.1/lib/server/libjvm.so(+0xd5ad63) [0x7f35854e8d63]
12  0x7f35854e92d0 /home/burak/java/jdk-9.0.1_linux-x64_bin/jdk-9.0.1/lib/server/libjvm.so(+0xd5b2d0) [0x7f35854e92d0]
13  0x7f35854e97e2 /home/burak/java/jdk-9.0.1_linux-x64_bin/jdk-9.0.1/lib/server/libjvm.so(+0xd5b7e2) [0x7f35854e97e2]
14  0x7f35852fbe12 /home/burak/java/jdk-9.0.1_linux-x64_bin/jdk-9.0.1/lib/server/libjvm.so(+0xb6de12) [0x7f35852fbe12]
15  0x7f3586312609 /lib64/libpthread.so.0(+0x7609) [0x7f3586312609]
16  0x7f3585c2ce6f /lib64/libc.so.6(clone+0x3f) [0x7f3585c2ce6f]
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007f342d3ed29c, pid=29845, tid=29868
#
# JRE version: Java(TM) SE Runtime Environment (9.0+11) (build 9.0.1+11)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (9.0.1+11, mixed mode, tiered, compressed oops, g1 gc, linux-amd64)
# Problematic frame:
# C  [libjavascriptcoregtk-4.0.so.18+0xd7729c]  WTFCrash+0x1c
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/lib/systemd/systemd-coredump %P %u %g %s %t %c %e" (or dumping to /home/burak/core.29845)
#
# An error report file with more information is saved as:
# /home/burak/hs_err_pid29845.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
Comment 24 Leo Ufimtsev CLA 2017-11-29 13:35:07 EST
(In reply to burak sarac from comment #23)
> I confirm I can reproduce issue with gtk4, Version: Oxygen.1a Release
> (4.7.1a)
> Build id: 20171005-1200

Gtk4 isn't out yet. You mean gtk3?

Backport is available in 4.7.2. (Try nightly).
Comment 25 burak sarac CLA 2017-11-30 03:43:01 EST
Hi Leo, 
 Yes might be GTK3 I just looked webkitgtk4.x86_64 then assumed it is GTK4:), thanks I will download nightly build now.
Comment 26 Martin Oberhuber CLA 2017-11-30 11:14:36 EST
I have also seen crashes on shutdown recently with backtrace like in comment 23, using Oxygen.2rc2 C/C++ IDE on Ubuntu 17.04. It doesn't crash always for me, but it has crashed on shutdown 3-4 times in the past 2 days.

/var/log/apt/history.log tells me that my libwebkit2gtk has been upgraded from 
(2.18.0-0ubuntu0.17.04.2  -->  2.18.3-0ubuntu0.17.04.1) on 2017-11-17.

Oxygen.1a had not crashed for me previously; I cannot tell if the crashes started due to Oxygen.2rc2 , or due to the libwebkit2gtk Ubuntu update.

At any rate I can say, that Oxygen.2rc2 _has been_ crashing on shutdown with libwebkitgtk-2.18.3 , though the crashes did not prevent restart of Eclipse on the same workspace.
Comment 27 Dani Megert CLA 2017-11-30 11:28:44 EST
(In reply to Martin Oberhuber from comment #26)

Can you please check with http://download.eclipse.org/eclipse/downloads/drops4/M20171130-0510/
Comment 28 Martin Oberhuber CLA 2017-11-30 12:46:07 EST
@Dani I've launched M20171130-0510 3 times, did some various small things in it (including opening a web browser view, welcome, ...) no crash on shutdown.

I do see a number of these on the console though:
GLib-CRITICAL **: g_base64_encode_step: assertion 'in != NULL' failed
Gtk-WARNING   **: Negative content width -5 (allocation 1, extents 3x3) while allocating gadget (node progressbar, owner GtkProgressBar)
Gtk-CRITICAL  **: gtk_box_gadget_distribute: assertion 'size >= 0' failed in GtkScrollbar
Comment 29 Dani Megert CLA 2017-12-04 03:27:13 EST
Can we consider this fixed? If not, please move to 4.7.3.
Comment 30 Leo Ufimtsev CLA 2017-12-04 20:20:28 EST
Fixed in 4.7.2 (& 4.8*).
Comment 31 Leo Ufimtsev CLA 2017-12-04 20:22:53 EST
Hmm, I closed by accident. (it's 8:22 pm >_>').

I've implemented a hotfix and backported to 4.7.2/master of 4.8. I should investigate the root cause of this.
Comment 32 Alexander Kurtakov CLA 2018-01-15 14:47:52 EST
Leo, what should be done here?
Comment 33 Leo Ufimtsev CLA 2018-01-16 11:23:59 EST
(In reply to Alexander Kurtakov from comment #32)
> Leo, what should be done here?

I fixed the issue via magic. (Adding an extra reference to first item, to prevent garbage collection).
It would be good for me to spend a day or two to see if there is a better fix. Otherwise close it as perm-fix.

Issue probably occurs in combination with our fixed container not taking over references probably. But this is just a wild guess.
Comment 34 Leo Ufimtsev CLA 2018-01-25 17:18:28 EST
I've spent some time investigating this.

From what I gather, in the newer webkit, they've added an exit handler that get's run on a finalization of the webkit context.
But with our custom swtfixed container, webkit is disposed before the exit handlers kick in and when they do they find a destroyed object, which leads to the crash.

There isn't really a good solution to this other than using a custom context, which would be more or less the same as just adding an extra reference to the current one (as is done in the fix).

So since there isn't much left to do in this bug. -> Closing.
Comment 35 Leo Ufimtsev CLA 2018-01-25 17:18:45 EST
Verified on newer builds.