Bug 509812 - Unable to interrupt arm remote target with Neon and GDB 7.12
Summary: Unable to interrupt arm remote target with Neon and GDB 7.12
Status: RESOLVED FIXED
Alias: None
Product: CDT
Classification: Tools
Component: cdt-debug-dsf-gdb (show other bugs)
Version: 9.2.0   Edit
Hardware: PC Linux
: P3 critical (vote)
Target Milestone: 9.3.0   Edit
Assignee: Alvaro Sanchez-Leon CLA
QA Contact: Marc Khouzam CLA
URL:
Whiteboard:
Keywords:
: 509788 (view as bug list)
Depends on:
Blocks:
 
Reported: 2017-01-02 05:17 EST by Norbert Lange CLA
Modified: 2017-06-02 04:59 EDT (History)
5 users (show)

See Also:


Attachments
Traces from the debug sessions (10.27 KB, application/gzip)
2017-01-02 05:17 EST, Norbert Lange CLA
no flags Details
Launch configurations to debug under valgrind (1.57 KB, application/gzip)
2017-01-06 09:21 EST, Norbert Lange CLA
no flags Details
trace from debugging arm target with mi-async on (3.27 KB, application/gzip)
2017-01-09 06:52 EST, Norbert Lange CLA
no flags Details
complete project to launch vgd under Hardware Debugger (4.12 KB, application/gzip)
2017-01-14 16:27 EST, Norbert Lange CLA
no flags Details
Screenshot of Ac6 STM32 debugger configuration (129.03 KB, image/png)
2017-06-01 10:44 EDT, Julien Dehaudt CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description Norbert Lange CLA 2017-01-02 05:17:53 EST
Created attachment 266085 [details]
Traces from the debug sessions

Hello,

it seems that some codepaths related to handling threads are different if gdb 7.12 is detected by eclipse.

This issue is definitely coming from eclipse and *not the gdb binary*, to demonstrate this, the following script is used to start the very same gdb, but reporting an older version to eclipse.

# cat /usr/local/bin/arm-none-eabi-gdb
----------------------------------------------------------------
#!/bin/sh
if [ "$1" = "--version" ]; then
  echo "GNU gdb (Debian 7.11-1) 7.11"
  exit
fi
# The original, un-spoofed script (its necessary to set architecture early)
exec gdb-multiarch -iex "set architecture arm" "$@"
----------------------------------------------------------------

The traces are done by:

*) Start the remote debugging
*) Allow a breakpoint to be hit (this is primary to verify its working)
*) Continue using the "Resume" Icon
*) Wait ~3 seconds and hit the "Suspend" Icon

What happens:
*) Older gdb versions (7.11.1) will halt correctly
*) The version-spoofed gdb 7.12 will halt correctly
*) The normal gdb 7.12 will not respond.


The traces show some differences in thread handling commands, I assume this is the reason for the problem.

I am using Debian Stretch AMD64 and CDT Neon.2.

This issue is a blocking one, if you have to debug embedded plattforms
Comment 1 Norbert Lange CLA 2017-01-02 05:37:38 EST
Just found this related Bug #509788
Comment 2 Marc Khouzam CLA 2017-01-05 21:57:03 EST
*** Bug 509788 has been marked as a duplicate of this bug. ***
Comment 3 Marc Khouzam CLA 2017-01-05 22:36:49 EST
The bug here is that when running GDB 7.12 we assume GDB is in mi-async mode, but that is not the case for Hardware debugging.  So, GDBRunControl_7_12#doSuspend() will send -exec-interrupt instead of a ^C, even though GDB will only accept the ^C.

We either need a new GDBJtagDSFFinalLaunchSequence_7_12 that will enable mi-async mode, or we need to send the ^C when mi-async is off.
Comment 4 Marc Khouzam CLA 2017-01-05 23:16:49 EST
A workaround that worked for me is to add the following line to your gdbinit file:

set target-async on

If you have trouble setting a gdbinit file, here is what I did:
1- go to preferences->C/C++->Debug->GDB
2- set the field "GDB Command file" to something like /tmp/gdbinit
3- add the single line
       set target-async on
   to /tmp/gdbinit
4- create a _new_ GDB Hardware launch configuration.  Because it is new it will use the preference set to /tmp/gdbinit as its init file
5- launch this new launch config

The interrupt should now work.
Comment 5 Norbert Lange CLA 2017-01-06 08:35:20 EST
Regarding your workaround, I would just adjust the script to

exec gdb-multiarch -iex "set mi-async on" -iex "set architecture arm" "$@"

I prefer not touching gdbinit for stuff thats only valid for some targets. I can`t test it until next week, but will address some of your remarks:

target-async is deprecated and has some different effect in gdb 7.7 and earlier [1]

mi-async off seems to still have some uses, doesnt seems like all targets fully support it (or older gdbs for that matter).

assuming anything doesnt sounds like a good idea, "show mi-async" will give you a clear answer.

Launching gdb probably could be improved by adding more settings already in the commandline, like "-symbols sfile -exec file" (sfile, file as configured in the startup tab). For example gdb can then automatically set the correct architecture, and I dont have to specify it with -iex for remote targets. Gonna make a new bug report about that, but it probably fixes some other issues too.

[1] https://sourceware.org/gdb/onlinedocs/gdb/Asynchronous-and-non_002dstop-modes.html
Comment 6 Norbert Lange CLA 2017-01-06 09:21:05 EST
Created attachment 266159 [details]
Launch configurations to debug under valgrind

Well, I cant test ARM Hardware, but I use the GDB Hardware Debugging Plugin for something else too.
I attached a launch configuration that
* Starts a program under valgrind
* Attaches to valgrinds vgdb server (needs GDB Hardware Debugging for this).

I enabled mi-async with the gdbinit file and the -iex argument and can verify its enabled with the show mi-async command

You should be able to test this with a simple programm.

What happen is, that even with enabled mi-async, you cant stop the program. however you can enter "interrupt" in the "Debugger Console" and it will work then, and afterwards the normal Buttons will work too.

(Again, with 7.11 and the version-spoofed 7.12 everything works from the start)

Some unrelated point: the (very nice) Full GDB Console has a drwaback you will encounter if you click the "load symbols from file" option with the attached configuration. GDB will load the symbols from the vgdb server a second time and will then ask whether the old ones should be replaced. You see this only in the traces, and need to enter "y" in the other Debugger Console to continue.
You should either display these interactions in the "Debugger Console" or disable those interactions
Comment 7 Norbert Lange CLA 2017-01-09 06:52:46 EST
Created attachment 266180 [details]
trace from debugging arm target with mi-async on

I just testet with the ARM target, same issue.
mi-async is enabled and I verified this with "show mi-async". 

* Start debugging (no breakpoint enabled)
* Trying to break into debugger using GUI - doesn`t do anything?
  in the Trace : 783,931 33-exec-interrupt --thread 1
* Break into debugger by entering "interrupt" in the Debugger Console.
* Continue using GUI
* The debugger will break again, seems like the GUI interrupt was processed now

Afterwards the GUI controls will work correctly, they also will work after a breakpoint and a GUI "resume".
The Progress also gets stuck at "98% configuring GDB"

Seems like something is amiss when starting the session.
Comment 8 Eclipse Genie CLA 2017-01-09 11:43:35 EST
New Gerrit change created: https://git.eclipse.org/r/88285
Comment 9 Norbert Lange CLA 2017-01-09 14:40:22 EST
I tested your patch, and the issue from my previous two posts still persists.

Additionally, if you detect a version < 7.12 you will unconditionally assume mi-async off. Which means if you enabled it through gdbinit or wrapper scripts (like I did), you will send the wrong "interrupt" command again. I couldnt stop my program in this case.

Inspecting the value of mi-async (on GDB Versions that support it) after the init scripts and commands are run would be way more robust.
Comment 10 Alvaro Sanchez-Leon CLA 2017-01-10 14:36:32 EST
(In reply to Norbert Lange from comment #9)
> I tested your patch, and the issue from my previous two posts still persists.
> 
I have setup the valgrind case using a "GDB Hardware debug" launch, 
I am able to see the issue before the Gerrit change but not after, 
I am not using any .gdbinit file though,  anything thoughts on what I may be missing ?


> Additionally, if you detect a version < 7.12 you will unconditionally assume
> mi-async off. Which means if you enabled it through gdbinit or wrapper
> scripts (like I did), you will send the wrong "interrupt" command again. I
> couldnt stop my program in this case.

I see your point, we shall address this issue on top of the original problem though as this imply further changes besides the specifics of the GDB Hardware debug.
Comment 11 Norbert Lange CLA 2017-01-11 15:49:21 EST
(In reply to Alvaro Sanchez-Leon from comment #10)
> (In reply to Norbert Lange from comment #9)
> > I tested your patch, and the issue from my previous two posts still persists.
> > 
> I have setup the valgrind case using a "GDB Hardware debug" launch, 
> I am able to see the issue before the Gerrit change but not after, 
> I am not using any .gdbinit file though,  anything thoughts on what I may be
> missing ?

Sorry, no. I just verified that I did not mess anything up, pretty much everything is the default.
Do you have some fixes I am missing maybe?

> 
> > Additionally, if you detect a version < 7.12 you will unconditionally assume
> > mi-async off. Which means if you enabled it through gdbinit or wrapper
> > scripts (like I did), you will send the wrong "interrupt" command again. I
> > couldnt stop my program in this case.
> 
> I see your point, we shall address this issue on top of the original problem
> though as this imply further changes besides the specifics of the GDB
> Hardware debug.

Ok great, just wanted to make sure this point is not getting lost.
Comment 12 Alvaro Sanchez-Leon CLA 2017-01-12 08:32:36 EST
(In reply to Norbert Lange from comment #11)
> Do you have some fixes I am missing maybe?
> 
Nope, I don't have additional fixes, 
I am using the patch set commit directly from Gerrit,
I have just tested it with patch set 2 and patch set 4 and it works fine, 

The GDB client I am using is the actual GNU GDB 7.12, 
valgrind version: valgrind-3.10.1

I am setting vgdb as follows:

> valgrind --tool=memcheck --vgdb=yes --vgdb-error=0 ./Prog
> vgdb --port=1234

and then using a GDB HW debug launch from Eclipse pointing to localhost:1234, 
 
Let me know if you notice any difference with your environment,
Comment 13 Norbert Lange CLA 2017-01-14 16:27:27 EST
Created attachment 266306 [details]
complete project to launch vgd under Hardware Debugger

I added a self-contained Project for testing. I use the serial mode of vgdb, but it din`t make a difference using TCP.
Please import this in a new workpace for testing.

Have you enabled the "resume" checkbox in the startup tab?
Maybe you are using Non-Stop Mode?
Comment 14 Alvaro Sanchez-Leon CLA 2017-01-16 14:28:29 EST
(In reply to Norbert Lange from comment #13)
> Have you enabled the "resume" checkbox in the startup tab?
   
I was not using the "resume" option so I was able to reproduce the problem, 

This resume option was translated to the use of the GDB CLI command 'continue', 
which causes GDB to block even if it is set to use mi-async.
  When using the new Console we rely on the use of MI -exec-interrupt but GDB needs to be in a state where it can receive commands.

So when using mi-async, we shouldn't use CLI 'continue' but the corresponding mi -exec-continue
Comment 15 Alvaro Sanchez-Leon CLA 2017-01-17 07:09:52 EST
(In reply to Alvaro Sanchez-Leon from comment #14)
Patch set 6 is now available and addresses the issue related to the use of 'continue',
Comment 16 Norbert Lange CLA 2017-01-17 11:21:35 EST
That fixed it, thanks.
Comment 18 Marc Khouzam CLA 2017-01-17 12:54:03 EST
We should probably fix this in the 9.2 branch, but we'll need a simpler fix that does not add any API
Comment 19 Eclipse Genie CLA 2017-01-17 13:48:57 EST
New Gerrit change created: https://git.eclipse.org/r/88890
Comment 21 Julien Dehaudt CLA 2017-05-04 11:18:09 EDT
(In reply to Norbert Lange from comment #7)
> Created attachment 266180 [details]
> trace from debugging arm target with mi-async on
> 
> I just testet with the ARM target, same issue.
> mi-async is enabled and I verified this with "show mi-async". 
> 
> * Start debugging (no breakpoint enabled)
> * Trying to break into debugger using GUI - doesn`t do anything?
>   in the Trace : 783,931 33-exec-interrupt --thread 1
> * Break into debugger by entering "interrupt" in the Debugger Console.
> * Continue using GUI
> * The debugger will break again, seems like the GUI interrupt was processed
> now
> 
> Afterwards the GUI controls will work correctly, they also will work after a
> breakpoint and a GUI "resume".
> The Progress also gets stuck at "98% configuring GDB"
> 
> Seems like something is amiss when starting the session.

I'm facing exactly the same issue with ARM GNU GDB 7.12.1 (2017-q1) and CDT 9.2.1.
Did I miss something? The bug is marked as resolved and seems to be merged into CDT 9.2.

Thanks by advance.
Comment 22 Freddie Chopin CLA 2017-05-04 11:31:44 EDT
You need to update "GDB Hardware Debugging" plugin.
Comment 23 Julien Dehaudt CLA 2017-05-05 02:54:46 EDT
I'm on version 9.2.1.201704050430 for this feature, and no updates are available.

Related plugins versions are:
org.eclipse.cdt.debug.gdbjtag        7.0.0.201704050430
org.eclipse.cdt.debug.gdbjtag.core   9.0.0.201704050430
org.eclipse.cdt.debug.gdbjtag.ui     8.0.0.201704050430
Comment 24 Freddie Chopin CLA 2017-05-05 03:53:32 EDT
It worked for me... I just updated Eclipse, manually updated "GDB Hardware Debugging" plugin and the debugger now works fine. However I'm using my own toolchain, not the one provided by ARM - https://github.com/FreddieChopin/bleeding-edge-toolchain It also has GDB 7.12.1, so I don't think this makes any difference. 

Maybe try with a new (clean) version of Eclipse?
Comment 25 Julien Dehaudt CLA 2017-05-05 09:56:19 EDT
I had read your contribution about this toolchain on ST Community and planned to try it, this is the right time :)

...1/2 day build later

Indeed this works on Windows with ARM and your toolchains.

However I forgot to mention that I'm on Linux (tested Ubuntu 16.04 and 14.04) and things are really different.
I'm not not able to get it working with both packages, nothing happen on suspend and the following message is displayed into the console:

!ENTRY org.eclipse.cdt.dsf 4 10005 2017-05-05 15:46:13.744
!MESSAGE Request for monitor: 'RequestMonitor (org.eclipse.cdt.dsf.concurrent.RequestMonitor@7db7f53): Status ERROR: org.eclipse.cdt.dsf.gdb code=10004 Suspend operation timeout. null' resulted in an error.
!SUBENTRY 1 org.eclipse.cdt.dsf.gdb 4 10004 2017-05-05 15:46:13.744
!MESSAGE Suspend operation timeout.

Any idea to investigate?
Comment 26 Freddie Chopin CLA 2017-05-05 10:40:29 EDT
Please note that I'm using Linux too (Arch), so I actually don't know how things work on Windows, as I did not check it yet with the most recent version of Eclipse. I've only checked the toolchain with Neon.1 a few days ago.

Anyway - as you imagine I use this toolchain with most recent Eclipse (Neon.3) and things are working fine here... Have you tried to set-up a clean Eclipse version? You may do that directly in your home folder, without package manager - extract Eclipse, install the required plugin and check whether it works.

Or maybe you use some strange configuration option in your debugger configuration? Could you post screenshots of the way your debugger is configured?

In "Installed software" I got:
  C/C++ GDB Hardware Debugging	9.2.1.201704050430	org.eclipse.cdt.debug.gdbjtag.feature.group	Eclipse CDT

Under "Plug-ins" I see the same thing as you posted.
Comment 27 Julien Dehaudt CLA 2017-06-01 10:44:42 EDT
Created attachment 268705 [details]
Screenshot of Ac6 STM32 debugger configuration

Sorry for this huge delay to reply, reasons are outside my control :)

I tried what you suggested, Neon.3 + up-to-date version of "GDB Hardware Debugging" feature + ARM GNU toolchain or your own. Things still don't work from toolbar actions (resume/suspend) on Linux only, and work like a charm on Windows and Mac.

But I've recently noticed a working use-case that might guide and help to locate the issue; doing the resume through the Debugger Console (-exec-continue) allows to do the suspend through the toolbar.

Also in attachment a screenshot of my debugger configuration tab, hope it could help.
Comment 28 Freddie Chopin CLA 2017-06-01 11:21:15 EDT
> Also in attachment a screenshot of my debugger configuration tab, hope it could help.

You should be using "GDB Hardware Debugging" configuration, not the one from Ac6.
Comment 29 Julien Dehaudt CLA 2017-06-02 04:59:22 EDT
And ... it's work like a charm, so there is for sure a glitch in the Ac6 layer.

Thanks Freddie for your support, and sorry for the noise.