Bugzilla will undergo maintenance 2024-03-29 18h00 CET. Bugzilla will be placed in read-only mode at that time.

Some Eclipse Foundation services are deprecated, or will be soon. Please ensure you've read this important communication.
Bug 340300 - Parallel build using generated makefile doesn't manage dependencies
Summary: Parallel build using generated makefile doesn't manage dependencies
Status: RESOLVED FIXED
Alias: None
Product: CDT
Classification: Tools
Component: cdt-build (show other bugs)
Version: 8.0   Edit
Hardware: PC Linux
: P3 minor with 6 votes (vote)
Target Milestone: 9.3.0   Edit
Assignee: Morten Kristiansen CLA
QA Contact: Andrew Gvozdev CLA
URL:
Whiteboard:
Keywords:
: 419428 501065 (view as bug list)
Depends on:
Blocks:
 
Reported: 2011-03-17 07:09 EDT by dpanth3r CLA
Modified: 2021-08-19 17:34 EDT (History)
8 users (show)

See Also:


Attachments
Various adjustments related to pre and post steps (7.42 KB, patch)
2014-04-09 08:51 EDT, Raphael Zulliger CLA
no flags Details | Diff
makefile demonstrating pre-build dependency insufficiencies (524 bytes, application/octet-stream)
2016-12-31 02:41 EST, Morten Kristiansen CLA
no flags Details

Note You need to log in before you can comment on or make changes to this bug.
Description dpanth3r CLA 2011-03-17 07:09:53 EDT
Build Identifier: 20110204-0611

When generating makefiles and compiling using parallel build, Eclipse CDT runs :

make -j3 pre-build main-build (as I use 3 parallel jobs)

However, I sometimes get errors, and when opening the makefile I could see that the “main-build” doesn't have any dependency on “pre-build”

So both “pre” and “main” are running in parallel... which pretty much ruins the whole concept of a “pre-build step”
(in my case, generating precompiled headers before compiling the project)


Workaround : Not using parallel build, or being lucky with a pre-build step that is faster :)

Inconveniance : Not being able to use parallel build on large projects :/

Reproducible: Always

Steps to Reproduce:
1. Set up a project to have a pre-build step (preferrably upon which the main build is dependant)
2. Set up the behaviour to use parallel compiling jobs
3. Build the project
Comment 1 Robert Bogomip CLA 2012-09-25 05:17:40 EDT
Why is this such low priority? Parallel builds with pre-build steps are _broken_.

Workaround:
Add this line to your makefile.targets (create it if it does not exist) in the project folder.

    main-build: | pre-build

This specifies pre-build as an order-only pre-requisite of main-build. Make will ensure pre-build is complete /before/ moving on to main-build. The vertical bar is important.
Comment 2 Raphael Zulliger CLA 2014-04-09 08:42:07 EDT
(In reply to Robert Bogomip from comment #1)
> Workaround:
> Add this line to your makefile.targets (create it if it does not exist) in
> the project folder.
> 
>     main-build: | pre-build
> 
> This specifies pre-build as an order-only pre-requisite of main-build. Make
> will ensure pre-build is complete /before/ moving on to main-build. The
> vertical bar is important.

Are you sure this works? It didn't in my case (invoking 'make -j 4 -r -s all')
Comment 3 Raphael Zulliger CLA 2014-04-09 08:51:49 EDT
Created attachment 241784 [details]
Various adjustments related to pre and post steps

This patch is most probably not exactly what could be used by mainstream CDT, because it changes behaviour of pre and post build step handling by:
 - The post build step is also executed even if nothing changes
 - A failing pre or post build step causes a build failure
 - The post build step will be executed after the "main targets" have been built *AND* any "secondary output" has been generated

=> That's why I didn't push this patch to Gerrit
=> Nevertheless, I thought this patch could be useful for others encountering the bug described here...

Beside the above changes, the patch should fix the issues with pre and post build steps which are not really executed "pre" and "post" the main build, as complained in this bug report. With this patch applied, the main build steps should really only start executing after the pre build step has completed, regardless of how long it takes and regardless of whether you use 'make -j' or just 'make'. Also, the post build step is executed after the "main" *and* "secondary" outputs have been generated. Original CDT executes it regardless of whether the "secondary outputs" have been generated or not. Which is a bug in my opinion.

One more thing: I'm not a 'makefile' guru... Improvement suggestions would be very welcome.
Comment 4 Jens Haustedt CLA 2015-05-06 12:26:25 EDT
There is an easy workaround for a pre-build step, which is executed as a system command:
Instead of setting the prebuild command as is described in the bug, simply create a Builder which itself calls the command with the right parameters etc.
Put this preBuild builder on top of the CDT builder.
Thus ecplise is forced to execute the prebuild builder and wait for its execution before calling the CDT builder.

As far as my experiments show, the bug does not affect the configured post build steps.
Comment 5 Raphael Zulliger CLA 2015-05-07 01:02:14 EDT
I also once used such builders. If I remember correctly, one drawback of them is that you don't have access to various CDT related env-vars. I'm really not 100% sure any more, but I think to remember that, among others, you don't have access to the "Configuration name" during the build step. Thus, using builders may only help if no such variables need to be passed to the script.
Comment 6 Eclipse Genie CLA 2016-12-25 10:06:57 EST
New Gerrit change created: https://git.eclipse.org/r/87732
Comment 7 Morten Kristiansen CLA 2016-12-30 17:17:57 EST
The problem is that gmake will attempt to build pre-build in parallel to compilation of source files. Even though main-build depends on pre-build, any steps leading up to main-build does not depend on pre-build and hence get built before pre-build is completed.

I have submitted a patch that executes pre-build and main-build as two separate steps. It also changes how make is invoked. Previously it would do "make all" if you don't specify pre-build task, and "make pre-build main-build" if you do. In project settings you can change the target from all to anything you like, except it gets ignored if you have pre-build tasks. 

The change I submitted makes it call "make all" (or whatever you configured) regardless of any pre-build tasks.
Comment 8 Jonah Graham CLA 2016-12-30 18:11:01 EST
Hi Morten,

Thanks for submitting a patch to gerrit on this. As your solution is slightly different to the previous discussions I would be grateful for a bit of extra details/confirmation as this bug and the discussions will help document for future readers.

(In reply to Morten Kristiansen from comment #7)
> The problem is that gmake will attempt to build pre-build in parallel to
> compilation of source files. Even though main-build depends on pre-build,
> any steps leading up to main-build does not depend on pre-build and hence
> get built before pre-build is completed.
>
> I have submitted a patch that executes pre-build and main-build as two
> separate steps. It also changes how make is invoked. Previously it would do
> "make all" if you don't specify pre-build task, and "make pre-build
> main-build" if you do. In project settings you can change the target from
> all to anything you like, except it gets ignored if you have pre-build
> tasks. 

Why have the all target do two make calls instead of making main-build depend on pre-build? The previous discussion in this thread have all gone down the path of adding a such a dependency. Conceptually does main-build depend on pre-build, or are we talking about two separate steps? The current generator does not have such a dependency, which inclines me to believe that your proposal is the correct one in terms of not breaking backwards compatibility. Thus continuing to allow an invocation of "make main-build" to run without the pre-build steps (does anyone actually do this as it would have to be done outside Eclipse?). 


> The change I submitted makes it call "make all" (or whatever you configured)
> regardless of any pre-build tasks.
Great. We'll have to make sure this is listed in the N&N when the time comes.
Comment 9 Morten Kristiansen CLA 2016-12-31 02:41:44 EST
Created attachment 266080 [details]
makefile demonstrating pre-build dependency insufficiencies
Comment 10 Morten Kristiansen CLA 2016-12-31 02:45:03 EST
(In reply to Jonah Graham from comment #8)
Hi Jonah and happy Newyear

> Thanks for submitting a patch to gerrit on this. As your solution is
> slightly different to the previous discussions I would be grateful for a bit
> of extra details/confirmation as this bug and the discussions will help
> document for future readers.
I'll do my very best. 

> Why have the all target do two make calls instead of making main-build
> depend on pre-build? ...
IMHO the previously suggested dependencies does not solve the problem. Let me use an example to illustrate this:

1. My pre-build task generates a header file, say config.h
2. A source file, main.c includes that header file.
3. In the current implementation, Eclipse does make -j 8 pre-build main-build
4. Make starts to do pre-build
5. It can't do main-build since it depends on pre-build, but it can compile main.c to main.o, since main.c does NOT depend on pre-build. 
6. Compilation fails, or even worse, completes with an incorrect config.h

I've created an attachement demonstrating this.

I've tried googling for other ways to tell make that "generally do things in parallel; but these particular two things should be done sequential". I've found two solutions:

1. The proposed solution where make is invoked recursively
2. Make all sources depend on pre-build.

While solution two from a make-perspective could be somewhat more correct; I find it to cumbersome to implement in Eclipse. This would be something you'd have to carry around all over the code. 

>The previous discussion in this thread have all gone
> down the path of adding a such a dependency. Conceptually does main-build
> depend on pre-build, or are we talking about two separate steps? The current

I've tried adding that dependency; but it doesn't solve the problem as make is clever enough to figure out that files even though it can't build main-build before pre-build, it can build main-builds dependencies. 

> generator does not have such a dependency, which inclines me to believe that
> your proposal is the correct one in terms of not breaking backwards
> compatibility. Thus continuing to allow an invocation of "make main-build"
> to run without the pre-build steps (does anyone actually do this as it would
> have to be done outside Eclipse?). 

I haven't heard of anyone doing that, but you definitely have a point there. For the scenario I've depicted, the make result would be:

* make all
  Current: Works, as things are built sequentially
  With dependency: Same
  New proposal: Same

* make -j all
  Current: Fails as main.c is built before config.h is updated
  With dependency: Fails as main.c is built before config.h is updated
  New proposal: Works

* make [-j] pre-build
  All works the same

* make [-j] main-build
  All works the same

> > The change I submitted makes it call "make all" (or whatever you configured)
> > regardless of any pre-build tasks.
> Great. We'll have to make sure this is listed in the N&N when the time comes.
Yeah, that had me baffled for a while.
Comment 11 Morten Kristiansen CLA 2016-12-31 02:47:07 EST
Comment on attachment 266080 [details]
makefile demonstrating pre-build dependency insufficiencies

This example demonstrates that dependencies are insufficent
to handle pre-build parallel issues.

If you run make clean followed by make -j, the build will fail
because main.o does not depend on pre-build
Comment 12 Morten Kristiansen CLA 2016-12-31 02:51:46 EST
Ups, got sent a little prematurely.

For the scenario I've depicted, the make result would be:

* make all
  Current: Works, as things are built sequentially
  With dependency: Same
  New proposal: Same

* make -j all
  Current: Fails as main.c is built before config.h is updated
  With dependency: Fails as main.c is built before config.h is updated
  New proposal: Works

* make [-j] pre-build
  All works the same

* make main-build
  Current: Builds only main-build
  With dependency: Build pre-build as well
  New proposal: As Current

* make -j main-build
  Current: Builds only main-build
  With dependency: Build pre-build, and fails
  New proposal: As Current
Comment 13 Jonah Graham CLA 2017-01-03 07:14:35 EST
Hi Morten,

Thank you for the extended example. It perfectly explains the problem with the previous discussions.

Jonah
Comment 15 Jonah Graham CLA 2017-01-03 07:30:19 EST
Hi Morten,

Thank you for the fix for this. Can you add something to the N&N for CDT 9.3: https://wiki.eclipse.org/CDT/User/NewIn93#Build as this is an important fix.

Jonah
Comment 16 Morten Kristiansen CLA 2017-01-03 07:39:44 EST
I'll see what I can do; but it may become weekend before I get it done :-)
Comment 17 Morten Kristiansen CLA 2017-01-03 14:48:15 EST
Until CDT 9.3 is out, the following workaround can be used:


1. Add a makefile.init in the project root with the following content:
   allwithpre : 
       $(MAKE) --no-print-directory pre-build
       $(MAKE) --no-print-directory main-build
2. Remove the pre-build task from the configuration
3. In the project settings, C/C++ Build -> Behavior -> Workbench Build
   Behavior, Change "all" to "allwithpre"
Comment 18 Jonah Graham CLA 2021-07-08 11:01:30 EDT
*** Bug 419428 has been marked as a duplicate of this bug. ***
Comment 19 Jonah Graham CLA 2021-08-19 17:34:02 EDT
*** Bug 501065 has been marked as a duplicate of this bug. ***