Community
Participate
Working Groups
Build Identifier: 20110204-0611 When generating makefiles and compiling using parallel build, Eclipse CDT runs : make -j3 pre-build main-build (as I use 3 parallel jobs) However, I sometimes get errors, and when opening the makefile I could see that the “main-build” doesn't have any dependency on “pre-build” So both “pre” and “main” are running in parallel... which pretty much ruins the whole concept of a “pre-build step” (in my case, generating precompiled headers before compiling the project) Workaround : Not using parallel build, or being lucky with a pre-build step that is faster :) Inconveniance : Not being able to use parallel build on large projects :/ Reproducible: Always Steps to Reproduce: 1. Set up a project to have a pre-build step (preferrably upon which the main build is dependant) 2. Set up the behaviour to use parallel compiling jobs 3. Build the project
Why is this such low priority? Parallel builds with pre-build steps are _broken_. Workaround: Add this line to your makefile.targets (create it if it does not exist) in the project folder. main-build: | pre-build This specifies pre-build as an order-only pre-requisite of main-build. Make will ensure pre-build is complete /before/ moving on to main-build. The vertical bar is important.
(In reply to Robert Bogomip from comment #1) > Workaround: > Add this line to your makefile.targets (create it if it does not exist) in > the project folder. > > main-build: | pre-build > > This specifies pre-build as an order-only pre-requisite of main-build. Make > will ensure pre-build is complete /before/ moving on to main-build. The > vertical bar is important. Are you sure this works? It didn't in my case (invoking 'make -j 4 -r -s all')
Created attachment 241784 [details] Various adjustments related to pre and post steps This patch is most probably not exactly what could be used by mainstream CDT, because it changes behaviour of pre and post build step handling by: - The post build step is also executed even if nothing changes - A failing pre or post build step causes a build failure - The post build step will be executed after the "main targets" have been built *AND* any "secondary output" has been generated => That's why I didn't push this patch to Gerrit => Nevertheless, I thought this patch could be useful for others encountering the bug described here... Beside the above changes, the patch should fix the issues with pre and post build steps which are not really executed "pre" and "post" the main build, as complained in this bug report. With this patch applied, the main build steps should really only start executing after the pre build step has completed, regardless of how long it takes and regardless of whether you use 'make -j' or just 'make'. Also, the post build step is executed after the "main" *and* "secondary" outputs have been generated. Original CDT executes it regardless of whether the "secondary outputs" have been generated or not. Which is a bug in my opinion. One more thing: I'm not a 'makefile' guru... Improvement suggestions would be very welcome.
There is an easy workaround for a pre-build step, which is executed as a system command: Instead of setting the prebuild command as is described in the bug, simply create a Builder which itself calls the command with the right parameters etc. Put this preBuild builder on top of the CDT builder. Thus ecplise is forced to execute the prebuild builder and wait for its execution before calling the CDT builder. As far as my experiments show, the bug does not affect the configured post build steps.
I also once used such builders. If I remember correctly, one drawback of them is that you don't have access to various CDT related env-vars. I'm really not 100% sure any more, but I think to remember that, among others, you don't have access to the "Configuration name" during the build step. Thus, using builders may only help if no such variables need to be passed to the script.
New Gerrit change created: https://git.eclipse.org/r/87732
The problem is that gmake will attempt to build pre-build in parallel to compilation of source files. Even though main-build depends on pre-build, any steps leading up to main-build does not depend on pre-build and hence get built before pre-build is completed. I have submitted a patch that executes pre-build and main-build as two separate steps. It also changes how make is invoked. Previously it would do "make all" if you don't specify pre-build task, and "make pre-build main-build" if you do. In project settings you can change the target from all to anything you like, except it gets ignored if you have pre-build tasks. The change I submitted makes it call "make all" (or whatever you configured) regardless of any pre-build tasks.
Hi Morten, Thanks for submitting a patch to gerrit on this. As your solution is slightly different to the previous discussions I would be grateful for a bit of extra details/confirmation as this bug and the discussions will help document for future readers. (In reply to Morten Kristiansen from comment #7) > The problem is that gmake will attempt to build pre-build in parallel to > compilation of source files. Even though main-build depends on pre-build, > any steps leading up to main-build does not depend on pre-build and hence > get built before pre-build is completed. > > I have submitted a patch that executes pre-build and main-build as two > separate steps. It also changes how make is invoked. Previously it would do > "make all" if you don't specify pre-build task, and "make pre-build > main-build" if you do. In project settings you can change the target from > all to anything you like, except it gets ignored if you have pre-build > tasks. Why have the all target do two make calls instead of making main-build depend on pre-build? The previous discussion in this thread have all gone down the path of adding a such a dependency. Conceptually does main-build depend on pre-build, or are we talking about two separate steps? The current generator does not have such a dependency, which inclines me to believe that your proposal is the correct one in terms of not breaking backwards compatibility. Thus continuing to allow an invocation of "make main-build" to run without the pre-build steps (does anyone actually do this as it would have to be done outside Eclipse?). > The change I submitted makes it call "make all" (or whatever you configured) > regardless of any pre-build tasks. Great. We'll have to make sure this is listed in the N&N when the time comes.
Created attachment 266080 [details] makefile demonstrating pre-build dependency insufficiencies
(In reply to Jonah Graham from comment #8) Hi Jonah and happy Newyear > Thanks for submitting a patch to gerrit on this. As your solution is > slightly different to the previous discussions I would be grateful for a bit > of extra details/confirmation as this bug and the discussions will help > document for future readers. I'll do my very best. > Why have the all target do two make calls instead of making main-build > depend on pre-build? ... IMHO the previously suggested dependencies does not solve the problem. Let me use an example to illustrate this: 1. My pre-build task generates a header file, say config.h 2. A source file, main.c includes that header file. 3. In the current implementation, Eclipse does make -j 8 pre-build main-build 4. Make starts to do pre-build 5. It can't do main-build since it depends on pre-build, but it can compile main.c to main.o, since main.c does NOT depend on pre-build. 6. Compilation fails, or even worse, completes with an incorrect config.h I've created an attachement demonstrating this. I've tried googling for other ways to tell make that "generally do things in parallel; but these particular two things should be done sequential". I've found two solutions: 1. The proposed solution where make is invoked recursively 2. Make all sources depend on pre-build. While solution two from a make-perspective could be somewhat more correct; I find it to cumbersome to implement in Eclipse. This would be something you'd have to carry around all over the code. >The previous discussion in this thread have all gone > down the path of adding a such a dependency. Conceptually does main-build > depend on pre-build, or are we talking about two separate steps? The current I've tried adding that dependency; but it doesn't solve the problem as make is clever enough to figure out that files even though it can't build main-build before pre-build, it can build main-builds dependencies. > generator does not have such a dependency, which inclines me to believe that > your proposal is the correct one in terms of not breaking backwards > compatibility. Thus continuing to allow an invocation of "make main-build" > to run without the pre-build steps (does anyone actually do this as it would > have to be done outside Eclipse?). I haven't heard of anyone doing that, but you definitely have a point there. For the scenario I've depicted, the make result would be: * make all Current: Works, as things are built sequentially With dependency: Same New proposal: Same * make -j all Current: Fails as main.c is built before config.h is updated With dependency: Fails as main.c is built before config.h is updated New proposal: Works * make [-j] pre-build All works the same * make [-j] main-build All works the same > > The change I submitted makes it call "make all" (or whatever you configured) > > regardless of any pre-build tasks. > Great. We'll have to make sure this is listed in the N&N when the time comes. Yeah, that had me baffled for a while.
Comment on attachment 266080 [details] makefile demonstrating pre-build dependency insufficiencies This example demonstrates that dependencies are insufficent to handle pre-build parallel issues. If you run make clean followed by make -j, the build will fail because main.o does not depend on pre-build
Ups, got sent a little prematurely. For the scenario I've depicted, the make result would be: * make all Current: Works, as things are built sequentially With dependency: Same New proposal: Same * make -j all Current: Fails as main.c is built before config.h is updated With dependency: Fails as main.c is built before config.h is updated New proposal: Works * make [-j] pre-build All works the same * make main-build Current: Builds only main-build With dependency: Build pre-build as well New proposal: As Current * make -j main-build Current: Builds only main-build With dependency: Build pre-build, and fails New proposal: As Current
Hi Morten, Thank you for the extended example. It perfectly explains the problem with the previous discussions. Jonah
Gerrit change https://git.eclipse.org/r/87732 was merged to [master]. Commit: http://git.eclipse.org/c/cdt/org.eclipse.cdt.git/commit/?id=10428dd53afc4d794d711cdde9e970ca9b9bf732
Hi Morten, Thank you for the fix for this. Can you add something to the N&N for CDT 9.3: https://wiki.eclipse.org/CDT/User/NewIn93#Build as this is an important fix. Jonah
I'll see what I can do; but it may become weekend before I get it done :-)
Until CDT 9.3 is out, the following workaround can be used: 1. Add a makefile.init in the project root with the following content: allwithpre : $(MAKE) --no-print-directory pre-build $(MAKE) --no-print-directory main-build 2. Remove the pre-build task from the configuration 3. In the project settings, C/C++ Build -> Behavior -> Workbench Build Behavior, Change "all" to "allwithpre"
*** Bug 419428 has been marked as a duplicate of this bug. ***
*** Bug 501065 has been marked as a duplicate of this bug. ***