Bug 336262 - Hudson or Jenkins, what to do with the CI instance at eclipse.org
Summary: Hudson or Jenkins, what to do with the CI instance at eclipse.org
Status: CLOSED FIXED
Alias: None
Product: Community
Classification: Eclipse Foundation
Component: CI-Jenkins (show other bugs)
Version: unspecified   Edit
Hardware: All All
: P3 normal (vote)
Target Milestone: ---   Edit
Assignee: Eclipse Webmaster CLA
QA Contact:
URL:
Whiteboard:
Keywords:
Depends on: 461307 461309 461310 472322 480657 484914 461174
Blocks: 472425 474406 418213 420937 486460
  Show dependency tree
 
Reported: 2011-02-03 12:01 EST by Remy Suen CLA
Modified: 2017-11-20 14:20 EST (History)
42 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Remy Suen CLA 2011-02-03 12:01:29 EST
We need to ask ourselves the same question that was posed to the Apache mailing list recently.
http://mail-archives.apache.org/mod_mbox/www-builds/201102.mbox/%3CAANLkTikCymNt_HxHSC9-JJfK6U+LSs8BOf2j_-unhjXw@mail.gmail.com%3E

1. Do we go with Hudson or Jenkins?
2. If we go with Jenkins, do we then rename hudson.eclipse.org to jenkins.eclipse.org?

If we opt to go for Jenkins, I think we should still maintain status quo for now until Indigo goes GA and/or when the current Hudson instance needs to be brought down and updated [for whatever reasons the webmasters deem appropriate]. No point causing more work for the webmasters then is necessary.
Comment 1 Chris Aniszczyk CLA 2011-02-03 12:03:38 EST
I think we should move to Jenkins and do it post Indigo.

This allows time for the "dust" to settle :)
Comment 2 Manuel Doninger CLA 2011-03-13 14:41:49 EDT
I would also vote for Jenkins.
Comment 3 Denis Roy CLA 2011-08-12 14:15:34 EDT
Since Hudson is now an Eclipse project, we'll eat our own dog food.
Comment 4 Doug Schaefer CLA 2015-02-17 10:48:46 EST
(In reply to Denis Roy from comment #3)
> Since Hudson is now an Eclipse project, we'll eat our own dog food.

While that seemed logical at the time, Jenkins has clearly won the CI wars. We're missing out on a lot of plug-ins that appear to be Jenkins only.

As an example, we've been struggling with the Andmore instance getting the UI tests not to hang. I wanted to try the Xvnc recorder to see if we can see what was happening. No dice on Hudson.

I also wonder if Jenkins scales better. I have first hand knowledge of a massive Jenkins instance that had hundreds if not thousands of jobs. All the real work is supposed to be done in slaves anyway. Then we could get rid of this HIPP craziness.

Reopening to see if this can get any traction.
Comment 5 Denis Roy CLA 2015-02-17 10:55:03 EST
(In reply to Doug Schaefer from comment #4)
> I also wonder if Jenkins scales better. I have first hand knowledge of a
> massive Jenkins instance that had hundreds if not thousands of jobs. All the
> real work is supposed to be done in slaves anyway. Then we could get rid of
> this HIPP craziness.

Where is this massive Jenkins anyway?  URL please, I want to poke around.
Comment 6 Doug Schaefer CLA 2015-02-17 11:15:09 EST
(In reply to Denis Roy from comment #5)
> (In reply to Doug Schaefer from comment #4)
> > I also wonder if Jenkins scales better. I have first hand knowledge of a
> > massive Jenkins instance that had hundreds if not thousands of jobs. All the
> > real work is supposed to be done in slaves anyway. Then we could get rid of
> > this HIPP craziness.
> 
> Where is this massive Jenkins anyway?  URL please, I want to poke around.

http://jenkins.rim.net, but you probably can't get to it :). And most of the projects there have moved over to Electric Commander anyway. We have another one here at QNX that has about 600 jobs running from it. Again, the key is to have enough slaves to actually do the builds.

That's another advantage a single Jenkins instance has, it can load balance the builds to your slave pool. If every project has it's own instance, it's hard to share resources.
Comment 7 Denis Roy CLA 2015-02-17 11:29:14 EST
> http://jenkins.rim.net, but you probably can't get to it :)

Massive Jenkins infra performs well when it's sheltered behind corporate firewall.

Apples != Oranges. Gotcha.




> That's another advantage a single Jenkins instance has

There's no arguing the advantages (and disadvantages) of a single instance. Scalability is the only reason we went with HIPP. That's all discussed in the HIPP bug.
Comment 8 Doug Schaefer CLA 2015-02-17 11:37:39 EST
(In reply to Denis Roy from comment #7)
> > http://jenkins.rim.net, but you probably can't get to it :)
> 
> Massive Jenkins infra performs well when it's sheltered behind corporate
> firewall.
> 
> Apples != Oranges. Gotcha.

Well, certainly Eclipse's problems aren't unique. Here's a pretty good example:

https://jenkins.qa.ubuntu.com/view/All/

taken from the list found here:

https://wiki.jenkins-ci.org/pages/viewpage.action?pageId=58001258

There's no Gotcha to get. Just trying to make things better for myself and my fellow committers.
Comment 9 Julien Vermillard CLA 2015-02-17 13:39:31 EST
I think I really hate Hudson, but I hate also jenkins:
it's unstable, bloated and the JSON API is a joke.
builds.apache.org which is based on jenkins is always broken in a way or in another.

My jenkins instance at work is always flappy once a week and need a reboot..

I used drone.io and it worked like a charm and it's simplicity is a breath.
Just scheduling job inside docker container.
https://github.com/drone/drone
It's linux only but it's rock solid. It's supporting java and dozen of other languages.
Comment 10 Doug Schaefer CLA 2015-02-17 16:12:33 EST
(In reply to Julien Vermillard from comment #9)
> I think I really hate Hudson, but I hate also jenkins:
> it's unstable, bloated and the JSON API is a joke.
> builds.apache.org which is based on jenkins is always broken in a way or in
> another.
> 
> My jenkins instance at work is always flappy once a week and need a reboot..

You mileage may vary I guess. We used to have issues with our Jenkins but that was mainly our fault and the crazy nfs mounts we were trying to do. Since we simplified our structure, it's been rock solid.

> I used drone.io and it worked like a charm and it's simplicity is a breath.
> Just scheduling job inside docker container.
> https://github.com/drone/drone
> It's linux only but it's rock solid. It's supporting java and dozen of other
> languages.

Jenkins supports jobs in docker containers.

Part of the reason for re-opening this bug was to try and follow where the community went. I can't tell if there's as big a community behind drone.
Comment 11 Doug Schaefer CLA 2015-02-17 16:16:41 EST
Speaking of community, it would be interesting to know how many committers use Jenkins outside of Eclipse and how many use Hudson.

From the few I've talked to (I haven't talked to any who use Hudson outside Eclipse), part of the problem is you get to know how Jenkins works and set-up in your day job and when you come to Eclipse and Hudson and it's just seems like a big step back. That could be just perception, but I've brought up a couple of examples here (Xvnc recorder and Docker) which make it seem more real.
Comment 12 Denis Roy CLA 2015-02-18 11:45:50 EST
Hudson/Jenkins are both kind of old-school.  If we're going to revisit CI at Eclipse.org why not investigate more modern components:

Gradle: https://gradle.org/

Buck: http://facebook.github.io/buck/
Comment 13 Mikaël Barbero CLA 2015-02-18 12:21:10 EST
Gradle and Buck are build systems (like Maven or Make), not CI tools.

Example of modern Hudson/Jenkins are:

- https://travis-ci.org/ (MIT)
- https://drone.io/ (Apache 2 Licence)
- http://www.thoughtworks.com/products/go-continuous-delivery (Apache 2 Licence)
- https://circleci.com/ (proprietary)
Comment 14 Marc Khouzam CLA 2015-02-18 12:44:35 EST
(In reply to Mikael Barbero from comment #13)
> Gradle and Buck are build systems (like Maven or Make), not CI tools.
> 
> Example of modern Hudson/Jenkins are:
> 
> - https://travis-ci.org/ (MIT)
> - https://drone.io/ (Apache 2 Licence)
> - http://www.thoughtworks.com/products/go-continuous-delivery (Apache 2
> Licence)
> - https://circleci.com/ (proprietary)

I've never used it but I also heard of:

http://buildbot.net/
Comment 15 Doug Schaefer CLA 2015-02-18 21:11:04 EST
(In reply to Denis Roy from comment #12)
> Hudson/Jenkins are both kind of old-school.  If we're going to revisit CI at
> Eclipse.org why not investigate more modern components:

I don't mind thinking outside the box. We just need to make sure its significantly better and worth the investment that everyone setting up and using the CI system needs to make.

travis-ci is probably the closest to what we need that I've seen. But I can't see where the source is. 

Jenkins seems to be the one I hear about most and I know committers use for their in house needs. That familiarity reduces the overall investment we have to make.
Comment 16 Marc Dumais CLA 2015-02-19 06:17:55 EST
(In reply to Doug Schaefer from comment #11)
> Speaking of community, it would be interesting to know how many committers
> use Jenkins outside of Eclipse and how many use Hudson.
> 
> From the few I've talked to (I haven't talked to any who use Hudson outside
> Eclipse), part of the problem is you get to know how Jenkins works and
> set-up in your day job and when you come to Eclipse and Hudson and it's just
> seems like a big step back. That could be just perception, but I've brought
> up a couple of examples here (Xvnc recorder and Docker) which make it seem
> more real.

+1  

eclipse.org is the only place I use Hudson. I vote to switch to Jenkins.

BTW, the trend seems to clearly go towards Jenkins:
http://www.google.ca/trends/explore#q=jenkins%20build%2C%20hudson%20build&cmpt=q&tz=
Comment 17 Denis Roy CLA 2015-02-19 08:12:20 EST
> Jenkins seems to be the one I hear about most and I know committers use for
> their in house needs. That familiarity reduces the overall investment we
> have to make.

+1 to that.
Comment 18 Marc-André Laperle CLA 2015-02-19 17:50:30 EST
I don't have a strong opinion about this but it's pretty obvious that Hudson has lost a lot of momentum and Jenkins has taken over. Documentation on Hudson is pretty sparse compared to Jenkins. Also, we had a few cases where things were working in Jenkins but not in Hudson. To be fair, after reporting the bugs and waiting for a bit, the bugs were fixed in Hudson too. All things considered, it seems inevitable that Eclipse will have to switch to Jenkins eventually so why not start planning the transition?
Comment 19 Doug Schaefer CLA 2015-02-20 12:10:35 EST
We have another important plug-in that Jenkins has that Hudson does not (apparently). That's the github pull request build plug-in which gives projects who chose github over gerrit the ability to do verification jobs when contributors submit pull requests.
Comment 20 Thanh Ha CLA 2015-02-20 12:26:21 EST
(In reply to Denis Roy from comment #5)
> (In reply to Doug Schaefer from comment #4)
> > I also wonder if Jenkins scales better. I have first hand knowledge of a
> > massive Jenkins instance that had hundreds if not thousands of jobs. All the
> > real work is supposed to be done in slaves anyway. Then we could get rid of
> > this HIPP craziness.
> 
> Where is this massive Jenkins anyway?  URL please, I want to poke around.

Probably the best public massive Jenkins instance to look at is OpenStack's Jenkins [1]. OpenStack also details some of their infra here [2].

With 1000s of jobs configured in a multi-master setup capable of spinning up dynamic slaves for each build. Using gearman we can allow jobs to be queued across multiple Jenkins masters that have unused slaves which helps with the Jenkins scaling issues.

Using Jenkins Job Builder [2] we can manage project jobs in Gerrit and easily deploy them to multiple master instances. Since jobs are managed in Gerrit we can make part of our infra become a project and thus take advantage of code review and CI to verify job configuration.

The OpenStack model is the model we're moving towards at LF.

[1] https://jenkins.openstack.org/view/All/
[2] http://ci.openstack.org/jenkins.html
[3] http://ci.openstack.org/jenkins-job-builder/
Comment 21 David Carver CLA 2015-02-20 13:23:34 EST
My biggest reason for wanting to use Jenkins is that the plugin community and third party plugins I need to really make integration with Github easier on my project are maintained by that community.   If Hudson had the equivalent plugins then I'd be fine with using it.  But right now I'm having to do work arounds, have filed bugs, and have had just general frustration.

Maybe it is because our project choose to stay on Github, as the general barrier to getting somebody to do submit code is less.  A new tool doesn't need to be learned.  We could use Gerrit, but that still leaves a huge workflow whole for those that would send Pull Requests in through github.

As long as we don't go with Travis CI, I'm fine.  I HATE that CI system.

I just want something that works well with the choosen hosting platform and tools I've choosen to use.   Jenkins has it currently, Hudson does not.
Comment 22 Denis Roy CLA 2015-02-20 13:41:44 EST
FWIW, there is no disagreement about the momentum behind Jenkins vs. Hudson.



(In reply to Doug Schaefer from comment #19)
> That's the github pull request build plug-in which gives
> projects who chose github over gerrit the ability to do verification jobs
> when contributors submit pull requests.

Wouldn't we want a Jenkins "builder in an isolated box" (ie, docker) as a prerequisite to that?  It seems to easy to plop a bunch of unverified malicious code inside a PR and have some process run it as part of a verification job.



(In reply to Marc-Andre Laperle from comment #18)
> why not start planning the transition?

You are witnessing exactly that.



(In reply to David Carver from comment #21)
> I just want something that works well with the choosen hosting platform and
> tools I've choosen to use.

There is no "I" in team :) We cannot accommodate every one of everyone's needs. That said, I concede that Jenkins likely accommodates everyone's needs better than Hudson.

Keep the feedback coming.  I can't promise action tomorrow morning on this front, but you know how we operate. We tend to follow the community's input.
Comment 23 David Carver CLA 2015-02-20 14:01:10 EST
> There is no "I" in team :) We cannot accommodate every one of everyone's
> needs. That said, I concede that Jenkins likely accommodates everyone's
> needs better than Hudson.

Yes there is take a look at the A  it is right there in the middle. :)
Comment 24 Doug Schaefer CLA 2015-02-20 14:42:49 EST
(In reply to Denis Roy from comment #22)
> (In reply to Doug Schaefer from comment #19)
> > That's the github pull request build plug-in which gives
> > projects who chose github over gerrit the ability to do verification jobs
> > when contributors submit pull requests.
> 
> Wouldn't we want a Jenkins "builder in an isolated box" (ie, docker) as a
> prerequisite to that?  It seems to easy to plop a bunch of unverified
> malicious code inside a PR and have some process run it as part of a
> verification job.

That's a great point. My biggest fear is a change request that has too much access to the system. That could happen with the Gerrit verifier we already have.

And yes, Docker would be great for that. And the Jenkins Docker plug-in does just that.

https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin
Comment 25 David Carver CLA 2015-02-23 11:20:41 EST
(In reply to Doug Schaefer from comment #24)
> (In reply to Denis Roy from comment #22)
> > (In reply to Doug Schaefer from comment #19)
> > > That's the github pull request build plug-in which gives
> > > projects who chose github over gerrit the ability to do verification jobs
> > > when contributors submit pull requests.
> > 
> > Wouldn't we want a Jenkins "builder in an isolated box" (ie, docker) as a
> > prerequisite to that?  It seems to easy to plop a bunch of unverified
> > malicious code inside a PR and have some process run it as part of a
> > verification job.
> 
> That's a great point. My biggest fear is a change request that has too much
> access to the system. That could happen with the Gerrit verifier we already
> have.
> 
> And yes, Docker would be great for that. And the Jenkins Docker plug-in does
> just that.
> 
> https://wiki.jenkins-ci.org/display/JENKINS/Docker+Plugin

The nice thing about the Jenkins Pull Request plugin is that you can set it up so that it only runs automatically for White Listed users.  All other users would need to have the commit verified by a committer giving the plugin the OK to go ahead an run it.

The nice thing about this is, that it allows for a circle of trust to occur, if somebody is submitting a lot of good pull requests and we start to trust them, getting them white listed could be a first step for them to eventually be committers.

Merging of the pull request back into develop would still need to be done by a committer regardless.
Comment 26 David Carver CLA 2015-02-23 11:24:07 EST
Here is the actual text:

When a new pull request is opened in the project and the author of the pull request isn't white-listed, builder will ask "Can one of the admins verify this patch?".

"ok to test" to accept this pull request for testing
"test this please" for a one time test run
"add to whitelist" to add the author to the whitelist
If the build fails for other various reasons you can rebuild.

"retest this please" to start a new build
Comment 27 Denis Roy CLA 2015-03-24 10:54:17 EDT
In the last few months we've upgraded most Hudson instances to the latest Hudson release.  If there are Jenkins plugins that are not working on the latest Hudson, can we file bugs against the Hudson project and mark them as blockers to this bug? The Hudson team can't fix it if they don't know about it.
Comment 28 Mikaël Barbero CLA 2015-03-24 11:12:24 EDT
(In reply to Denis Roy from comment #27)
> can we file bugs against the Hudson project and mark them as
> blockers to this bug? The Hudson team can't fix it if they don't know about
> it.

I created bugs a month ago for the lacking plugins/feature in Hudson. I added them as blocker to this bug.
Comment 29 Matthew Khouzam CLA 2015-10-26 10:29:49 EDT
Hi,

I added a blocker.
Comment 30 Matthias Sohn CLA 2015-12-26 18:23:24 EST
Upgrade of the JGit HIPP to Hudson 3.3.2 again broke the gerrit-trigger plugin, I filed bug 484914 to track this. We saw the same or similar breakage already two times with earlier Hudson upgrades (3.2.2 and a version from 2012).
Comment 31 Doug Schaefer CLA 2016-01-12 15:20:14 EST
(In reply to Denis Roy from comment #3)
> Since Hudson is now an Eclipse project, we'll eat our own dog food.

The dog food not tasting so good :(

I'll have to dig up whether Hudson is working on Github integration. Sucks that we're not getting proper verify jobs for projects hosted on github.
Comment 32 Matthias Sohn CLA 2016-01-12 18:24:16 EST
I think it's time to move to Jenkins, see
https://bugs.eclipse.org/bugs/show_bug.cgi?id=484914#c4
and Winston's reply in
https://bugs.eclipse.org/bugs/show_bug.cgi?id=484914#c6
Comment 33 Lorenzo Bettini CLA 2016-01-13 11:41:37 EST
(In reply to Matthias Sohn from comment #32)
> I think it's time to move to Jenkins, see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=484914#c4
> and Winston's reply in
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=484914#c6

+1 for moving to Jenkins

It's far too better and all its plugins are up-to-date... the same does not hold for Hudson I'm afraid... see all the blockers (my experience with the Jacoco plugin for Hudson is really bad, starting from the report it produces, while the one for Jenkins is really good).
Comment 34 Doug Schaefer CLA 2016-01-13 13:41:38 EST
(In reply to Matthias Sohn from comment #32)
> I think it's time to move to Jenkins, see
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=484914#c4
> and Winston's reply in
> https://bugs.eclipse.org/bugs/show_bug.cgi?id=484914#c6

Uh, yeah. I guess we have permission now.

Of course the next real question is how do we make this change in a practical way. IMHO, we fire up a Jenkins instance and let people have at it. Reset things back before HIPP and see if we still need that kind of separate in the Jenkins world. But I'm sure I'm just being naive and Denis and company could put together a real plan. It's certainly not going to be free.
Comment 35 Thanh Ha CLA 2016-01-13 14:43:10 EST
I found this slide deck over the holidays which I found helpful:

https://narkisr.github.io/jenkins-scaling/
Comment 36 Denis Roy CLA 2016-01-13 14:48:26 EST
> Reset things back before HIPP and see if we still need that kind of
> separate in the Jenkins world.

HIPP has given us a lot of good experience.  If for no other reason than a perfect separation of project privileges, JIPP would be the way to go IMHO and we could do a HIPP->JIPP transition on a project-by-project basis.

But one comment from Winston on the other bug -- Hudson does go through a more rigorous release process, and HIPP has been very stable.  If Jenkins' release model leads to a sloppier product that webmasters are unable to support, we're no further ahead.

As usual, we have to tread cautiously but with some planning and patience, we'll make it right.
Comment 37 David Carver CLA 2016-01-13 14:57:56 EST
(In reply to Denis Roy from comment #36)
> > Reset things back before HIPP and see if we still need that kind of
> > separate in the Jenkins world.
> 
> HIPP has given us a lot of good experience.  If for no other reason than a
> perfect separation of project privileges, JIPP would be the way to go IMHO
> and we could do a HIPP->JIPP transition on a project-by-project basis.
> 
> But one comment from Winston on the other bug -- Hudson does go through a
> more rigorous release process, and HIPP has been very stable.  If Jenkins'
> release model leads to a sloppier product that webmasters are unable to
> support, we're no further ahead.
> 
> As usual, we have to tread cautiously but with some planning and patience,
> we'll make it right.

For Jenkins stick with the Stable versions.  yes they release versions pretty frequently, but they have what they consider the Stable Release which is more static.
Comment 38 Matthias Sohn CLA 2016-01-13 17:47:01 EST
(In reply to Denis Roy from comment #36)
> > Reset things back before HIPP and see if we still need that kind of
> > separate in the Jenkins world.
> 
> HIPP has given us a lot of good experience.  If for no other reason than a
> perfect separation of project privileges, JIPP would be the way to go IMHO
> and we could do a HIPP->JIPP transition on a project-by-project basis.

+1 to move from HIPP to JIPP (like this name :)
Comment 39 Doug Schaefer CLA 2016-01-13 23:04:23 EST
The 'IPPs do have some advantages. I just hate running out of executors on CDT when I know PTP has some free ones. Or if CDT builds are taking a long time and I'm only using one executor, I'd like to know why. Is it the build broken or someone sharing my underlying machine chewing up resources.

Anyway, we can have that discussion over beers at EclipseCon. :)

I wouldn't worry too much about the stability of Jenkins. The community is huge. We have a number of instances at my employer all performing well with hundreds of jobs and dozens of slaves. We trust our business with it.
Comment 40 Vincenzo Caselli CLA 2016-01-14 03:58:04 EST
a +1 for Jenkins, for the more wider and active community.
Also I'm keen on renaming hudson.eclipse.org to jenkins.eclipse.org
Comment 41 Mickael Istria CLA 2016-01-14 04:15:13 EST
Despite being stable, Hudson misses some important integration with modern tools and services such as GitHub Pull Request builder, Gradle or most recent versions of SonarQube IIRC.
Some good things about Jenkins: it wouldn't require much effort in adoption for users, hopefully not too much from Foundation admins, and would make Eclipse CI infrastructure more open to the innovation happening very fast in the Jenkins ecosystem that is missing with Hudson.
Despite the hype is on Travis currently, Jenkins still has a huge amount of satisfied users, there are many very big Jenkins instances working correctly (such as the Apache one), and seems to have a more active and efficient community when it comes to development of extensions and support. Also, Travis uses different style of configuration and different paradigm, which would make it more difficult to adopt (it's a bit like changing your IDE or your build tool, whereas Hudson->Jenkins is like doing a version change of the same tool).

For all these reasons, I believe Jenkins is the best candidate for now.
Let's go on a JIPP !
Comment 42 Mikaël Barbero CLA 2016-01-14 11:20:44 EST
During 2016Q1 I will investigate if and how we can migrate from Hudson to Jenkins. Once I'll have done internal tests, I will probably ask for one or more volunteers for some test migrations. If you want to be one, please add a comment to this bug. I will keep you posted when we are ready for these test phase.
Comment 43 Mickael Istria CLA 2016-01-14 11:21:40 EST
I would volunteer to try a JIPP instance for SWTBot.
Comment 44 Doug Schaefer CLA 2016-01-14 11:31:27 EST
(In reply to Mickael Istria from comment #43)
> I would volunteer to try a JIPP instance for SWTBot.

Thanks Denis! We should try Andmore there and test out the github PR verifier there.
Comment 45 Thanh Ha CLA 2016-01-14 11:58:36 EST
I would highly recommend setting up JIPP to be capable of running Jenkins Job Builder [1] so that projects can self manage and update jobs via Gerrit code review. Typically our setup just needs 2 things installed on the Jenkins host:

1. Python
2. Python Virtualenv

Then a builder-merge and builder-verify job can be created that setup up JJB in a virtualenv as part of the job startup.

    virtualenv venv
    source venv/bin/activate
    pip install jenkins-job-builder
    jenkins-jobs update|test --r -o jobs_output jjb/

Projects could store their Jenkins configuration files in a jjb subproject repo or a directory under releng or something. The reason for depending on virtualenv is that it allows projects to pin specifc JJB versions or upgrade to other versions of JJB should the find support for plugins they require.

This setup significantly simplifies job maintenance and also backs up your job configurations to Gerrit in case anything goes wrong. I would be willing to volunteer time to help set up JJB if projects are interested.

Hope this helps.

[1] http://docs.openstack.org/infra/jenkins-job-builder/
Comment 46 Mikaël Barbero CLA 2016-01-14 12:03:38 EST
Thanks. JJB is definitely something I would like to investigate if the move to jenkins is successful. However, I think it's premature to think about it before any migration path has been setup ;)
Comment 47 Matthias Sohn CLA 2016-01-18 04:58:00 EST
I would volunteer to try a JIPP instance for JGit and EGit
Comment 48 James Sutton CLA 2016-01-19 05:46:52 EST
I'm a contributor on the Paho Project (Mainly the Java component). We're currently moving our source code to GitHub and so are really interested in taking advantage of the Jenkins GitHub Plugins to allow us to do Pull request builds.
If you're in need of volunteers, It would be great to get our Java builds moved over please! (With more to follow once our other Repositories have moved to GitHub too)
Comment 49 Lorenzo Bettini CLA 2016-01-19 06:50:10 EST
I would volunteer to try a JIPP instance for EMF Parsley
Comment 50 Marc-André Laperle CLA 2016-04-24 13:37:14 EDT
I would volunteer to try a JIPP instance for Trace Compass.

Ideally, we would keep the HIPP instance running for a while until we have tested the JIPP instance enough.

Was there any recent progress on the JIPP effort?
Comment 51 Mikaël Barbero CLA 2016-04-25 05:59:47 EDT
JIPP has been in standby for some times now even if ECF is using it successfully (https://hudson.eclipse.org/ecf/). Effort will start again in 2016Q3. Thanks for your patience.
Comment 52 Pascal Rapicault CLA 2016-08-26 07:41:18 EDT
Is there more details available on the timeline for this work?
EGerrit will be happy to be a guinea pig for for the initial roll out.
Comment 53 Doug Schaefer CLA 2016-08-26 10:10:10 EDT
With all the zombie processes Hudson is leaving around for us, we'd be happy to switch to Jenkins as well.
Comment 54 Frederic Gurr CLA 2016-08-29 04:56:19 EDT
We will run some migration tests in the coming weeks. Once this is done, we will provide JIPPs for guinea pigs.
Comment 55 Thanh Ha CLA 2016-08-29 08:47:13 EDT
(In reply to Frederic Gurr from comment #54)
> We will run some migration tests in the coming weeks. Once this is done, we
> will provide JIPPs for guinea pigs.

Is there interest in setting up Jenkins Job Builder [0] to manage the jobs?

I would love to help out with setting up JJB if there is.

[0] http://docs.openstack.org/infra/jenkins-job-builder/
Comment 56 Frederic Gurr CLA 2016-08-29 09:00:20 EDT
Thanks Thanh, once we have setup the basic JIPP functionality we will definitely have a look at the Jenkins Job Builder.
Comment 57 Pascal Rapicault CLA 2016-08-29 09:10:29 EDT
Btw, I assume that we are talking about Jenkins 2.x right?
Comment 58 Frederic Gurr CLA 2016-08-29 09:30:24 EDT
Since Jenkins 1.x has reached end-of-life (https://jenkins.io/blog/2016/07/07/jenkins-2/) we are indeed talking about Jenkins 2.x.
Comment 59 Denis Roy CLA 2016-08-29 10:45:42 EDT
So, one question that will come up is the hudson.eclipse.org hostname.

I don't feel like creating a new jenkins.eclipse.org hostname and running two in parallel.  Can we create something like ci.eclipse.org or buidfarm.eclipse.org or captainspock.eclipse.org and run both hipp and jipp under the same roof?
Comment 60 Doug Schaefer CLA 2016-08-29 11:14:22 EDT
(In reply to Denis Roy from comment #59)
> So, one question that will come up is the hudson.eclipse.org hostname.
> 
> I don't feel like creating a new jenkins.eclipse.org hostname and running
> two in parallel.  Can we create something like ci.eclipse.org or
> buidfarm.eclipse.org or captainspock.eclipse.org and run both hipp and jipp
> under the same roof?

ci.eclipse.org sounds good. Spock wasn't a captain for long, you'd be better using ambassadorspock.eclipse.org. :D
Comment 61 Frederic Gurr CLA 2016-08-29 11:17:30 EDT
+1 for ci.eclipse.org
Comment 62 Thanh Ha CLA 2016-08-29 11:24:19 EDT
+1 for ci.eclipse.org
Comment 63 Eclipse Webmaster CLA 2016-08-29 11:27:41 EDT
Ok I've added a record for ci.eclipse.org that points to the current hudson.eclipse.org site. The changes should be live by tomorrow morning.

Once the dust settles with this bug we can look at retiring hudson.eclipse.org as a name.

-M.
Comment 64 Gunnar Wagenknecht CLA 2016-09-26 07:33:42 EDT
I'm also signing up for the JIPP beta template. :)
Comment 65 Marc-André Laperle CLA 2016-11-07 11:32:59 EST
Hello, sorry for the noise. How is this effort going? Anything we can do to help?
Comment 66 Pascal Rapicault CLA 2016-11-28 20:36:08 EST
ping
Comment 67 Frederic Gurr CLA 2016-11-29 05:49:43 EST
I'm currently working on scripts that translate Hudson config files to Jenkins config files.
Stay tuned... and thanks for your patience.
Comment 68 Frederic Gurr CLA 2017-02-10 12:49:54 EST
Just to give you a long overdue update on this topic, here is what's happening at the moment.

- As of the beginning of 2017 we will not provision any new Hudson instances, unless there's a special request and requirements that can not be met with Jenkins. So if a project requests a new CI instance, they will get Jenkins by default (the latest LTS version at that time).

- We are in the finishing stages for a little tool that helps to convert the Hudson config file structure to the Jenkins config file structure. So when Hudson is replaced with Jenkins, most of the configuration still works. We will test this with a few guinea pig projects in the coming weeks.

- Once that converter tool is battle tested, we will convert a certain number of smaller HIPPs (or HIPPs with simple jobs) first (e.g. 5 per week) and ramp up when possible. We will do this to reduce the risk and possible impacts of any roadblocks that we might face.

- In the last stage we will convert HIPPs that are part of the SimRel and essential like the Shared or Platform HIPP.

We can't give an exact time line for the whole HIPP to JIPP migration, but be assured that this is one of the main objectives in Eclipse CBI for 2017.

I've also sent this comment as a post to the cbi-dev mailing list.
Comment 69 Doug Schaefer CLA 2017-02-10 13:09:22 EST
Awesome! Thanks Frederic. CDT is certainly interested in moving as soon as you think it's the right time.
Comment 70 Matthias Sohn CLA 2017-02-10 15:14:52 EST
Great, JGit and EGit are interested in moving.
Can't wait to use Jenkins pipelines :-)
Comment 71 Lorenzo Bettini CLA 2017-02-11 04:30:29 EST
EMF Parsley is looking forward to as well!
Comment 72 Frederic Gurr CLA 2017-09-05 08:13:56 EDT
Status update:
--------------

Number of HIPPs: 161
Number of JIPPs:  31

Successfully migrated HIPPs:
* app4mc
* CBI
* Eclemma
* Eclipsescada
* EGit
* JGit
* OMR
* package-drone
* SimRel

The tool that is used for the migration is now on GitHub:
https://github.com/eclipse/hipp2jipp

Even though most of the common configurations/plugins are now covered and converted reliably, there are some issues (see known issues on the hipp2jipp Github page). To fix those and iron out issues in the tool, all jobs of the migrated HIPPs were checked/compared manually.

Obviously this can not be done for all of the 161 remaining HIPPs with the limited resources of the Eclipse Foundation's IT team. Therefore we will have to rely on the support of the community. The current plan is to do a best effort migration and give projects a chance to still see the old configurations to compare it to the converted configurations (the old configurations are backed up anyways).

If a project finds that a configuration option or plugin is not covered by the hipp2jipp tool (and is feasible to do with the tool), a bug can be reported. Pull requests are appreciated even more. :)

To reduce the migration effort, we will look for dormant HIPPs and delete them after giving projects a chance to speak up if they still need it. We also encourage projects to clean out old jobs and convert cascaded jobs to normal ones (cascaded jobs are not supported in Jenkins). HIPPs with less jobs and less complex configurations are likely to be migrated first.

If you have any ideas how the migration can be simplified, improved, made faster, please let us know.
Comment 73 Denis Roy CLA 2017-09-06 14:14:54 EDT
Nice work!

> To reduce the migration effort, we will look for dormant HIPPs and delete
> them after giving projects a chance to speak up if they still need it.

+1 on that.


> We
> also encourage projects to clean out old jobs and convert cascaded jobs to
> normal ones (cascaded jobs are not supported in Jenkins). HIPPs with less
> jobs and less complex configurations are likely to be migrated first.

+1 also
Comment 74 Gunnar Wagenknecht CLA 2017-09-08 03:34:04 EDT
FWIW, the Gyrex HIPP is no longer necessary and can be removed.
Comment 75 Frederic Gurr CLA 2017-09-21 10:08:08 EDT
(In reply to Gunnar Wagenknecht from comment #74)
> FWIW, the Gyrex HIPP is no longer necessary and can be removed.

Thanks for the feedback.

The first batch of HIPPs has been retired (see bug 522275). More to come soon.
Comment 76 Frederic Gurr CLA 2017-09-21 10:11:19 EDT
I think this issue can be closed. We know that the way forward is migrating all existing HIPPs to JIPPs. Projects should create individual tickets to ask for the actual migration.

Please re-open if you disagree or if anything is still missing.