Bug 553288 - Re-sign bundles that will expire on 2020-12-29
Summary: Re-sign bundles that will expire on 2020-12-29
Status: RESOLVED FIXED
Alias: None
Product: Orbit
Classification: Tools
Component: releng (show other bugs)
Version: unspecified   Edit
Hardware: PC Linux
: P3 normal (vote)
Target Milestone: 2020-12 M3   Edit
Assignee: Roland Grunberg CLA
QA Contact: Project Inbox CLA
URL:
Whiteboard:
Keywords:
Depends on:
Blocks:
 
Reported: 2019-11-20 16:58 EST by Roland Grunberg CLA
Modified: 2020-11-18 14:30 EST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Roland Grunberg CLA 2019-11-20 16:58:07 EST
As discussed in https://bugs.eclipse.org/bugs/show_bug.cgi?id=552251#c23 .

"The signer certificate expired on 2018-03-08. However, the JAR will be valid until the timestamp expires on 2020-12-29."

com.google.inject_3.0.0.v201605172100.jar
com.google.inject.multibindings_3.0.0.v201605172100.jar
com.google.inject.multibindings.source_3.0.0.v201605172100.jar
com.google.inject.source_3.0.0.v201605172100.jar
de.tuberlin.eecs.agg_2.1.0.v201512080800.jar
javaewah_0.7.9.v201605172130.jar
javaewah.source_0.7.9.v201605172130.jar
javax.annotation_1.2.0.v201602091430.jar
javax.annotation.source_1.2.0.v201602091430.jar
javax.servlet.jsp_2.2.0.v201112011158.jar
javax.servlet.jsp.source_2.2.0.v201112011158.jar
org.apache.ant_1.9.6.v201510161327.jar
org.apache.ant.source_1.9.6.v201510161327.jar
org.apache.commons.collections_3.2.2.v201511171945.jar
org.apache.commons.collections.source_3.2.2.v201511171945.jar
org.apache.derby_10.11.1.1_v201605202053.jar
org.apache.derby_10.8.2.2_v201605172130.jar
org.apache.derby.source_10.11.1.1_v201605202053.jar
org.apache.derby.source_10.8.2.2_v201605172130.jar
org.apache.httpcomponents.httpclient_4.3.6.v201511171540.jar
org.apache.httpcomponents.httpclient.source_4.3.6.v201511171540.jar
org.apache.olingo_2.0.3.v201605172220.jar
org.apache.tika.parsers_1.3.0.v201605180015.jar
org.apache.tika.parsers.source_1.3.0.v201605180015.jar
org.apache.xmlbeans_2.3.0.v201605172150.jar
org.apache.xmlbeans.source_2.3.0.v201605172150.jar
org.custommonkey.xmlunit_1.3.0.v201605172130.jar
org.custommonkey.xmlunit.source_1.3.0.v201605172130.jar
org.mockito_1.9.5.v201605172210.jar
org.mockito.source_1.9.5.v201605172210.jar
org.restlet_2.0.5.v201605172130.jar
org.restlet.ext.servlet_2.0.5.v201605172130.jar
org.restlet.ext.servlet.source_2.0.5.v201605172130.jar
org.restlet.source_2.0.5.v201605172130.jar
org.swtchart_0.10.0.v201605200358.jar
org.swtchart.source_0.10.0.v201605200358.jar
org.yaml.snakeyaml_1.14.0.v201604211500.jar

(including their corresponding packed jars)
Comment 1 Roland Grunberg CLA 2020-11-17 12:07:56 EST
As of http://download.eclipse.org/tools/orbit/downloads/drops/I20201117163612/repository , the bundles above, and their packed artifacts have been re-signed.

I tested by installing each one locally to confirm no issues, but if there is something, we can re-open.

Note that the older version of snakeyaml has been removed.
Comment 2 Jonah Graham CLA 2020-11-17 16:43:04 EST
Can the bundles get new qualifiers? At the moment the simple resigning means that multiple bundles are out there with identical qualifiers, but different contents.

This makes it extra hard to make sure that what projects are contributing to simrel are indeed up to date. Ref Bug 568001 and Bug 499207 Comment 14
Comment 3 Roland Grunberg CLA 2020-11-17 19:10:25 EST
(In reply to Jonah Graham from comment #2)
> Can the bundles get new qualifiers? At the moment the simple resigning means
> that multiple bundles are out there with identical qualifiers, but different
> contents.
> 
> This makes it extra hard to make sure that what projects are contributing to
> simrel are indeed up to date. Ref Bug 568001 and Bug 499207 Comment 14

I'm really not sure of a way to do this correctly. It's not far off from just deleting them and just rebuilding in orbit-recipes.

The qualifier of a bundle is used in the artifact repository (artifacts.jar), and more heavily in the metadata repository (content.jar) for a bundle's provided capabilities. Also we'd have to change the Bundle-Version in the manifest.

If we aimed to modify just a few properties, I'm not sure what we could safely change to make the artifact different while not violating any assumptions p2/Tycho might make about artifacts and their metadata.
Comment 4 Jonah Graham CLA 2020-11-17 19:28:31 EST
Do we still have the ability to rebuild the CVS repo? I am guessing this is not worth it. 

I have seen the CI job and I now understand the resigning is a rather manual process. I'm not sure what you do after the signing though? 

Anyway, we'll need another way of making sure the simrel repo has only the current artifacts from orbit. But as Orbit doesn't contribute directly to simrel we rely on all the projects to update.
Comment 5 Alexander Kurtakov CLA 2020-11-18 02:17:16 EST
(In reply to Roland Grunberg from comment #3)
> (In reply to Jonah Graham from comment #2)
> > Can the bundles get new qualifiers? At the moment the simple resigning means
> > that multiple bundles are out there with identical qualifiers, but different
> > contents.
> > 
> > This makes it extra hard to make sure that what projects are contributing to
> > simrel are indeed up to date. Ref Bug 568001 and Bug 499207 Comment 14
> 
> I'm really not sure of a way to do this correctly. It's not far off from
> just deleting them and just rebuilding in orbit-recipes.

If ^^^ is easy to achieve it might be best to go that path so we have simpler build system, no composite repo and so on.

> 
> The qualifier of a bundle is used in the artifact repository
> (artifacts.jar), and more heavily in the metadata repository (content.jar)
> for a bundle's provided capabilities. Also we'd have to change the
> Bundle-Version in the manifest.
> 
> If we aimed to modify just a few properties, I'm not sure what we could
> safely change to make the artifact different while not violating any
> assumptions p2/Tycho might make about artifacts and their metadata.
Comment 6 Alexander Kurtakov CLA 2020-11-18 02:26:49 EST
Also looking at the list at least ant, javaewah, javax.annotation, derby, httpclient, mockito have newer versions in recipes. Swtchart is eclipse project now.
I haver gone through the list and only the following don't have newer version in Orbit:
* google.inject
* google.inject.multibinding
* javax.servlet.jsp
* olingo
* tika.parsers
* snakeyaml
* restlet.ext.servlet
Comment 7 Jonah Graham CLA 2020-11-18 10:57:18 EST
(In reply to Alexander Kurtakov from comment #6)
> * snakeyaml

snakeyaml does, just didn't in M2.
Comment 8 Jonah Graham CLA 2020-11-18 11:59:42 EST
I don't think the resigning worked. I can't install (for example javax.xml) bundles from https://download.eclipse.org/tools/orbit/downloads/drops/R20201117151609/repository/

but I can with:
https://download.eclipse.org/tools/orbit/downloads/drops/R20191115185527/repository

Every single mirror is tried and this is the error I get:


An error occurred while collecting items to be installed
session context was:(profile=SDKProfile, phase=org.eclipse.equinox.internal.p2.engine.phases.Collect, operand=, action=).
Failed to transfer artifact canonical: osgi.bundle,javax.xml,1.3.4.v201005080400.
Retry another mirror
Problems downloading artifact: osgi.bundle,javax.xml,1.3.4.v201005080400.
MD5 hash is not as expected. Expected: 6a0a58cd393760e4631865f64d03ae43 and found 7f7c93e568d12d93268300aa85060dd1.
Retry another mirror
Problems downloading artifact: osgi.bundle,javax.xml,1.3.4.v201005080400.
MD5 hash is not as expected. Expected: 6a0a58cd393760e4631865f64d03ae43 and found 7f7c93e568d12d93268300aa85060dd1.
--- snip ---
above repeated dozens of times for every single mirror.
Comment 9 Roland Grunberg CLA 2020-11-18 12:31:29 EST
javax.xml_1.3.4.v201005080400 hasn't been touched since R20180330011457 (Oxygen.3a), which is when its md5 was 7f7c93e568d12d93268300aa85060dd1. Since then it has always been 6a0a58cd393760e4631865f64d03ae43 .

What happens if you use http://download.eclipse.org/tools/orbit/downloads/drops/I20201117163612/repository ?

Are you combining the R2020 repo with a platform one to get org.eclipse.osgi, which may have an older javax.xml within ?
Comment 10 Jonah Graham CLA 2020-11-18 12:53:44 EST
(In reply to Roland Grunberg from comment #9)
I don't know what is going wrong - but it doesn't seem to be related to this change. Sorry for the noise.

The "bad" jar is being downloaded from mirrors on a different path. If I figure out a reproducer I will provide it as a new bug.

I had to take a screenshot to see it, but the URL it is using on the mirror is this release: R20160520211859
Comment 11 Jonah Graham CLA 2020-11-18 12:58:33 EST
Actually the problem is here:

The p2.mirrorsURL (and stats one) points to the 2016 repo in https://download.eclipse.org/tools/orbit/downloads/drops/R20201117151609/repository/artifacts.jar

<property name="p2.mirrorsURL" value="http://www.eclipse.org/downloads/download.php?format=xml&file=/tools/orbit/downloads/drops/R20160520211859/repository/"/>
<property name="p2.statsURI" value="http://download.eclipse.org/stats/tools/orbit/downloads/drops/R20160520211859"/>
Comment 12 Roland Grunberg CLA 2020-11-18 13:12:15 EST
(In reply to Jonah Graham from comment #11)
> Actually the problem is here:
> 
> The p2.mirrorsURL (and stats one) points to the 2016 repo in
> https://download.eclipse.org/tools/orbit/downloads/drops/R20201117151609/
> repository/artifacts.jar
> 
> <property name="p2.mirrorsURL"
> value="http://www.eclipse.org/downloads/download.php?format=xml&file=/tools/
> orbit/downloads/drops/R20160520211859/repository/"/>
> <property name="p2.statsURI"
> value="http://download.eclipse.org/stats/tools/orbit/downloads/drops/
> R20160520211859"/>

Yup, looks like I updated this previously. It should be fixed now.
Comment 13 Jonah Graham CLA 2020-11-18 14:27:49 EST
(In reply to Roland Grunberg from comment #12)
> Yup, looks like I updated this previously. It should be fixed now.

Thanks.
Comment 14 Jonah Graham CLA 2020-11-18 14:30:28 EST
(In reply to Alexander Kurtakov from comment #6)
> Also looking at the list at least ant, javaewah, javax.annotation, derby,
> httpclient, mockito have newer versions in recipes. Swtchart is eclipse
> project now.
> I haver gone through the list and only the following don't have newer
> version in Orbit:
> * google.inject
> * google.inject.multibinding
> * javax.servlet.jsp
> * olingo
> * tika.parsers
> * snakeyaml
> * restlet.ext.servlet

Raised Bug 568936.