Community
Participate
Working Groups
Then you have two Git servers (e.g. Gerrit) sharing the repository over NFS, the JGit GC of the repository on one node can cause the failing of reading Git Notes on the second node. Gerrit Code Review uses the Notes for storing information about their accounts external ids: failing to read them means that existing users may be unable to login. Restarting the JVM is the only way to solve the problem. --- Insights on the root cause of the issue --- The first node does a JGit GC which packs loose objects into packfiles and keep the loose objects around for the pruneexpire time. Eventually, loose objects are getting removed and they may leave behind some stale file handles on the other nodes. Stale file handles are correctly managed already when reading SHA1s from packfiles but NOT when reading from loose objects. The ObjectDirectory.openObject() has the following logic: @Override ObjectLoader openObject(WindowCursor curs, AnyObjectId objectId) throws IOException { if (unpackedObjectCache.isUnpacked(objectId)) { ObjectLoader ldr = openLooseObject(curs, objectId); if (ldr != null) { return ldr; } } ObjectLoader ldr = openPackedFromSelfOrAlternate(curs, objectId, null); if (ldr != null) { return ldr; } return openLooseFromSelfOrAlternate(curs, objectId, null); } However, if the openLooseObject() fails because of a "stale file handle" problem, the method exists instead of trying to load the object from packs or alternate locations.
@luca, would you mind providing a stack trace that would help to replicate this issue?
> However, if the openLooseObject() fails because of a "stale file handle" problem, the method exists instead of trying to load the object from packs or alternate locations. openLooseObject() just delegates to open(), which does the following: ObjectLoader open(WindowCursor curs, AnyObjectId id) throws IOException { File path = fileFor(id); try (FileInputStream in = new FileInputStream(path)) { unpackedObjectCache.add(id); return UnpackedObject.open(in, path, id, curs); } catch (FileNotFoundException noFile) { if (path.exists()) { throw noFile; } unpackedObjectCache.remove(id); return null; } } Looks like the only way an exception can be thrown by this method, is when both these conditions arise: - a FileNotFoundException is caught when opening the path to a loose object. - The path to the loose object exists. In any other case, the method will just return null, which in turn will cause the object to be searched in the packfiles and alternate[1]. I am interested to see the stack trace because I am struggling to understand how a stale file handle exception can be thrown/cause abrupt termination of the search. [1]https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/refs/heads/stable-5.11/org.eclipse.jgit/src/org/eclipse/jgit/internal/storage/file/ObjectDirectory.java#345
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/181216
> ObjectLoader open(WindowCursor curs, AnyObjectId id) throws IOException { > File path = fileFor(id); > try (FileInputStream in = new FileInputStream(path)) { > unpackedObjectCache.add(id); > return UnpackedObject.open(in, path, id, curs); > } catch (FileNotFoundException noFile) { > if (path.exists()) { > throw noFile; > } > unpackedObjectCache.remove(id); > return null; > } >} >Looks like the only way an exception can be thrown by this method, is when both >these conditions arise: >- a FileNotFoundException is caught when opening the path to a loose object. >- The path to the loose object exists. You forgot another condition: the file handle exists but it is stale, which is exactly the case of this bug. > In any other case, the method will just return null, which in turn will cause > the object to be searched in the packfiles and alternate[1]. I believe there is a problem in JGit in general: the IOException for "Stale file handle" should always be treated as a FileNotFoundException, however, that isn't consistently implemented everywhere :-( See below the stack-trace: [2021-04-30T17:56:54.955-07:00] [HTTP POST /plugins/high-availability/index/change/batch/myproject~1234] WARN com.google.gerrit.server.account.AccountCacheImpl : Cannot load AccountState for username johndoe@mycompany [CONTEXT PLUGIN=""gerrit"" ] java.io.IOException: Stale file handle at java.base/java.io.FileInputStream.available0(Native Method) at java.base/java.io.FileInputStream.available(FileInputStream.java:330) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:359) at java.base/java.util.zip.InflaterInputStream.fill(InflaterInputStream.java:243) at java.base/java.util.zip.InflaterInputStream.read(InflaterInputStream.java:159) at org.eclipse.jgit.internal.storage.file.UnpackedObject.readSome(UnpackedObject.java:318) at org.eclipse.jgit.internal.storage.file.UnpackedObject.open(UnpackedObject.java:78) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openLooseObject(ObjectDirectory.java:479) at org.eclipse.jgit.internal.storage.file.ObjectDirectory.openObject(ObjectDirectory.java:399) at org.eclipse.jgit.internal.storage.file.WindowCursor.open(WindowCursor.java:132) at org.eclipse.jgit.treewalk.CanonicalTreeParser.reset(CanonicalTreeParser.java:191) at org.eclipse.jgit.treewalk.CanonicalTreeParser.<init>(CanonicalTreeParser.java:86) at org.eclipse.jgit.notes.NoteParser.<init>(NoteParser.java:71) at org.eclipse.jgit.notes.NoteParser.parse(NoteParser.java:58) at org.eclipse.jgit.notes.NoteMap.load(NoteMap.java:360) at org.eclipse.jgit.notes.NoteMap.readTree(NoteMap.java:138) at org.eclipse.jgit.notes.NoteMap.read(NoteMap.java:112) at org.eclipse.jgit.notes.NoteMap.read(NoteMap.java:87) at com.google.gerrit.server.account.externalids.ExternalIdNotes.onLoad(ExternalIdNotes.java:680) at com.google.gerrit.server.git.meta.VersionedMetaData.load(VersionedMetaData.java:188) at com.google.gerrit.server.git.meta.VersionedMetaData.load(VersionedMetaData.java:160) at com.google.gerrit.server.git.meta.VersionedMetaData.load(VersionedMetaData.java:138) at com.google.gerrit.server.account.externalids.ExternalIdNotes.load(ExternalIdNotes.java:341) at com.google.gerrit.server.account.externalids.ExternalIdNotes.loadReadOnly(ExternalIdNotes.java:215) at com.google.gerrit.server.account.externalids.ExternalIdReader.get(ExternalIdReader.java:138) at com.google.gerrit.server.account.externalids.ExternalIds.get(ExternalIds.java:57) at com.google.gerrit.server.account.AccountCacheImpl.getByUsername(AccountCacheImpl.java:142) at com.google.gerrit.server.account.AccountResolver$ByUsername.search(AccountResolver.java:322) at com.google.gerrit.server.account.AccountResolver$ByUsername.search(AccountResolver.java:314) at com.google.gerrit.server.account.AccountResolver$Searcher.trySearch(AccountResolver.java:230) at com.google.gerrit.server.account.AccountResolver.searchImpl(AccountResolver.java:622) at com.google.gerrit.server.account.AccountResolver.resolve(AccountResolver.java:526) at com.googlesource.gerrit.owners.common.AccountsImpl.findUserOrEmail(AccountsImpl.java:104) at com.googlesource.gerrit.owners.common.AccountsImpl.find(AccountsImpl.java:76) at com.googlesource.gerrit.owners.common.ConfigurationParser.lambda$toMatcher$2(ConfigurationParser.java:92) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:1033) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at com.googlesource.gerrit.owners.common.ConfigurationParser.toMatcher(ConfigurationParser.java:93) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.ArrayList$Itr.forEachRemaining(ArrayList.java:1033) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ForEachOps$ForEachOp.evaluateSequential(ForEachOps.java:150) at java.base/java.util.stream.ForEachOps$ForEachOp$OfRef.evaluateSequential(ForEachOps.java:173) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.forEach(ReferencePipeline.java:497) at com.googlesource.gerrit.owners.common.ConfigurationParser.lambda$addMatchers$0(ConfigurationParser.java:72) at java.base/java.util.Optional.ifPresent(Optional.java:183) at com.googlesource.gerrit.owners.common.ConfigurationParser.addMatchers(ConfigurationParser.java:72) at com.googlesource.gerrit.owners.common.ConfigurationParser.getOwnersConfig(ConfigurationParser.java:49) at com.googlesource.gerrit.owners.common.PathOwners.lambda$getOwnersConfig$3(PathOwners.java:250) at java.base/java.util.Optional.flatMap(Optional.java:294) at com.googlesource.gerrit.owners.common.PathOwners.getOwnersConfig(PathOwners.java:250) at com.googlesource.gerrit.owners.common.PathOwners.fetchOwners(PathOwners.java:109) at com.googlesource.gerrit.owners.common.PathOwners.<init>(PathOwners.java:70) at com.googlesource.gerrit.owners.OwnersStoredValues$1.createValue(OwnersStoredValues.java:47) at com.googlesource.gerrit.owners.OwnersStoredValues$1.createValue(OwnersStoredValues.java:41) at com.google.gerrit.server.rules.StoredValue.get(StoredValue.java:62) at gerrit_owners.PRED_matcher_path_1.exec(PRED_matcher_path_1.java:46) at com.googlecode.prolog_cafe.lang.PrologControl.executePredicate(PrologControl.java:191) at com.googlecode.prolog_cafe.lang.BufferingPrologControl.run(BufferingPrologControl.java:147) at com.googlecode.prolog_cafe.lang.BufferingPrologControl.all(BufferingPrologControl.java:121) at com.google.gerrit.server.rules.PrologRuleEvaluator.evaluateImpl(PrologRuleEvaluator.java:408) at com.google.gerrit.server.rules.PrologRuleEvaluator.evaluate(PrologRuleEvaluator.java:164) at com.google.gerrit.server.rules.PrologRule.evaluate(PrologRule.java:52) at com.google.gerrit.server.rules.PrologRule.evaluate(PrologRule.java:48) at com.google.gerrit.server.project.SubmitRuleEvaluator.lambda$evaluate$0(SubmitRuleEvaluator.java:133) at com.google.gerrit.server.plugincontext.PluginContext.call(PluginContext.java:332) at com.google.gerrit.server.plugincontext.PluginSetEntryContext.call(PluginSetEntryContext.java:150) at com.google.gerrit.server.project.SubmitRuleEvaluator.lambda$evaluate$1(SubmitRuleEvaluator.java:133) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at com.google.gerrit.server.project.SubmitRuleEvaluator.evaluate(SubmitRuleEvaluator.java:136) at com.google.gerrit.server.query.change.ChangeData.submitRecords(ChangeData.java:841) at com.google.gerrit.server.index.change.ChangeField.storedSubmitRecords(ChangeField.java:938) at com.google.gerrit.server.index.change.ChangeField.lambda$static$53(ChangeField.java:912) at com.google.gerrit.index.FieldDef.get(FieldDef.java:142) at com.google.gerrit.index.Schema.fieldValues(Schema.java:187) at com.google.gerrit.index.Schema.lambda$buildFields$0(Schema.java:219) at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195) at com.google.common.collect.CollectSpliterators$1WithCharacteristics.lambda$forEachRemaining$1(CollectSpliterators.java:67) at java.base/java.util.stream.Streams$RangeIntSpliterator.forEachRemaining(Streams.java:104) at com.google.common.collect.CollectSpliterators$1WithCharacteristics.forEachRemaining(CollectSpliterators.java:67) at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484) at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474) at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913) at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578) at com.google.gerrit.index.Schema.buildFields(Schema.java:221) at com.google.gerrit.lucene.AbstractLuceneIndex.toDocument(AbstractLuceneIndex.java:318) at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:271) at com.google.gerrit.lucene.LuceneChangeIndex.replace(LuceneChangeIndex.java:109) at com.google.gerrit.server.index.change.ChangeIndexer.indexImpl(ChangeIndexer.java:211) at com.google.gerrit.server.index.change.ChangeIndexer.doIndex(ChangeIndexer.java:177) at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:173) at com.google.gerrit.server.index.change.ChangeIndexer.index(ChangeIndexer.java:245) at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexChangeHandler.reindex(ForwardedIndexChangeHandler.java:118) at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexChangeHandler.doIndex(ForwardedIndexChangeHandler.java:80) at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexChangeHandler.doIndex(ForwardedIndexChangeHandler.java:70) at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexChangeHandler.doIndex(ForwardedIndexChangeHandler.java:42) at com.ericsson.gerrit.plugins.highavailability.forwarder.ForwardedIndexingHandler.index(ForwardedIndexingHandler.java:63) at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.AbstractIndexRestApiServlet.process(AbstractIndexRestApiServlet.java:91) at com.ericsson.gerrit.plugins.highavailability.forwarder.rest.AbstractIndexRestApiServlet.doPost(AbstractIndexRestApiServlet.java:73) at javax.servlet.http.HttpServlet.service(HttpServlet.java:661) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) at com.google.gerrit.httpd.plugins.HttpPluginServlet.service(HttpPluginServlet.java:230) at javax.servlet.http.HttpServlet.service(HttpServlet.java:742) at com.google.inject.servlet.ServletDefinition.doServiceImpl(ServletDefinition.java:290) at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:280) at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:184) at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:89) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:85) at com.google.gerrit.httpd.raw.StaticModule$PolyGerritFilter.doFilter(StaticModule.java:387) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.GetUserFilter.doFilter(GetUserFilter.java:92) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.RequireSslFilter.doFilter(RequireSslFilter.java:72) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.RunAsFilter.doFilter(RunAsFilter.java:120) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.SetThreadNameFilter.doFilter(SetThreadNameFilter.java:62) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:139) at com.googlesource.gerrit.plugins.readonly.ReadOnly.doFilter(ReadOnly.java:79) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239) at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215) at com.googlesource.gerrit.plugins.javamelody.GerritMonitoringFilter.doFilter(GerritMonitoringFilter.java:66) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135) at com.google.gerrit.httpd.AllowRenderInFrameFilter.doFilter(AllowRenderInFrameFilter.java:56) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy$1.doFilter(AllRequestFilter.java:135) at com.google.gerrit.httpd.AllRequestFilter$FilterProxy.doFilter(AllRequestFilter.java:141) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.RequestCleanupFilter.doFilter(RequestCleanupFilter.java:60) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.RequestMetricsFilter.doFilter(RequestMetricsFilter.java:57) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.gerrit.httpd.RequestContextFilter.doFilter(RequestContextFilter.java:64) at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:82) at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:121) at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:133) at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:193) at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1601) at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:548) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1624) at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:233) at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1435) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:188) at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:501) at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1594) at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:186) at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1350) at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141) at org.eclipse.jetty.server.handler.RequestLogHandler.handle(RequestLogHandler.java:54) at org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:179) at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:127) at org.eclipse.jetty.server.Server.handle(Server.java:516) at org.eclipse.jetty.server.HttpChannel.lambda$handle$1(HttpChannel.java:388) at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:633) at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:380) at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:273) at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:311) at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:105) at org.eclipse.jetty.io.ChannelEndPoint$1.run(ChannelEndPoint.java:104) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.runTask(EatWhatYouKill.java:336) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.doProduce(EatWhatYouKill.java:313) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.tryProduce(EatWhatYouKill.java:171) at org.eclipse.jetty.util.thread.strategy.EatWhatYouKill.run(EatWhatYouKill.java:129) at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:375) at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:773) at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:905) at java.base/java.lang.Thread.run(Thread.java:834) I am interested to see the stack trace because I am struggling to understand how a stale file handle exception can be thrown/cause abrupt termination of the search.
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/181295
Gerrit change https://git.eclipse.org/r/c/jgit/jgit/+/181295 was merged to [stable-5.9]. Commit: http://git.eclipse.org/c/jgit/jgit.git/commit/?id=24d6d605388c82201092cf1699b51095299380a2
