Community
Participate
Working Groups
When new pack file creation fails during the packed refs file updating lock file is not removed. When this happens all following write operations are failing because of the LOCK_FAILURE. Currently only way to resolve the issue is to manually remove the lock file from the repository. JGit should recognise stale lock file for packed refs file and remove it.
(In reply to Marcin Czech from comment #0) > When new pack file creation fails during the packed refs file updating lock > file is not removed. - How is creation of a pack file related to packed-refs ? Which command or API was run to update/create packed-refs ? - Which lock file was not removed ? - How did this fail ? - with an exception ? If yes, please provide a stacktrace - process crashed or was terminated forcefully ? - did you try using the latest release ? > When this happens all following write operations are > failing because of the LOCK_FAILURE. Currently only way to resolve the issue > is to manually remove the lock file from the repository. > > JGit should recognise stale lock file for packed refs file and remove it.
Here is a fix https://git.eclipse.org/r/c/jgit/jgit/+/204213 assuming you meant that if the JVM shuts down gracefully during repacking of packed-refs the file lock isn't removed.
> - How is creation of a pack file related to packed-refs ? Which command or > API was run to update/create packed-refs ? > - Which lock file was not removed ? > - How did this fail ? > - with an exception ? If yes, please provide a stacktrace > - process crashed or was terminated forcefully ? > - did you try using the latest release ? > - How is creation of a pack file related to packed-refs ? If I understand correctly during the write operation packed-refs file is locked and updated(I believe new packed-refs file is created and then swapped) > - Which lock file was not removed ? packed-refs lock file > - How did this fail ? System was under the heavy load and thread which was executing the write operation failed with out of memory. >- process crashed or was terminated forcefully ? crashed
(In reply to Matthias Sohn from comment #2) > Here is a fix > https://git.eclipse.org/r/c/jgit/jgit/+/204213 > assuming you meant that if the JVM shuts down gracefully during repacking of > packed-refs the file lock isn't removed. Thank you for the pointer to the fix. This will definitely help when we do graceful shutdown. In our case one of the Gerrit threads failed due to heavy load on the system and it left the lock file so all other operation were blocked. Currently we have a cron job which is checking if there is a packed-refs lock file older than x. Looks like it's a common problem: https://groups.google.com/g/repo-discuss/c/82BOvCYQ4m8/m/WkpWwIp3AwAJ
(In reply to Matthias Sohn from comment #2) > Here is a fix > https://git.eclipse.org/r/c/jgit/jgit/+/204213 the fix was merged as d4d6c2b5af9b984ae824fb0073e1b368b39b1aa4
The problem keeps on happening on Gerrit. In the meantime I've created a simple Groovy plugin that can check for packed-refs.lock staleness, log the issue and remove the lock automatically. See https://gerrit-review.googlesource.com/c/plugins/scripts/+/393337
