Community
Participate
Working Groups
Support the ability to do a "shallow clone" with the CloneCommand to match the functionality of "git clone --depth". This is desirable when just the code in a repo is needed, but there is a large amount of irrelevant history that the user does not want to download. This ticket "UploadPack: implement shallow clone" (https://bugs.eclipse.org/bugs/show_bug.cgi?id=301627) from 2010 seems to imply that some sort of shallow clone support was added, but there is no tangible references to it on the internet or in the docs.
In case anyone is curious about a use-case for this feature, I am using JGit in CommandBox-- a CLI, REPL, Package Manager, and Embedded Server for CFML devs. We use JGit to install packages from Git endpoints via clone. Some larger repos with a lot of history can cause quite a bit of unnecessary download when the entire repo is copied. http://ortus.gitbooks.io/commandbox-documentation/content/packages/endpoints/git.html If there is anything I can do to help implement this, let me know. I wouldn't mind trying to take a stab at it if someone can point me in the right direction.
Patches are welcome, see https://wiki.eclipse.org/EGit/Contributor_Guide
Thank you for the link to the contributor guide Andrey. I would need more help along the lines of what classes to modify and what kinds of things to change though :)
Another use case: JGit is often used with Jenkins because of its hassle-free installation and configuration compared to Git. Depending on your repo content shallow clones can save a lot of download traffic, disc space and time. And as CI-jobs on Jenkins are often executed many times a day the savings would really add up.
Dear all, I am currently playing around with the JGit source-code and this bug (you can view my experiments made so far at https://github.com/timeraider4u/jgit in the branch called depth). I have added test cases for the CloneCommand at https://github.com/timeraider4u/jgit/blob/depth/org.eclipse.jgit.test/tst/org/eclipse/jgit/api/CloneCommandTest.java#L659 The Git repository which is set-up and used by these JUnit test cases can also be found at https://github.com/timeraider4u/jgit_test_repo I have also attached a file which contains hyperlinks to the most relevant source code parts (from my point of view). I have found a description of the protocol at https://github.com/git/git/blob/master/Documentation/technical/pack-protocol.txt#L243 and https://github.com/git/git/blob/master/Documentation/technical/shallow.txt You can also use GIT_TRACE_PACKET=1 git clone -b master --depth 2 https://github.com/timeraider4u/jgit_test_repo.git to print some debugging information about the Git server-client transfer protocol. But, unfortunately, now I am stuck when I am trying to read the "shallow <GIT_COMMIT_SHA1_HASH>" because an exception is thrown. In the git trace I can also see the line "git< 008cwant c21676d47a0f506cfa26596c5bb8f33430603054 multi_ack_detailed no-done side-band-64k thin-pack include-tag ofs-delta agent=git/2.10.2000cdeepen 20000" which seems to not appear as a JGit packet. Seems like I am missing some very important point...
Created attachment 266181 [details] hyperlink highlights which seem relevant for this bug
+1, this feature will really save a lot of disk space in some scenarios. Harald, any news on the pending bugs?
@George: Well, I have not tried again because I do not understand how JGit is implementing the git protocol (or maybe I mis-understand the git protocol itself). At some point I am simply stuck.
@Harald yeah, that looks complicated. Can someone from the jGit core team help in this?
I have sent an email to the JGit developers mailing list at https://dev.eclipse.org/mailman/listinfo/jgit-dev Hopefully there is someone out there to help...
is shallow clone implemented in jgit?
(In reply to anu chawla from comment #11) > is shallow clone implemented in jgit? No.
New Gerrit change created: https://git.eclipse.org/r/102957
I have pushed my changes made so far to Eclipse Gerrit, if anybody wants to play around with it (the code also includes some temporary System.out s debug code). As already mentioned in comment#5, shallow clone is not working yet because the whole subject is still work-in-progress. There is still no response on the mailing list. Any help is welcome. BTW: In the changes proposed so far, you can find file /org.eclipse.jgit.test/tst/org/eclipse/jgit/api/CloneCommandTest.java with the added methods testCloneRepositoryWithDepth1 and testCloneRepositoryWithDepth2 which will both fail with the following exception "org.eclipse.jgit.errors.PackProtocolException: Expected ACK/NAK, got: shallow c21676d47a0f506cfa26596c5bb8f33430603054". So if you have any idea, please let me know.
> So if you have any idea, please let me know. You'd have to implement the whole negotiation. It's not enough to tell server the desired depth. The server will then respond with these shallow lines to tell the client which will be the boundary commits it'll get. (I.e., the commits of which the server will not send the parents even though they exist.) The client is supposed to store these, too (in $GITDIR/shallow); so that on a subsequent shallow fetch the client can tell the server what the current boundary is. If the client sends shallow lines, the server may also respond with unshallow lines to inform the client that formerly boundary commits will no longer be on the boundary. See https://github.com/git/git/blob/master/Documentation/technical/pack-protocol.txt#L275 . The $GIT_DIR/shallow file needs to be locked when written. And I suppose it should be updated only at if the pack file had been received and integrated correctly. For such details, you'd have to study the C-git implementation...
@Thomas: In class /org.eclipse.jgit/src/org/eclipse/jgit/transport/BasePackFetchConnection.java in method sendWants(...), the client sends the "want efa94f4e27ac48a8277036443e18c5b23fe5baa9" line, followed by a "deepen 1" line to the server. The server responds with a "shallow c21676d47a0f506cfa26596c5bb8f33430603054" line. This line is not handled properly by the client at the moment. And that is exactly where I am stuck at the moment. I do not know where and how I should read this line correctly within JGit source code. The method negotiate(...) in class BasePackFetchConnection seems to be the right place for reading this line(s). But it is a little bit confusing as it is quite complex. That's the point where I would need help: Where should I read this line(s). At the moment I am not handling the whole shallow/unshallow stuff. I will leave this open for later once the deepen-handling is working. Then I will try to create additional JUnit tests for complete shallow/unshallow negotiation.
Also according to https://github.com/git/git/blob/master/Documentation/technical/shallow.txt#L39 locking of $GITDIR/shallow is not mandatory ;-).
(In reply to Harald Weiner from comment #17) > Also according to > https://github.com/git/git/blob/master/Documentation/technical/shallow. > txt#L39 locking of $GITDIR/shallow is not mandatory ;-). It is. That mention is in in a list with the heading "There are some unfinished ends of the whole shallow business:" . Moreover, a FileRepository in JGit currently is thread-safe,[1] and should remain so. Also, it's probably the easiest to use the already existing lock mechanisms in JGit for this instead of implementing something else. Also note that this "shallow.txt" is from 2007, and is essentially unchanged since then.[2] One really has to study the C-git implementation to check for details; I wouldn't be surprised if the current code had fixed some of these issues. If you look at the history of shallow.c,[3] you'll see that it has had quite a few edits. And that it nowadays locks the file. [1] https://git.eclipse.org/c/jgit/jgit.git/tree/org.eclipse.jgit/src/org/eclipse/jgit/lib/Repository.java#n109 [2] https://github.com/git/git/blame/master/Documentation/technical/shallow.txt#L39 [3] https://github.com/git/git/blob/master/shallow.c
@Thomas: Yes, you are right that locking the file $GIDIR/shallow will be the right solution. Anyway, the creation/locking is only useful once reading the line "shallow <commit-id>" that is sent as a server response to the client is working correctly.
Created attachment 269815 [details] git clone without depth limit GIT_TRACE_PACKET=1 git clone -b master https://github.com/timeraider4u/jgit_test_repo.git jgit_test_repo_full &> git-clone-depth-unlimited.log
Created attachment 269816 [details] git clone without depth limit Output of the following command: GIT_TRACE_PACKET=1 git clone -b master https://github.com/timeraider4u/jgit_test_repo.git jgit_test_repo_full &> git-clone-depth-unlimited.log
Created attachment 269817 [details] git clone with depth limit 2000 (which is more than commit history length) Output of the following command: GIT_TRACE_PACKET=1 git clone -b master --depth 2000 https://github.com/timeraider4u/jgit_test_repo.git jgit_test_repo_more_depth &> git-clone-depth-2000.log
Created attachment 269818 [details] git clone with depth limit 2 (at least one commit will become shallow) Output of the following command: GIT_TRACE_PACKET=1 git clone -b master --depth 2 https://github.com/timeraider4u/jgit_test_repo.git &> git-clone-depth-2.log
Okay, I think I made a progress: If there is a server response to the client which contains some shallow/unshallow lines, the client also needs to read the flush-line (otherwise the PacketLineIn.readACK(...) method will fail). Seems I have missed this fact previously or not properly used this knowledge on the JGit implementation. So the best place to properly handle the shallow/unshallow lines might be in the "READ_RESULT: while " loop in class /org.eclipse.jgit/src/org/eclipse/jgit/transport/BasePackFetchConnection.java Will report back, as soon as I know more.
I mean BEFORE that loop, sorry for that typo.
use the existing class org.eclipse.jgit.internal.storage.file.LockFile to implement file locks
Okay, it seems that I managed to implement the reading of the shallow lines (see recent changes in Gerrit Code Review). Now I would need to obtain the file $GIDIR/shallow, lock it, read its contain and then later write any changes to it and unlock it again. What would be the best library functions to achieve this? I have seen that with org.eclipse.jgit.lib.Repository.getDirectory() I can get the $GITDIR as java.io.File. Now I can do File shallowFile = new File(gitDir, "shallow"); And locking should be possible with org.eclipse.jgit.internal.storage.file.LockFile lockShallowFile = new LockFile(shallowFile); lockShallowFile.lock(); LockFile also seems to support writing to it. Is this how these methods are supposed to be used? And if yes, is there also any library function that should be used for reading from a file?
@Matthias: Thank you! Seems like that you were a little bit faster with posting ;-).
It seems that git.log().call() ignores $GITDIR/shallow. Because if I manually execute git log for the git repo created by JUnit test testCloneRepositoryWithDepth1 it now shows me only one commit: the final commit.
Forget my last comment! After investigating this weird behaviour, I finally found out that I have made a classical typo and used "git.log().call()" instead of "git2.log().call()" in my JUnit tests which of course returns wrong results. So by now my JUnit tests for git clone --depth <depth> work. Proper reading of $GITDIR/shallow lines and sending them to the server is still missing. Also unit tests for this would be nice. Maybe I will add this functionality in the next few weeks...
(In reply to Harald Weiner from comment #27) > I have seen that with org.eclipse.jgit.lib.Repository.getDirectory() I can > get the $GITDIR as java.io.File. Now I can do File shallowFile = new > File(gitDir, "shallow"); And locking should be possible with > org.eclipse.jgit.internal.storage.file.LockFile lockShallowFile = new > LockFile(shallowFile); lockShallowFile.lock(); LockFile also seems to > support writing to it. see [1] for an example how to use LockFile [1] https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/refs/heads/master/org.eclipse.jgit/src/org/eclipse/jgit/storage/file/FileBasedConfig.java#203
(In reply to Matthias Sohn from comment #31) > (In reply to Harald Weiner from comment #27) > > I have seen that with org.eclipse.jgit.lib.Repository.getDirectory() I can > > get the $GITDIR as java.io.File. Now I can do File shallowFile = new > > File(gitDir, "shallow"); And locking should be possible with > > org.eclipse.jgit.internal.storage.file.LockFile lockShallowFile = new > > LockFile(shallowFile); lockShallowFile.lock(); LockFile also seems to > > support writing to it. > > see [1] for an example how to use LockFile > > [1] > https://git.eclipse.org/r/plugins/gitiles/jgit/jgit/+/refs/heads/master/org. > eclipse.jgit/src/org/eclipse/jgit/storage/file/FileBasedConfig.java#203 Any idea by when we can expect this bug would be resolved?
@Matthias: Thank you for the link you provided, it answered my questions. @Ankin: There are still some TODOs left (JUnit tests to write, code to implement, issues to resolve) but in general the stuff left to do should not be complicated any more. I am planning to fix this open points until end of August and then get it ready for review by project maintainers/any other interested party. Will report back in two weeks as I am quite busy now.
(In reply to Harald Weiner from comment #33) > @Matthias: Thank you for the link you provided, it answered my questions. > > @Ankin: There are still some TODOs left (JUnit tests to write, code to > implement, issues to resolve) but in general the stuff left to do should not > be complicated any more. I am planning to fix this open points until end of > August and then get it ready for review by project maintainers/any other > interested party. Will report back in two weeks as I am quite busy now. @Harald: Thanks, do you any update on the same.
My current efforts to fix this bug can be found at https://git.eclipse.org/r/#/c/102957/ I think the code submitted has reached a level where I can ask for feedback, suggestions and review by the community. Please give it a try and let me know if you find any regressions. I would also welcome any feedback on improving the code or unit testing by any subscribers of this bug. Two known limitations of the current implementation are: 1.) Shallow cloning of tags, like cloneCommand.setDepth(1).setBranch("refs/tags/some-tag"), is not supported (yet)! There are some minor and some more serious problems which prevent fixing this (e.g., unwanted side-effects when a deepen line is sent by BasePackFetchConnection for a tag). At the moment, this will simply result in an exception. See https://git.eclipse.org/r/#/c/102957/16/org.eclipse.jgit.test/tst/org/eclipse/jgit/api/CloneCommandTest.java@1000 for an example. 2.) Fetching inside an existing repository with a fixed depth to deepen/shallow/unshallow/truncate the commit history will not work. JGit will not send a request for any commit-id that is already known locally. At the moment, it will just do nothing. See https://git.eclipse.org/r/#/c/102957/16/org.eclipse.jgit.test/tst/org/eclipse/jgit/api/CloneCommandTest.java@768 for an example. I would recommend that this is later treated in a separate bug report and with a separate change request.
I've tried to use this patch in order to perform shallow cloning, but I get the following exception: "Starting read stage without written request data pending is not supported" Full Stack trace: Caused by: org.eclipse.jgit.errors.TransportException: https://github.com/marmots-inc/chocolate-factory.git: Starting read stage without written request data pending is not supported at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:383) at org.eclipse.jgit.transport.TransportHttp$SmartHttpFetchConnection.doFetch(TransportHttp.java:1090) at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:310) at org.eclipse.jgit.transport.BasePackFetchConnection.fetch(BasePackFetchConnection.java:299) at org.eclipse.jgit.transport.FetchProcess.fetchObjects(FetchProcess.java:246) at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:162) at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:123) at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1262) at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:243) ... 27 more Caused by: org.eclipse.jgit.errors.TransportException: https://github.com/marmots-inc/chocolate-factory.git: Starting read stage without written request data pending is not supported at org.eclipse.jgit.transport.TransportHttp$MultiRequestService.execute(TransportHttp.java:1412) at org.eclipse.jgit.transport.TransportHttp$Service$HttpExecuteStream.read(TransportHttp.java:1345) at org.eclipse.jgit.util.io.UnionInputStream.read(UnionInputStream.java:145) at org.eclipse.jgit.util.IO.readFully(IO.java:247) at org.eclipse.jgit.transport.PacketLineIn.readLength(PacketLineIn.java:225) at org.eclipse.jgit.transport.PacketLineIn.readString(PacketLineIn.java:155) at org.eclipse.jgit.transport.PacketLineIn.readACK(PacketLineIn.java:119) at org.eclipse.jgit.transport.BasePackFetchConnection.negotiate(BasePackFetchConnection.java:708) at org.eclipse.jgit.transport.BasePackFetchConnection.doFetch(BasePackFetchConnection.java:369) ... 35 more
Is shallow clone supported in latest JGit 4.11 ?
Is this not still actively pursued?
(In reply to premshaw kaliyamoorthi from comment #38) > Is this not still actively pursued? it seems the series started by Harald stalled https://git.eclipse.org/r/#/c/103003/ and was auto-abandoned. If anyone wants to continue working on this you can restore these changes.
(In reply to Matthias Sohn from comment #39) > (In reply to premshaw kaliyamoorthi from comment #38) > > Is this not still actively pursued? > > it seems the series started by Harald stalled > https://git.eclipse.org/r/#/c/103003/ > and was auto-abandoned. If anyone wants to continue working on this you can > restore these changes. How much work still remains? Could someone who is not familiar at all with how JGit internally works get to up to speed to finish the work?
This is still needed. Anyone available to finish the fix that Harald started?
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/193329
(In reply to Eclipse Genie from comment #42) > New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/193329 On which release will this change be published?
Please pardon my ignorance but I'm kind of interested in using the `--filter=blob:none` feature of git and felt this was pretty similar. I've been finding that if I try to use this flag when cloning with gerrit, I get errors like this for every file ``` $ git clone --filter=blob:none ssh://my-gerrit:29418/my-repo Cloning into 'my-repo'... remote: Finding sources: 100% (262/262) remote: Getting sizes: 100% (123/123) Receiving objectsremote: Total 262 (delta 71), reused 262 (delta 71) Receiving objects: 100% (262/262), 30.23 KiB | 2.16 MiB/s, done. Resolving deltas: 100% (71/71), done. fatal: remote error: want a8fb9aafcda4a7a4e82faf2edbbc09e4dc11a580 not valid fatal: internal server error fatal: remote error: want a1dbc63aca26b49cc110581cb173f4ba5ca4b0eb not valid fatal: internal server error error: unable to read sha1 file of .gitignore (a1dbc63aca26b49cc110581cb173f4ba5ca4b0eb) fatal: remote error: want ad4ecb53ac5a63682bdd5a59e9f7dc98841bca2b not valid error: unable to read sha1 file of README.md (ad4ecb53ac5a63682bdd5a59e9f7dc98841bca2b) fatal: internal server error fatal: remote error: want a03da8bbb9d9316c899fe832fcbd7f9f8f659a44 not valid error: unable to read sha1 file of build.gradle (a03da8bbb9d9316c899fe832fcbd7f9f8f659a44) ... ``` However, if I try to clone over http instead, it clones successfully. ``` git clone --filter=blob:none "https://my-gerrit:8443/a/my-repo" Cloning into 'my-repo'... info: detecting host provider for 'https://my-gerrit:8443/'... info: detecting host provider for 'https://my-gerrit:8443/'... remote: Finding sources: 100% (262/262) remote: Getting sizes: 100% (123/123) Receiving oremote: Total 262 (delta 71), reused 262 (delta 71) Receiving objects: 100% (262/262), 30.23 KiB | 2.52 MiB/s, done. Resolving deltas: 100% (71/71), done. info: detecting host provider for 'https://my-gerrit:8443/'... info: detecting host provider for 'https://my-gerrit:8443/'... remote: Counting objects: 44, done remote: Finding sources: 100% (44/44) remote: Getting sizes: 100% (43/43) remote: Compressing objects: 100% (15618/15618) remote: Total 44 (delta 3), reused 36 (delta 1) Receiving objects: 100% (44/44), 97.14 KiB | 9.71 MiB/s, done. Resolving deltas: 100% (3/3), done. Updating files: 100% (46/46), done. ``` Are there different mechanisms under the hood for cloning over ssh vs https? When cloning over https, are the --filter/--depth flags even considered? I believe gerrit 3.6.1 is using jgit v6.1.0.202203080745-r.
Gerrit change https://git.eclipse.org/r/c/jgit/jgit/+/193329 was merged to [master]. Commit: http://git.eclipse.org/c/jgit/jgit.git/commit/?id=207dd4c938830e84c9101d30edb7fe626e04bbe1
(In reply to Eclipse Genie from comment #45) > Gerrit change https://git.eclipse.org/r/c/jgit/jgit/+/193329 was merged to > [master]. > Commit: > http://git.eclipse.org/c/jgit/jgit.git/commit/ > ?id=207dd4c938830e84c9101d30edb7fe626e04bbe1 Thank you, Robin! Now we still need to add the options to org.eclipse.jgit.pgm.Clone and org.eclipse.jgit.pgm.Fetch.
(In reply to Chad Hollman from comment #44) > Please pardon my ignorance but I'm kind of interested in using the > `--filter=blob:none` feature of git and felt this was pretty similar. I've > been finding that if I try to use this flag when cloning with gerrit, I get > errors like this for every file > > ``` > $ git clone --filter=blob:none ssh://my-gerrit:29418/my-repo > Cloning into 'my-repo'... > remote: Finding sources: 100% (262/262) > remote: Getting sizes: 100% (123/123) > Receiving objectsremote: Total 262 (delta 71), reused 262 (delta 71) > Receiving objects: 100% (262/262), 30.23 KiB | 2.16 MiB/s, done. > Resolving deltas: 100% (71/71), done. > fatal: remote error: want a8fb9aafcda4a7a4e82faf2edbbc09e4dc11a580 not valid > fatal: internal server error > fatal: remote error: want a1dbc63aca26b49cc110581cb173f4ba5ca4b0eb not valid > fatal: internal server error > error: unable to read sha1 file of .gitignore > (a1dbc63aca26b49cc110581cb173f4ba5ca4b0eb) > fatal: remote error: want ad4ecb53ac5a63682bdd5a59e9f7dc98841bca2b not valid > error: unable to read sha1 file of README.md > (ad4ecb53ac5a63682bdd5a59e9f7dc98841bca2b) > fatal: internal server error > fatal: remote error: want a03da8bbb9d9316c899fe832fcbd7f9f8f659a44 not valid > error: unable to read sha1 file of build.gradle > (a03da8bbb9d9316c899fe832fcbd7f9f8f659a44) > ... > ``` > > However, if I try to clone over http instead, it clones successfully. > > ``` > git clone --filter=blob:none "https://my-gerrit:8443/a/my-repo" > Cloning into 'my-repo'... > info: detecting host provider for 'https://my-gerrit:8443/'... > info: detecting host provider for 'https://my-gerrit:8443/'... > remote: Finding sources: 100% (262/262) > remote: Getting sizes: 100% (123/123) > Receiving oremote: Total 262 (delta 71), reused 262 (delta 71) > Receiving objects: 100% (262/262), 30.23 KiB | 2.52 MiB/s, done. > Resolving deltas: 100% (71/71), done. > info: detecting host provider for 'https://my-gerrit:8443/'... > info: detecting host provider for 'https://my-gerrit:8443/'... > remote: Counting objects: 44, done > remote: Finding sources: 100% (44/44) > remote: Getting sizes: 100% (43/43) > remote: Compressing objects: 100% (15618/15618) > remote: Total 44 (delta 3), reused 36 (delta 1) > Receiving objects: 100% (44/44), 97.14 KiB | 9.71 MiB/s, done. > Resolving deltas: 100% (3/3), done. > Updating files: 100% (46/46), done. > ``` > > Are there different mechanisms under the hood for cloning over ssh vs https? > When cloning over https, are the --filter/--depth flags even considered? > > I believe gerrit 3.6.1 is using jgit v6.1.0.202203080745-r. Hi, How did you enable partial clone in your gerrit server?
I didn't. Was hoping someone who worked on the project would be able to shed some light.
*** Bug 301628 has been marked as a duplicate of this bug. ***
Please discuss the --filter=blob:none problem in bug 580632. It has got nothing to do with shallow clones. (And it also seems to be a problem in the server side part.)
New Gerrit change created: https://git.eclipse.org/r/c/jgit/jgit/+/196477
Gerrit change https://git.eclipse.org/r/c/jgit/jgit/+/196477 was merged to [master]. Commit: http://git.eclipse.org/c/jgit/jgit.git/commit/?id=eb3a708676e3487bbd97df384c09595034034d7e
