LKML Archive on
help / color / mirror / Atom feed
From: Junio C Hamano <>
To: Linus Torvalds <>
Cc: Rolf Eike Beer <>,
	Git List Mailing <>,
	Tobias Ulmer <>,
	Linux Kernel Mailing List <>
Subject: Re: data loss when doing ls-remote and piped to command
Date: Thu, 16 Sep 2021 13:42:22 -0700	[thread overview]
Message-ID: <xmqq7dfgtfpt.fsf@gitster.g> (raw)
In-Reply-To: <> (Linus Torvalds's message of "Thu, 16 Sep 2021 10:11:22 -0700")

Linus Torvalds <> writes:

> On Thu, Sep 16, 2021 at 5:17 AM Rolf Eike Beer <> wrote:
>> Am Donnerstag, 16. September 2021, 12:12:48 CEST schrieb Tobias Ulmer:
>> > > The redirection seems to be an important part of it. I now did:
>> > >
>> > > git ... 2>&1 | sha256sum
>> >
>> > I've tried to reproduce this since yesterday, but couldn't until now:
>> >
>> > 2>&1 made all the difference, took less than a minute.
> So if that redirection is what matters, and what causes problems, I
> can almost guarantee that the reason is very simple:
> ...
> Anyway. That was a long email just to tell people it's almost
> certainly user error, not the kernel.

Yes, 2>&1 will mix messages from the standard error stream at random
places in the output, which explains the checksum quite well.

I am not sure if it explains the initial report where

	ls-remote 2>&1 | less


    > 6f38b5d6cfd43dde3058a10c68baae9cf17af912        refs/tags/v5.0-rc2
    > 1c7fc5cbc33980acd13ae83d0b416db002fe95601e7f97f64b59514d936     refs/tags/v5.7-rc2^{}
    > d0709bb6da2ab6d49b11643e98abdf79b1a2817f        refs/tags/v5.7-rc3

    What we see on the second line is the beginning of peeled
    v5.0-rc2^{} up to the "acd13" (that is, the first 19 bytes of the
    line), followed by the full line for peeled v5.7-rc2^{} (which
    begins with "ae83d").  12407 bytes in between are missing, which
    is even more puzzling as it is not a nice round number.

I can sort of guess that the progress display during transfer, which
comes out on the standard error stream and uses terminal control
sequences like "go back to the end of the line without feeding a new
line", "erase to the end of the line", etc., would be contributing,
but because it is piped to "less", which would make it "visible"
(i.e. you do not get the raw escape but see three capital letters
ESC in reverse), it does not quite explain how the display was

In any case, I do not think the kernel is involved, or more
generally I do not think any "loss of output bytes" is happening
here.  It's just "| less" that failed to show a range about 12k
bytes long is mystery to me ;-).

  reply	other threads:[~2021-09-16 20:42 UTC|newest]

Thread overview: 9+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <6786526.72e2EbofS7@devpool47>
     [not found] ` <2279155.Qy0YqsFniq@devpool47>
     [not found]   ` <>
2021-09-16 12:17     ` data loss when doing ls-remote and piped to command Rolf Eike Beer
2021-09-16 15:49       ` Mike Galbraith
2021-09-17  6:38         ` Mike Galbraith
2021-09-16 17:11       ` Linus Torvalds
2021-09-16 20:42         ` Junio C Hamano [this message]
2021-09-17  6:59           ` Rolf Eike Beer
2021-09-17 19:13             ` Jeff King
2021-09-17 19:28             ` Linus Torvalds
2021-09-18  6:33             ` Mike Galbraith

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=xmqq7dfgtfpt.fsf@gitster.g \ \ \ \ \ \ \

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).