Paul Eggert <eggert@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Received: (at 71529) by debbugs.gnu.org; 14 Jun 2024 00:07:41 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jun 13 20:07:41 2024 Received: from localhost ([127.0.0.1]:39426 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1sHuTk-0004Az-U5 for submit <at> debbugs.gnu.org; Thu, 13 Jun 2024 20:07:41 -0400 Received: from mail.cs.ucla.edu ([131.179.128.66]:58110) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eggert@HIDDEN>) id 1sHuTf-0004Ai-3d for 71529 <at> debbugs.gnu.org; Thu, 13 Jun 2024 20:07:39 -0400 Received: from localhost (localhost [127.0.0.1]) by mail.cs.ucla.edu (Postfix) with ESMTP id 70FB23C005151; Thu, 13 Jun 2024 17:07:28 -0700 (PDT) Received: from mail.cs.ucla.edu ([127.0.0.1]) by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavis, port 10032) with ESMTP id ZNX7nN0WMFBu; Thu, 13 Jun 2024 17:07:28 -0700 (PDT) Received: from localhost (localhost [127.0.0.1]) by mail.cs.ucla.edu (Postfix) with ESMTP id 265ED3C00515C; Thu, 13 Jun 2024 17:07:28 -0700 (PDT) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 265ED3C00515C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu; s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1718323648; bh=XG0e6d3L9ZxyYGsiCilxZvXd5FKfK1uLBRp1IZymTts=; h=Message-ID:Date:MIME-Version:To:From; b=n/CoJFa7QbECRj/HmfUFSRTpKhMs4cfbHYZY2QKvcHx9z01Z7TvpQ24j8ebA7V8US nEm2v0ivSHdjO4XMqDnKAPQ/NXVn5GFRw7N7lk1Jej7TtsMUxwbsOYDSOGaNRnk13R 5QABdKVLQoiMGxaEXwgRUT4xBky0qAD/eJEDkZgyP8kIq9N+lLcPJ/W4w+ie6Q4jzf 520GZGBZSt7zEDsHzY5dWFoI8zaoinaihLUzoqubRzZU4se3VdATrQMfnQcAU7zIEK sln53auDCCxbaFo0NX+q2VQRwyWdKG8m2NMejg34Gj9f0p3JVZS3E9Xh42YrPpJcUF NL1E/a/7UKmCQ== X-Virus-Scanned: amavis at mail.cs.ucla.edu Received: from mail.cs.ucla.edu ([127.0.0.1]) by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id s14xz_pgKLeq; Thu, 13 Jun 2024 17:07:28 -0700 (PDT) Received: from [131.179.64.200] (Penguin.CS.UCLA.EDU [131.179.64.200]) by mail.cs.ucla.edu (Postfix) with ESMTPSA id 0C4133C005151; Thu, 13 Jun 2024 17:07:28 -0700 (PDT) Message-ID: <911834cc-bb37-4e6a-b6bf-0fede634a4f8@HIDDEN> Date: Thu, 13 Jun 2024 17:07:26 -0700 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [bug-diffutils] bug#71529: Crash for large directory due to memory outage To: Evan Greenup <evan_greenup@HIDDEN> References: <FGcQ0XZ8J96cxw9QV0rMr2vM2emVn1VwXhWy5W40FktIwK4cKqzkDP6L55mcxv43Y5nMPLqbFy_xNiTMDXZl_BCZI3qxzUohGar48ktt33U=@protonmail.com> Content-Language: en-US From: Paul Eggert <eggert@HIDDEN> Autocrypt: addr=eggert@HIDDEN; keydata= xsFNBEyAcmQBEADAAyH2xoTu7ppG5D3a8FMZEon74dCvc4+q1XA2J2tBy2pwaTqfhpxxdGA9 Jj50UJ3PD4bSUEgN8tLZ0san47l5XTAFLi2456ciSl5m8sKaHlGdt9XmAAtmXqeZVIYX/UFS 96fDzf4xhEmm/y7LbYEPQdUdxu47xA5KhTYp5bltF3WYDz1Ygd7gx07Auwp7iw7eNvnoDTAl KAl8KYDZzbDNCQGEbpY3efZIvPdeI+FWQN4W+kghy+P6au6PrIIhYraeua7XDdb2LS1en3Ss mE3QjqfRqI/A2ue8JMwsvXe/WK38Ezs6x74iTaqI3AFH6ilAhDqpMnd/msSESNFt76DiO1ZK QMr9amVPknjfPmJISqdhgB1DlEdw34sROf6V8mZw0xfqT6PKE46LcFefzs0kbg4GORf8vjG2 Sf1tk5eU8MBiyN/bZ03bKNjNYMpODDQQwuP84kYLkX2wBxxMAhBxwbDVZudzxDZJ1C2VXujC OJVxq2kljBM9ETYuUGqd75AW2LXrLw6+MuIsHFAYAgRr7+KcwDgBAfwhPBYX34nSSiHlmLC+ KaHLeCLF5ZI2vKm3HEeCTtlOg7xZEONgwzL+fdKo+D6SoC8RRxJKs8a3sVfI4t6CnrQzvJbB n6gxdgCu5i29J1QCYrCYvql2UyFPAK+do99/1jOXT4m2836j1wARAQABzSBQYXVsIEVnZ2Vy dCA8ZWdnZXJ0QGNzLnVjbGEuZWR1PsLBlQQTAQgAPwIbAwYLCQgHAwIGFQgCCQoLBBYCAwEC HgECF4AWIQR+N5Kp2Kz31jO8FYjtl+kOYqp+NAUCZiLOewUJHWQLDAAKCRDtl+kOYqp+NHGE D/9Wmbk+cAaQsYLPGBvyzIjZIRzo/V2p3ZwckVA1VEQivx5azu1cs86qDoVIe45AtwmKOvdV wTQd/QeglkZR6D2YPW7UR/7emajyJZZcy+etVTDKoaw1i6/hmd/CpGjUeUSvgoPs6nYR+1lo pSXTpaGrh1W0qQHalSkOOwCHG3HtGk9Ve2AERDUYxmcn8/eZHb7xpUJEJMBBI1bx/zcw1EtB rjsQ1R1faJ/r/7LPAyV36RLvnbX69PylHKQEbJoaY9aUb2Vpm63ni3FeTA7/3jpPvaSRWHJh vPYx6Fm2Ln8pI0Yf/W2B8QMiPTnF/LnH2kvUcf9VXm+1mQJ3fBFU25HZwBhuqZ24IeKymPEt BUMQAum97Dto0jSgR2OUvX7z+twhpQEgRGBzPHYwDi4SxF5Z4Q5Y7B7a++HP9tIxG6CVFIwI 4xVaZud18bPa0YBL+cISmMgxq7h7yoVXl6u3pm9Yiv+W6Lp9QGN8Rw1VuJMOoFCYuoxG8mXO TA5b1jvlQ32gHFFhqErDAhNJRsfgrpe9Gok4Ycp+rWljbvS5Wrl0uth5MP7FbaHN2kmTZibq KXAd//IqczhDyU6qnW6ao+h4iDBDgYgRbQjmToX/vmIdEMzvPGqWXKhe/q1TYMuOO+IfP+bI fyPFH29nVN/o9c4J7myeKvv3HKSXdSVjlh2V787BTQRMgHJkARAApoXrvxP3DIfjCNOtXU/P dwMShKdX/RlSs5PfunV1wbKP8herXHrvQdFVqECaTSxmlhzbk8X0PkY9gcVaU2O49T3qsOd1 cHeF52YFGEt0LhsBeMjgNX5uZ1V76r8gyeVlFpWWb0SIwJUBHrDXexF67upeRb2vdHBjYDNe ySn+0B7gFEqvVmZu+LadudDp6kQLjatFvHQHUSGNshBnkkcaTbiI9Pst0GCc2aiznBiPPA2W QxAPlPRh3OGTsn5THADmbjqY6FEMLasVX8DSCblMvLwNeO/8SxziBidhqLpJCqdQRWHku5Xx gIkGeKOz5OLDvXHWJyafrEYjjkS6Ak6B5z6svKliClWnjHQcjlPzyoFFgKTEfcqDxCj4RY0D 0DgtFD0NfyeOidrSB/SzTe2hwryQE3rpSiqo+0cGdzh4yAHKYJ+UrXZ4p93ZhjGfKD1xlrNY DlWyW9PGmbvqFuDmiIAQf9WD/wzEfICc+F+uDDI+uYkRxUFp92ykmdhDEFg1yjYsU8iGU69a Hyvhq36z4zctvbqhRNzOWB1bVJ/dIMDvsExGcXQVDIT7sDNXv0wE3jKSKpp7NDG1oXUXL+2+ SF99Kjy753AbQSAmH617fyBNwhJWvQYg+mUvPpiGOtses9EXUI3lS4v0MEaPG43flEs1UR+1 rpFQWVHo1y1OO+sAEQEAAcLBfAQYAQgAJgIbDBYhBH43kqnYrPfWM7wViO2X6Q5iqn40BQJm Is58BQkdZAsMAAoJEO2X6Q5iqn40Q68QAJ9GubS/ej30Vc4idoZdc0IyMcL7kQJbMohF+Tyn ZE+TGn9WvzP10yLyzoI0vNlcNfP92d2MS//pFjOuANb5mwyiEYA+rDZIdS4ZZpHxCs2sxMC4 afLCf3kv4aMnTeBvb9na403dlczz9cAacvsmniSFdpb1+BzMpYbybglU5oYMGhYT2nnCRjXN 6S2nKYt4mjJeeOuxHrdeqQQdVBNYeNfTcPePeqvZ2+bD6u9yxZtaV+wxdpqglosQvjqhOYz7 h50/ZTSq70/npoCq44TzdJKttaYvlW6ziRz0g4RRAqZyoxjYXiy5qj8r8zXJuB11ApZCGuKn /usbji9RYbflAhxFeh4LMmpDVi6BrF30b73Md59K7PuEKN1NxzlWiqqQHZZ9momN0GXLPcGq 4uyfq7yVEy7wP5PMOh6oqscKklE3gFQtq0P1Ki0xqdF6Fq5LPJc+0Db2CYkVIy7Xaa/f74I3 sOfQfEeDylVXR5iDfUJEYv/0DYhOr7q5/0b1kh3M4wkrB4C5jVNHjIIj+RsAK90c3t38OhAl jiSN7Bkwy24Afy8eIu6wWzvhnsQGpZPB+IffmxT1wkTy8UxZKjUWV0C82iphVgCUUi2f9sDV Q/tNcwVWmOS+gdv9Wk6tdGeM+Ee+Qs6YG05jcSoajzF0TL07ajLcayRq2j1Os2CtQ8qu Organization: UCLA Computer Science Department In-Reply-To: <FGcQ0XZ8J96cxw9QV0rMr2vM2emVn1VwXhWy5W40FktIwK4cKqzkDP6L55mcxv43Y5nMPLqbFy_xNiTMDXZl_BCZI3qxzUohGar48ktt33U=@protonmail.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: -0.0 (/) X-Debbugs-Envelope-To: 71529 Cc: 71529 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) What do you mean by "crash"? Did you need to reboot the system, or did "diff" dump core, or did diff print a diagnostic "memory exhausxted"? Can you send a recipe for reproducing the problem? I didn't have any trouble running diff 3.10 on a Fedora 40 machine with only 8 GiB RAM, in directories with 100,000 files. The following shell commands succeeded quietly: mkdir d e (cd d && touch $(seq 100000)) (cd e && touch $(seq 100000)) diff -r d e
bug-diffutils@HIDDEN:bug#71529; Package diffutils.
Full text available.Received: (at submit) by debbugs.gnu.org; 13 Jun 2024 05:23:49 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jun 13 01:23:49 2024 Received: from localhost ([127.0.0.1]:51704 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1sHcw8-0007Gp-Ay for submit <at> debbugs.gnu.org; Thu, 13 Jun 2024 01:23:49 -0400 Received: from lists.gnu.org ([209.51.188.17]:39988) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <evan_greenup@HIDDEN>) id 1sHWdP-0006sX-Pd for submit <at> debbugs.gnu.org; Wed, 12 Jun 2024 18:40:04 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <evan_greenup@HIDDEN>) id 1sHWdQ-0001VA-BB for bug-diffutils@HIDDEN; Wed, 12 Jun 2024 18:40:04 -0400 Received: from mail-0301.mail-europe.com ([188.165.51.139]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <evan_greenup@HIDDEN>) id 1sHWdO-0006d6-50 for bug-diffutils@HIDDEN; Wed, 12 Jun 2024 18:40:03 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=protonmail.com; s=protonmail3; t=1718231995; x=1718491195; bh=HCTCY6+lNhfpKuL6+IOelzEiEV0emQPcG9UrZjX/X/A=; h=Date:To:From:Subject:Message-ID:Feedback-ID:From:To:Cc:Date: Subject:Reply-To:Feedback-ID:Message-ID:BIMI-Selector; b=gKpekcBwnc8zr2a4sSHXV6cls6A6dLeV+pGYJ1INbNWaQsc+TkhK/8OHB9muvO/ce wOYFyeIlDCpv8BeyVmCQBoHKEv1tH4+OlRtQVgJ4RufzwESiwIIHmRdW2/2FNYwbae thwqiTXcdieZMQ2dqt5yZLA8HmaS/PWVaBy7+7JvmtjtdKmyOlelNKddgdlaDx2xvE 5FwYdePY+JNxYHd+nhnZLkuqlEGkHqfP0YWgl+8DPugJ/KnpyXarw3slu3Ja/NWfas DKwCTp9EcJwzVMe9bBAbv3mB1JKGO4cUJg72HuRz9MoPBRKbe0S5Gt5kwrvi6VqNUU Co+d7CRZoF3vQ== Date: Wed, 12 Jun 2024 22:39:51 +0000 To: "bug-diffutils@HIDDEN" <bug-diffutils@HIDDEN> From: Evan Greenup <evan_greenup@HIDDEN> Subject: Crash for large directory due to memory outage Message-ID: <FGcQ0XZ8J96cxw9QV0rMr2vM2emVn1VwXhWy5W40FktIwK4cKqzkDP6L55mcxv43Y5nMPLqbFy_xNiTMDXZl_BCZI3qxzUohGar48ktt33U=@protonmail.com> Feedback-ID: 9586382:user:proton X-Pm-Message-ID: 27a4eb8797d8b1417faa5169d0881751be1ee8bf MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Received-SPF: pass client-ip=188.165.51.139; envelope-from=evan_greenup@HIDDEN; helo=mail-0301.mail-europe.com X-Spam_score_int: -27 X-Spam_score: -2.8 X-Spam_bar: -- X-Spam_report: (-2.8 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001, RCVD_IN_DNSWL_LOW=-0.7, SPF_HELO_NONE=0.001, SPF_PASS=-0.001, T_SCC_BODY_TEXT_LINE=-0.01 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -0.6 (/) X-Debbugs-Envelope-To: submit X-Mailman-Approved-At: Thu, 13 Jun 2024 01:23:40 -0400 X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.3 (--) Hi, The `diff` command in `diffutils` can compare the difference between two d= irectories.=20 However, for large directories (the size for each file is small), when trav= ersal through them, `diff` command will eat up all the system memory. My co= mputer is 16GB RAM. And I believe this issue still exist for system with 32= GB or 64GB RAM. I would suggest that it can use more memory efficient algorithm to compare = the difference between directories. Regards!!! Evan Greenup.
Evan Greenup <evan_greenup@HIDDEN>:bug-diffutils@HIDDEN.
Full text available.bug-diffutils@HIDDEN:bug#71529; Package diffutils.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.