Received: (at 32729) by debbugs.gnu.org; 25 Oct 2019 07:00:49 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 25 03:00:49 2019 Received: from localhost ([127.0.0.1]:37197 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iNtaa-0003k0-Rv for submit <at> debbugs.gnu.org; Fri, 25 Oct 2019 03:00:49 -0400 Received: from eggs.gnu.org ([209.51.188.92]:57832) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iNtaY-0003je-HO; Fri, 25 Oct 2019 03:00:47 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:33049) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iNtaS-00011r-Vb; Fri, 25 Oct 2019 03:00:41 -0400 Received: from [176.228.60.248] (port=2992 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iNtaS-0003p6-33; Fri, 25 Oct 2019 03:00:40 -0400 Date: Fri, 25 Oct 2019 10:00:25 +0300 Message-Id: <83woctwbqu.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: "Benninghofen\, Benjamin Dr." <benjamin.benninghofen@HIDDEN> In-reply-to: <d4651c91e5c5431d9e78b3bdd4bd5298@HIDDEN> (benjamin.benninghofen@HIDDEN) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN> <87r23fiu66.fsf@HIDDEN> <d4651c91e5c5431d9e78b3bdd4bd5298@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, larsi@HIDDEN, 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN> > CC: "layer@HIDDEN" <layer@HIDDEN>, "32729 <at> debbugs.gnu.org" > <32729 <at> debbugs.gnu.org>, "32728 <at> debbugs.gnu.org" <32728 <at> debbugs.gnu.org> > Date: Fri, 25 Oct 2019 06:38:16 +0000 > > For me your discussion about the performance problem appears very theoretical. I don't think it was theoretical. It was an attempt to identify which factor(s) affect(s) this issue, so that further investigation could be more focused. > Why do you not just compare the relevant GNU Emacs Source Code with the Xemacs Source Code? Did you try doing that? Every time I did I found that the code is so different that any comparison is very hard at best, and usually simply meaningless.
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.
Received: (at 32729) by debbugs.gnu.org; 25 Oct 2019 06:38:28 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 25 02:38:28 2019
Received: from localhost ([127.0.0.1]:37183 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1iNtEx-0003BH-Pe
for submit <at> debbugs.gnu.org; Fri, 25 Oct 2019 02:38:28 -0400
Received: from mo1.myeers.net ([87.190.7.232]:37247)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <benjamin.benninghofen@HIDDEN>)
id 1iNtEw-0003Az-Kl; Fri, 25 Oct 2019 02:38:27 -0400
X-IronPort-AV: E=Sophos;i="5.68,227,1569276000"; d="scan'208";a="108953666"
Received: from ec2-44-225-67-160.us-west-2.compute.amazonaws.com (HELO
DE0-03HUB-P03.central.mail.corp) ([44.225.67.160])
by de0-44iro-p02-out.myeers.net with ESMTP/TLS/AES256-SHA;
25 Oct 2019 08:38:19 +0200
Received: from esa2e.demail.de.airbusds.corp (10.67.144.34) by
DE0-03HUB-P03.central.mail.corp (44.225.67.164) with Microsoft SMTP Server id
15.0.1473.3; Fri, 25 Oct 2019 08:38:17 +0200
Received: from unknown (HELO CD1-4DDAG02-P01.cdmail.common.airbusds.corp)
([10.67.164.142])
by esa2i.demail.de.airbusds.corp with ESMTP; 25 Oct 2019 08:38:16 +0200
Received: from CD1-4BDAG02-P03.cdmail.common.airbusds.corp (10.67.164.139) by
CD1-4DDAG02-P01.cdmail.common.airbusds.corp (10.67.164.142) with
Microsoft
SMTP Server (TLS) id 15.0.1473.3; Fri, 25 Oct 2019 08:38:16 +0200
Received: from CD1-4BDAG02-P03.cdmail.common.airbusds.corp ([10.67.164.139])
by CD1-4BDAG02-P03.cdmail.common.airbusds.corp ([10.67.164.139]) with mapi id
15.00.1473.003; Fri, 25 Oct 2019 08:38:16 +0200
From: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
To: Lars Ingebrigtsen <larsi@HIDDEN>, Eli Zaretskii <eliz@HIDDEN>
Subject: RE: bug#32729: Xemacs 23 times as fast as GNU Emacs
Thread-Topic: bug#32729: Xemacs 23 times as fast as GNU Emacs
Thread-Index: AQHVgLFJdXJGPCQprkOKuUrCkgOMJKdWnzbCgACsFgKAAO+gGYAAoE9/gAAQkl+AAOzrSYARIZUQ
Date: Fri, 25 Oct 2019 06:38:16 +0000
Message-ID: <d4651c91e5c5431d9e78b3bdd4bd5298@HIDDEN>
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
<871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN>
<83r23hkqr3.fsf@HIDDEN> <87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN>
<87r23fiu66.fsf@HIDDEN>
In-Reply-To: <87r23fiu66.fsf@HIDDEN>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-transport-fromentityheader: Hosted
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
X-GM-Security: forwarded
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32729
Cc: "layer@HIDDEN" <layer@HIDDEN>,
"32729 <at> debbugs.gnu.org" <32729 <at> debbugs.gnu.org>,
"32728 <at> debbugs.gnu.org" <32728 <at> debbugs.gnu.org>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
For me your discussion about the performance problem appears very theoretic=
al.
Why do you not just compare the relevant GNU Emacs Source Code with the Xem=
acs Source Code?
Benjamin Benninghofen
-----Original Message-----
From: Lars Ingebrigtsen [mailto:larsi@HIDDEN] =
Sent: Monday, October 14, 2019 10:54 AM
To: Eli Zaretskii
Cc: layer@HIDDEN; 32729 <at> debbugs.gnu.org; Benninghofen, Benjamin Dr.; 327=
28 <at> debbugs.gnu.org
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
Actually, my benchmarking is somewhat wrong.
start-process with a filter, but discard output:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
(set-process-filter proc (lambda (proc string)))
(while (and (process-live-p proc)
(accept-process-output proc 1))))))
=3D> (18.828236636 59 13.315468088000017)
filter, but insert the output:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
(set-process-filter proc (lambda (proc string)
(with-current-buffer (get-buffer " *zeroes*")
(goto-char (point-max))
(insert string))))
(while (and (process-live-p proc)
(accept-process-output proc 1))))))
=3D> (21.120281346 59 13.250166416000013)
With the default filter:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
(while (and (process-live-p proc)
(accept-process-output proc 1))))))
=3D> (34.046986424 116 26.025843717999976)
(!)
So the default filter is really slow?
Anyway, compare with call-process:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") n=
il "if=3D/dev/zero" "bs=3D4096" "count=3D250000")))
=3D> (1.694743653 0 0.0)
So what makes start-process 10x slower than call-process? If it is all
the string creation before calling the filters, default or not, then my
point stands, but this obviously requires a more in-depth dive into
process.c.
-- =
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
The information in this e-mail is confidential. The contents may not be dis=
closed or used by anyone other than the addressee. Access to this e-mail by=
anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and=
delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of=
this e-mail as it has been sent over public networks. If you have any conc=
erns over the content of this message or its Accuracy or Integrity, please =
contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus =
scanning software but you should take whatever measures you deem to be appr=
opriate to ensure that this message and any attachments are virus free.
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 14 Oct 2019 10:18:50 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 06:18:50 2019 Received: from localhost ([127.0.0.1]:38188 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJxRB-0000YX-4w for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 06:18:50 -0400 Received: from eggs.gnu.org ([209.51.188.92]:59244) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iJxR6-0000YA-2X; Mon, 14 Oct 2019 06:18:44 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:50206) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iJxR0-0004u8-IA; Mon, 14 Oct 2019 06:18:38 -0400 Received: from [176.228.60.248] (port=3772 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iJxQz-0007S4-Tt; Mon, 14 Oct 2019 06:18:38 -0400 Date: Mon, 14 Oct 2019 13:18:30 +0300 Message-Id: <83y2xniqa1.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Lars Ingebrigtsen <larsi@HIDDEN> In-reply-to: <87r23fiu66.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 14 Oct 2019 10:54:25 +0200) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN> <87r23fiu66.fsf@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, benjamin.benninghofen@HIDDEN, 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen <larsi@HIDDEN> > Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, > benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org > Date: Mon, 14 Oct 2019 10:54:25 +0200 > > So what makes start-process 10x slower than call-process? The fact that it goes through pselect, and only accepts the process output when Emacs is idle? > this obviously requires a more in-depth dive into process.c. Agreed.
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 14 Oct 2019 09:15:25 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 05:15:25 2019 Received: from localhost ([127.0.0.1]:38143 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJwRp-0007Te-1O for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 05:15:25 -0400 Received: from eggs.gnu.org ([209.51.188.92]:50760) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iJwRm-0007TI-SG; Mon, 14 Oct 2019 05:15:23 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:49425) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iJwRh-0001hX-28; Mon, 14 Oct 2019 05:15:17 -0400 Received: from [176.228.60.248] (port=2273 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iJwRg-0006dy-CZ; Mon, 14 Oct 2019 05:15:16 -0400 Date: Mon, 14 Oct 2019 12:15:09 +0300 Message-Id: <8336fvk7s2.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Lars Ingebrigtsen <larsi@HIDDEN> In-reply-to: <87v9sriv0k.fsf@HIDDEN> (message from Lars Ingebrigtsen on Mon, 14 Oct 2019 10:36:11 +0200) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87sgnwbl8w.fsf@HIDDEN> <83blujkaf8.fsf@HIDDEN> <87v9sriv0k.fsf@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen <larsi@HIDDEN> > Cc: benjamin.benninghofen@HIDDEN, layer@HIDDEN, > 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org > Date: Mon, 14 Oct 2019 10:36:11 +0200 > > Eli Zaretskii <eliz@HIDDEN> writes: > > > I don't understand what would trigger these callbacks, and how do you > > specify the region in advance, without knowing what will be inserted. > > accept_process_output inserts the data into the buffer and then calls > the callback with the region in question. Well, > read_and_dispose_of_process_output, I guess... Filter functions are called even if the Lisp program never calls accept-process-output, so your proposal doesn't seem to be equivalent to what we have now, right? OTOH, if one has to call accept-process-output, then why do we need callbacks? Just extend accept-process-output to call a function with the received output. No? > > Without understanding this, I don't think I see the utility, and most > > important: why this would be faster. > > It would avoid creating (and garbaging) the strings. I'm not sure I see how. The way it works now is that we get the process output as a C string; we then decode it and make a Lisp string from the result of decoding; and then we invoke the filter with that Lisp string. (If the filter is nil, we invoke internal-default-process-filter instead, but it still gets the text as a string.) Which part(s) of this will be avoided under your proposal? > > Btw, unlike what I originally implied, the default filter also > > receives a Lisp string, so the question why by default reading dd > > output is so much faster than when you define a non-default filter > > function still stands. > > Oh! That is curious indeed. Are the Lisp_Object strings somehow > ... special here when they never leave C land? No, I don't think so. > The speed differential is completely repeatable... hm... Is the > only difference that gc isn't given a chance to run in the > non-filter case? You could test that hypothesis by setting gc-cons-threshold to a very high value. Bottom line: I think we must understand better what takes the time in your last test case, before we discuss solutions. I'd start by profiling that with "M-x profiler-start".
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.
Received: (at 32729) by debbugs.gnu.org; 14 Oct 2019 08:54:32 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 04:54:32 2019
Received: from localhost ([127.0.0.1]:38118 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1iJw7b-0006wa-WA
for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 04:54:32 -0400
Received: from [80.91.231.51] (port=54068 helo=quimby.gnus.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <larsi@HIDDEN>)
id 1iJw7a-0006wM-H4; Mon, 14 Oct 2019 04:54:30 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.89) (envelope-from <larsi@HIDDEN>)
id 1iJw7V-00043Q-JR; Mon, 14 Oct 2019 10:54:28 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
<871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
<87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
<87k198bkrf.fsf@HIDDEN> <834l0clbzt.fsf@HIDDEN>
Date: Mon, 14 Oct 2019 10:54:25 +0200
In-Reply-To: <834l0clbzt.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 13 Oct
2019 21:46:30 +0300")
Message-ID: <87r23fiu66.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
@@CONTACT_ADDRESS@@ for details.
Content preview: Actually, my benchmarking is somewhat wrong. start-process
with a filter, but discard output: (let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*")) (benchmark-run 1 (let ((proc
(start-process "dd" (get-buffer-create " *zeroes*") "dd" "if=/dev/zero"
"bs=4096" "co [...]
Content analysis details: (-2.9 points, 5.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
X-Spam-Score: 1.3 (+)
X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
the administrator of that system for details.
Content preview: Actually, my benchmarking is somewhat wrong. start-process
with a filter, but discard output: (let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*")) (benchmark-run 1 (let ((proc
(start-process "dd" (get-buffer-create " *zeroes*") "dd" "if=/dev/zero"
"bs=4096" "co [...]
Content analysis details: (1.3 points, 10.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
0.0 SPF_NONE SPF: sender does not publish an SPF Record
0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record
0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was
blocked. See
http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block
for more information. [URIs: ingebrigtsen.no]
1.3 RDNS_NONE Delivered to internal network by a host with no rDNS
X-Debbugs-Envelope-To: 32729
Cc: layer@HIDDEN, benjamin.benninghofen@HIDDEN, 32729 <at> debbugs.gnu.org,
32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.3 (/)
Actually, my benchmarking is somewhat wrong.
start-process with a filter, but discard output:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=/dev/zero" "bs=4096" "count=250000")))
(set-process-filter proc (lambda (proc string)))
(while (and (process-live-p proc)
(accept-process-output proc 1))))))
=> (18.828236636 59 13.315468088000017)
filter, but insert the output:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=/dev/zero" "bs=4096" "count=250000")))
(set-process-filter proc (lambda (proc string)
(with-current-buffer (get-buffer " *zeroes*")
(goto-char (point-max))
(insert string))))
(while (and (process-live-p proc)
(accept-process-output proc 1))))))
=> (21.120281346 59 13.250166416000013)
With the default filter:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=/dev/zero" "bs=4096" "count=250000")))
(while (and (process-live-p proc)
(accept-process-output proc 1))))))
=> (34.046986424 116 26.025843717999976)
(!)
So the default filter is really slow?
Anyway, compare with call-process:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=4096" "count=250000")))
=> (1.694743653 0 0.0)
So what makes start-process 10x slower than call-process? If it is all
the string creation before calling the filters, default or not, then my
point stands, but this obviously requires a more in-depth dive into
process.c.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 14 Oct 2019 08:36:19 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 04:36:19 2019 Received: from localhost ([127.0.0.1]:38102 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJvpz-0006WP-33 for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 04:36:19 -0400 Received: from [80.91.231.51] (port=53630 helo=quimby.gnus.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <larsi@HIDDEN>) id 1iJvpx-0006WB-AF; Mon, 14 Oct 2019 04:36:17 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <larsi@HIDDEN>) id 1iJvps-0003rW-FV; Mon, 14 Oct 2019 10:36:15 +0200 From: Lars Ingebrigtsen <larsi@HIDDEN> To: Eli Zaretskii <eliz@HIDDEN> Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87sgnwbl8w.fsf@HIDDEN> <83blujkaf8.fsf@HIDDEN> Date: Mon, 14 Oct 2019 10:36:11 +0200 In-Reply-To: <83blujkaf8.fsf@HIDDEN> (Eli Zaretskii's message of "Mon, 14 Oct 2019 11:18:03 +0300") Message-ID: <87v9sriv0k.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eli Zaretskii <eliz@HIDDEN> writes: > I don't understand what would trigger these callbacks, and how do you > specify the region in advance, without knowing what will be inserted. accept_process_output inserts the data into the buffer and then calls the callback with the region in question. Well, read_and_dispose_of_process_output, I guess... Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 1.3 (+) X-Spam-Report: Spam detection software, running on the system "debbugs.gnu.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see the administrator of that system for details. Content preview: Eli Zaretskii <eliz@HIDDEN> writes: > I don't understand what would trigger these callbacks, and how do you > specify the region in advance, without knowing what will be inserted. accept_process_output inserts the data into the buffer and then calls the callback with the region in question. Well, read_and_dispose_of_process_output, I guess... Content analysis details: (1.3 points, 10.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- 0.0 URIBL_BLOCKED ADMINISTRATOR NOTICE: The query to URIBL was blocked. See http://wiki.apache.org/spamassassin/DnsBlocklists#dnsbl-block for more information. [URIs: gnu.org] 0.0 SPF_NONE SPF: sender does not publish an SPF Record 0.0 SPF_HELO_NONE SPF: HELO does not publish an SPF Record 1.3 RDNS_NONE Delivered to internal network by a host with no rDNS X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: 0.3 (/) Eli Zaretskii <eliz@HIDDEN> writes: > I don't understand what would trigger these callbacks, and how do you > specify the region in advance, without knowing what will be inserted. accept_process_output inserts the data into the buffer and then calls the callback with the region in question. Well, read_and_dispose_of_process_output, I guess... > Without understanding this, I don't think I see the utility, and most > important: why this would be faster. It would avoid creating (and garbaging) the strings. > Btw, unlike what I originally implied, the default filter also > receives a Lisp string, so the question why by default reading dd > output is so much faster than when you define a non-default filter > function still stands. Oh! That is curious indeed. Are the Lisp_Object strings somehow ... special here when they never leave C land? The speed differential is completely repeatable... hm... Is the only difference that gc isn't given a chance to run in the non-filter case? Even if you subtract the gc time, that doesn't explain the difference fully. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 14 Oct 2019 08:18:19 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Mon Oct 14 04:18:19 2019 Received: from localhost ([127.0.0.1]:38059 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJvYZ-0003yG-EV for submit <at> debbugs.gnu.org; Mon, 14 Oct 2019 04:18:19 -0400 Received: from eggs.gnu.org ([209.51.188.92]:43228) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iJvYW-0003xw-Rc; Mon, 14 Oct 2019 04:18:17 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:48753) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iJvYR-0002uA-Bd; Mon, 14 Oct 2019 04:18:11 -0400 Received: from [176.228.60.248] (port=2756 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iJvYQ-0001me-8H; Mon, 14 Oct 2019 04:18:10 -0400 Date: Mon, 14 Oct 2019 11:18:03 +0300 Message-Id: <83blujkaf8.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Lars Ingebrigtsen <larsi@HIDDEN> In-reply-to: <87sgnwbl8w.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sun, 13 Oct 2019 19:36:47 +0200) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87sgnwbl8w.fsf@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen <larsi@HIDDEN> > Cc: benjamin.benninghofen@HIDDEN, layer@HIDDEN, > 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org > Date: Sun, 13 Oct 2019 19:36:47 +0200 > > So this is the design I want to do: > > process-add-callback PROCESS FUNCTION > process-remove-callback PROCESS FUNCTION > > FUNCTION takes three parameters: The PROCESS and the start/end of the > region inserted. Perhaps it would make sense to do something with the > return values -- if the function returns non-nil, then further callbacks > are inhibited? I don't understand what would trigger these callbacks, and how do you specify the region in advance, without knowing what will be inserted. Without understanding this, I don't think I see the utility, and most important: why this would be faster. > > However, I would begin by measuring the effect of this resizing on the > > time it takes to receive large amounts of data. Maybe other factors > > make this part negligible. > > Sure. My simple dd test (without a filter) surprised me by being as > fast as it was, so Emacs was able to grow that buffer quicker than I > expected. But it's also a pretty simple test case -- I can try to see > what happens if I call enlarge_buffer_text to 1GB first and see what the > effects are. Btw, unlike what I originally implied, the default filter also receives a Lisp string, so the question why by default reading dd output is so much faster than when you define a non-default filter function still stands.
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 13 Oct 2019 18:46:54 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 14:46:54 2019 Received: from localhost ([127.0.0.1]:36968 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJitH-0006kS-Sh for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 14:46:52 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47624) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iJitD-0006dx-P7; Sun, 13 Oct 2019 14:46:48 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34745) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iJit8-0006mD-Ep; Sun, 13 Oct 2019 14:46:42 -0400 Received: from [176.228.60.248] (port=1042 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iJit2-0002y3-Ct; Sun, 13 Oct 2019 14:46:41 -0400 Date: Sun, 13 Oct 2019 21:46:30 +0300 Message-Id: <834l0clbzt.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Lars Ingebrigtsen <larsi@HIDDEN> In-reply-to: <87k198bkrf.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sun, 13 Oct 2019 19:47:16 +0200) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> <87k198bkrf.fsf@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen <larsi@HIDDEN> > Cc: benjamin.benninghofen@HIDDEN, layer@HIDDEN, > 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org > Date: Sun, 13 Oct 2019 19:47:16 +0200 > > >> Nope. Does Emacs need to do a lot of recomputing when going to > >> multibyte buffers? > > > > Of course: we try to display the multibyte text as characters, search > > for fonts, invoke bidi reordering, etc. > > I tried: > > (benchmark-run > 1 > (call-process "dd" nil (with-current-buffer > (get-buffer-create " *zeroes*") > (set-buffer-multibyte nil) > (current-buffer)) > nil "if=/dev/zero" "bs=4096" "count=250000")) > > and then jumped to the buffer, and Emacs hung anyway. (That is, I > killed Emacs after a minute, so I don't know whether it would have > recovered after a while...) Do that, _and_ move point to the beginning of the buffer.
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 13 Oct 2019 18:45:05 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 14:45:04 2019 Received: from localhost ([127.0.0.1]:36961 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJirY-00059E-5a for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 14:45:04 -0400 Received: from eggs.gnu.org ([209.51.188.92]:47440) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iJirW-00056n-Nq; Sun, 13 Oct 2019 14:45:03 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:34717) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iJirR-0004vx-1r; Sun, 13 Oct 2019 14:44:57 -0400 Received: from [176.228.60.248] (port=4907 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iJirP-0001qM-AL; Sun, 13 Oct 2019 14:44:55 -0400 Date: Sun, 13 Oct 2019 21:44:48 +0300 Message-Id: <835zkslc2n.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Lars Ingebrigtsen <larsi@HIDDEN> In-reply-to: <87wod8blsl.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sun, 13 Oct 2019 19:24:58 +0200) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN> <87wod8blsl.fsf@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: psainty@HIDDEN, layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen <larsi@HIDDEN> > Date: Sun, 13 Oct 2019 19:24:58 +0200 > Cc: Kevin Layer <layer@HIDDEN>, "Benninghofen, > Benjamin Dr." <benjamin.benninghofen@HIDDEN>, 32729 <at> debbugs.gnu.org, > 32728 <at> debbugs.gnu.org > > It is unfortunate that stuff like this makes Emacs hang, though. It'd > be nice if Emacs had a "OK, we just give up" mode if the buffer is too > intractable, where "too intractable" may be, for instance, if it finds a > line that's longer than a few megabytes, perhaps? > > I don't know what "just give up" would entail, though. Just put point > at the start of the buffer and refuse to scroll or do anything? Just > about anything would be better than the current situation where you have > to kill Emacs if you're playing with (some) binary files and try to > display the buffer. The problem is exactly to decide what to do when you "give up" in a way that will still get you a functional editor that can display something reasonable.
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.
Received: (at 32729) by debbugs.gnu.org; 13 Oct 2019 17:47:22 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 13:47:22 2019
Received: from localhost ([127.0.0.1]:36852 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1iJhxi-0007rX-0r
for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 13:47:22 -0400
Received: from quimby.gnus.org ([80.91.231.51]:34924)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <larsi@HIDDEN>)
id 1iJhxg-0007rL-R8; Sun, 13 Oct 2019 13:47:21 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.89) (envelope-from <larsi@HIDDEN>)
id 1iJhxc-0003Vk-T2; Sun, 13 Oct 2019 19:47:19 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: Eli Zaretskii <eliz@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
<871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN>
<87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN>
Date: Sun, 13 Oct 2019 19:47:16 +0200
In-Reply-To: <83r23hkqr3.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 13 Oct
2019 11:13:04 +0300")
Message-ID: <87k198bkrf.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
@@CONTACT_ADDRESS@@ for details.
Content preview: Eli Zaretskii <eliz@HIDDEN> writes: >> >> (Note: Don't visit
the " *zeroes*" buffer after this, because that will >> >> hang Emacs totally.
I guess the long-line display problem hasn't been >> >> fixed after all?)
>> > >> > It's unrelat [...]
Content analysis details: (-2.9 points, 5.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32729
Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN,
32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
Eli Zaretskii <eliz@HIDDEN> writes:
>> >> (Note: Don't visit the " *zeroes*" buffer after this, because that will
>> >> hang Emacs totally. I guess the long-line display problem hasn't been
>> >> fixed after all?)
>> >
>> > It's unrelated. Did you try to insert that into a unibyte buffer
>> > instead?
>>
>> Nope. Does Emacs need to do a lot of recomputing when going to
>> multibyte buffers?
>
> Of course: we try to display the multibyte text as characters, search
> for fonts, invoke bidi reordering, etc.
I tried:
(benchmark-run
1
(call-process "dd" nil (with-current-buffer
(get-buffer-create " *zeroes*")
(set-buffer-multibyte nil)
(current-buffer))
nil "if=/dev/zero" "bs=4096" "count=250000"))
and then jumped to the buffer, and Emacs hung anyway. (That is, I
killed Emacs after a minute, so I don't know whether it would have
recovered after a while...)
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 13 Oct 2019 17:36:55 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 13:36:55 2019 Received: from localhost ([127.0.0.1]:36841 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJhnb-0007aW-Bg for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 13:36:55 -0400 Received: from quimby.gnus.org ([80.91.231.51]:34758) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <larsi@HIDDEN>) id 1iJhnY-0007aF-Nf; Sun, 13 Oct 2019 13:36:53 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <larsi@HIDDEN>) id 1iJhnU-0003Ol-B9; Sun, 13 Oct 2019 19:36:50 +0200 From: Lars Ingebrigtsen <larsi@HIDDEN> To: Eli Zaretskii <eliz@HIDDEN> Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> <83r23hkqr3.fsf@HIDDEN> Date: Sun, 13 Oct 2019 19:36:47 +0200 In-Reply-To: <83r23hkqr3.fsf@HIDDEN> (Eli Zaretskii's message of "Sun, 13 Oct 2019 11:13:04 +0300") Message-ID: <87sgnwbl8w.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eli Zaretskii <eliz@HIDDEN> writes: >> No sentinel is called, because the process status doesn't change, >> typically. All the relevant network protocols do not close after having >> "done something" (IMAP, HTTP, etc), but instead use i [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Eli Zaretskii <eliz@HIDDEN> writes: >> No sentinel is called, because the process status doesn't change, >> typically. All the relevant network protocols do not close after having >> "done something" (IMAP, HTTP, etc), but instead use in-protocol markers >> to say that an operation is done. So a filter has to be used. > > OK, so not in sentinel, but in a timer or somesuch. A timer wastes resources by triggering work when there's nothing to do, and conversely, makes network connections sluggish by not having the code happen immediately when there's data ready. After thinking about this a bit more, I'm reminded of the problems I had in this area when doing the NSM, because I really wanted to put a filter on (some of) the processes, but I couldn't, because we only allow one filter per process. So this is the design I want to do: process-add-callback PROCESS FUNCTION process-remove-callback PROCESS FUNCTION FUNCTION takes three parameters: The PROCESS and the start/end of the region inserted. Perhaps it would make sense to do something with the return values -- if the function returns non-nil, then further callbacks are inhibited? Anyway, with this, the current filters can trivially be implemented as a callback instead -- the filters would just be wrapped in a function that does (funcall filter-function (buffer-substring start end)) so that we don't need duplicated code for these mechanisms on the C level. So set-process-filter and related moves to the Lisp level... > We could provide a Lisp interface for such cases. The C > implementation is in enlarge_buffer_text (but we need to remember the > decoding of human-readable text, when applicable). > > However, I would begin by measuring the effect of this resizing on the > time it takes to receive large amounts of data. Maybe other factors > make this part negligible. Sure. My simple dd test (without a filter) surprised me by being as fast as it was, so Emacs was able to grow that buffer quicker than I expected. But it's also a pretty simple test case -- I can try to see what happens if I call enlarge_buffer_text to 1GB first and see what the effects are. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 13 Oct 2019 17:25:07 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 13:25:07 2019 Received: from localhost ([127.0.0.1]:36826 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJhcB-0005Dh-Cf for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 13:25:07 -0400 Received: from quimby.gnus.org ([80.91.231.51]:34510) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <larsi@HIDDEN>) id 1iJhc7-0005DO-0t; Sun, 13 Oct 2019 13:25:05 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <larsi@HIDDEN>) id 1iJhc2-0003JB-Mq; Sun, 13 Oct 2019 19:25:01 +0200 From: Lars Ingebrigtsen <larsi@HIDDEN> To: Phil Sainty <psainty@HIDDEN> Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN> Date: Sun, 13 Oct 2019 19:24:58 +0200 In-Reply-To: <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN> (Phil Sainty's message of "Sun, 13 Oct 2019 23:49:41 +1300") Message-ID: <87wod8blsl.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Phil Sainty <psainty@HIDDEN> writes: > On 12/10/19 4:57 PM, Lars Ingebrigtsen wrote: >> (Note: Don't visit the " *zeroes*" buffer after this, because that >> will hang Emacs totally. I guess the long-line display problem >> hasn't been f [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 32729 Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Phil Sainty <psainty@HIDDEN> writes: > On 12/10/19 4:57 PM, Lars Ingebrigtsen wrote: >> (Note: Don't visit the " *zeroes*" buffer after this, because that >> will hang Emacs totally. I guess the long-line display problem >> hasn't been fixed after all?) > > I imagine you're thinking of so-long.el here, in which case there may > be a misconception. Yes, I hadn't followed the development closely, but just registered that ... things ... had gotten better. :-) > The biggest problem in this case (by far) is that point has been left > at the end of the line following the insertion, and in order for the > redisplay to *display* the end of the line, it's having to process a > gigabyte of data. My largest test case for so-long.el was a couple of > orders of magnitude smaller than this, and Emacs wasn't remotely happy > showing the end of that line, let alone this one. Ah, I see. Thanks for the explanation. It is unfortunate that stuff like this makes Emacs hang, though. It'd be nice if Emacs had a "OK, we just give up" mode if the buffer is too intractable, where "too intractable" may be, for instance, if it finds a line that's longer than a few megabytes, perhaps? I don't know what "just give up" would entail, though. Just put point at the start of the buffer and refuse to scroll or do anything? Just about anything would be better than the current situation where you have to kill Emacs if you're playing with (some) binary files and try to display the buffer. -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 13 Oct 2019 10:49:49 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 06:49:49 2019 Received: from localhost ([127.0.0.1]:35249 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJbRc-0007su-JX for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 06:49:48 -0400 Received: from smtp-2.orcon.net.nz ([60.234.4.43]:33701) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <psainty@HIDDEN>) id 1iJbRZ-0007sg-Ag; Sun, 13 Oct 2019 06:49:46 -0400 Received: from [116.251.203.173] (port=2746 helo=[192.168.20.103]) by smtp-2.orcon.net.nz with esmtpa (Exim 4.90_1) (envelope-from <psainty@HIDDEN>) id 1iJbRV-0003Bn-5Z; Sun, 13 Oct 2019 23:49:41 +1300 Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs To: Lars Ingebrigtsen <larsi@HIDDEN> References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> From: Phil Sainty <psainty@HIDDEN> Message-ID: <5b3b2f98-20e6-596d-2c02-0821def8ae40@HIDDEN> Date: Sun, 13 Oct 2019 23:49:41 +1300 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.9.0 MIME-Version: 1.0 In-Reply-To: <871rviobu2.fsf@HIDDEN> Content-Type: text/plain; charset=utf-8 Content-Language: en-GB Content-Transfer-Encoding: 7bit X-GeoIP: NZ X-Spam_score: -2.9 X-Spam_score_int: -28 X-Spam_bar: -- X-Spam-Score: -0.7 (/) X-Debbugs-Envelope-To: 32729 Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.7 (-) On 12/10/19 4:57 PM, Lars Ingebrigtsen wrote: > (Note: Don't visit the " *zeroes*" buffer after this, because that > will hang Emacs totally. I guess the long-line display problem > hasn't been fixed after all?) I imagine you're thinking of so-long.el here, in which case there may be a misconception. so-long helps tremendously in certain situations, but it doesn't (and cannot) eliminate all sources of potential slowness in the face of extremely long lines -- ultimately the C code still needs time to process the lines being displayed, and so-long.el can't make that code work any faster. The biggest problem in this case (by far) is that point has been left at the end of the line following the insertion, and in order for the redisplay to *display* the end of the line, it's having to process a gigabyte of data. My largest test case for so-long.el was a couple of orders of magnitude smaller than this, and Emacs wasn't remotely happy showing the end of that line, let alone this one. (What so-long is really good at is preventing situations whereby merely visiting a file -- with point still at the beginning of the buffer -- caused Emacs to hang badly; which can easily happen when various modes and options are active. It can work wonders in those cases. If you try to display the end of a really gargantuan line of data like this, though, you're simply at the mercy of other factors.) If you added a (goto-char (point-min)) before switching to the *zeroes* buffer then, given that there's not much else happening in that buffer which would create issues, Emacs shouldn't hang when you switch to it. The further you move into that line, though, the worse the performance is going to get (so the key take-away here would be to avoid doing that when dealing with this kind of data, if at all possible). Also note that `so-long' doesn't actually trigger automatically if you simply create a buffer and throw a ton of data into it. The automatic behaviour is tied to `set-auto-mode' and intended to deal with users visiting files. Auto-detecting the sudden insertion of large lines into an existing buffer is a different kettle of fish, and the current library does not attempt to cater for it. So if you wanted to trigger it in this example you would need to call it explicitly. -Phil
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 13 Oct 2019 08:13:22 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 04:13:21 2019 Received: from localhost ([127.0.0.1]:34886 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJZ0D-0003uG-IL for submit <at> debbugs.gnu.org; Sun, 13 Oct 2019 04:13:21 -0400 Received: from eggs.gnu.org ([209.51.188.92]:40287) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iJZ0B-0003tv-JR; Sun, 13 Oct 2019 04:13:20 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:55395) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iJZ05-00040m-RG; Sun, 13 Oct 2019 04:13:13 -0400 Received: from [176.228.60.248] (port=1844 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iJZ04-0004RJ-Vm; Sun, 13 Oct 2019 04:13:13 -0400 Date: Sun, 13 Oct 2019 11:13:04 +0300 Message-Id: <83r23hkqr3.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Lars Ingebrigtsen <larsi@HIDDEN> In-reply-to: <87y2xplufp.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sat, 12 Oct 2019 19:55:54 +0200) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> <87y2xplufp.fsf@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen <larsi@HIDDEN> > Cc: benjamin.benninghofen@HIDDEN, layer@HIDDEN, > 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org > Date: Sat, 12 Oct 2019 19:55:54 +0200 > > >> (Note: Don't visit the " *zeroes*" buffer after this, because that will > >> hang Emacs totally. I guess the long-line display problem hasn't been > >> fixed after all?) > > > > It's unrelated. Did you try to insert that into a unibyte buffer > > instead? > > Nope. Does Emacs need to do a lot of recomputing when going to > multibyte buffers? Of course: we try to display the multibyte text as characters, search for fonts, invoke bidi reordering, etc. > >> And it's a real world problem: When reading data from any network > >> source, you have to use filters because the protocol is usually based on > >> parsing the output to find out when it's over, so you can't use > >> sentinels. > > > > Why can't you use the default filter, and in the sentinel work on the > > buffer with the complete response? > > No sentinel is called, because the process status doesn't change, > typically. All the relevant network protocols do not close after having > "done something" (IMAP, HTTP, etc), but instead use in-protocol markers > to say that an operation is done. So a filter has to be used. OK, so not in sentinel, but in a timer or somesuch. > > Please also note that, GC aside, inserting stuff of size that is > > unknown in advance is not free, either: we need to enlarge the buffer > > text each time more stuff arrives. > > Yes, I was wondering about that -- how slow is resizing buffers, really? > Does Emacs rely on OS-level realloc that doesn't necessitate actually > copying all that data all the time? Not necessarily realloc, sometimes mmap or similar. And yes, this requires copying the data when the OS cannot grow the previous allocation, but instead gives us a memory block at another address. In most systems this copying is under the hood, but it does happen. > Also -- some of the networking operations know in advance how much data > is going to be received. For instance, when using non-chunked HTTP > (quite common when fetching "data", i.e., images, PDFs etc), we get the > content size in a header. Would it make sense to have a way to tell > Emacs about this in advance so that it could pre-size the buffer? We could provide a Lisp interface for such cases. The C implementation is in enlarge_buffer_text (but we need to remember the decoding of human-readable text, when applicable). However, I would begin by measuring the effect of this resizing on the time it takes to receive large amounts of data. Maybe other factors make this part negligible.
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 12 Oct 2019 17:56:03 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Oct 12 13:56:03 2019 Received: from localhost ([127.0.0.1]:34099 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJLcZ-0001t4-9p for submit <at> debbugs.gnu.org; Sat, 12 Oct 2019 13:56:03 -0400 Received: from quimby.gnus.org ([80.91.231.51]:36670) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <larsi@HIDDEN>) id 1iJLcV-0001sV-BU; Sat, 12 Oct 2019 13:56:01 -0400 Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie) by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.89) (envelope-from <larsi@HIDDEN>) id 1iJLcQ-0006TB-IJ; Sat, 12 Oct 2019 19:55:57 +0200 From: Lars Ingebrigtsen <larsi@HIDDEN> To: Eli Zaretskii <eliz@HIDDEN> Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> <83imouo1jp.fsf@HIDDEN> Date: Sat, 12 Oct 2019 19:55:54 +0200 In-Reply-To: <83imouo1jp.fsf@HIDDEN> (Eli Zaretskii's message of "Sat, 12 Oct 2019 10:39:22 +0300") Message-ID: <87y2xplufp.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org", has NOT identified this incoming email as spam. The original message has been attached to this so you can view it or label similar future email. If you have any questions, see @@CONTACT_ADDRESS@@ for details. Content preview: Eli Zaretskii <eliz@HIDDEN> writes: >> (benchmark-run 1 (call-process "dd" nil (get-buffer-create " >> *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000")) >> => (4.703641145 0 0.0) >> >> (Note: Don't visit the " *zeroes*" buffer a [...] Content analysis details: (-2.9 points, 5.0 required) pts rule name description ---- ---------------------- -------------------------------------------------- -1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP -1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1% [score: 0.0000] X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) Eli Zaretskii <eliz@HIDDEN> writes: >> (benchmark-run 1 (call-process "dd" nil (get-buffer-create " >> *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000")) >> => (4.703641145 0 0.0) >> >> (Note: Don't visit the " *zeroes*" buffer after this, because that will >> hang Emacs totally. I guess the long-line display problem hasn't been >> fixed after all?) > > It's unrelated. Did you try to insert that into a unibyte buffer > instead? Nope. Does Emacs need to do a lot of recomputing when going to multibyte buffers? >> And it's a real world problem: When reading data from any network >> source, you have to use filters because the protocol is usually based on >> parsing the output to find out when it's over, so you can't use >> sentinels. > > Why can't you use the default filter, and in the sentinel work on the > buffer with the complete response? No sentinel is called, because the process status doesn't change, typically. All the relevant network protocols do not close after having "done something" (IMAP, HTTP, etc), but instead use in-protocol markers to say that an operation is done. So a filter has to be used. > Please also note that, GC aside, inserting stuff of size that is > unknown in advance is not free, either: we need to enlarge the buffer > text each time more stuff arrives. Yes, I was wondering about that -- how slow is resizing buffers, really? Does Emacs rely on OS-level realloc that doesn't necessitate actually copying all that data all the time? Also -- some of the networking operations know in advance how much data is going to be received. For instance, when using non-chunked HTTP (quite common when fetching "data", i.e., images, PDFs etc), we get the content size in a header. Would it make sense to have a way to tell Emacs about this in advance so that it could pre-size the buffer? -- (domestic pets only, the antidote for overdose, milk.) bloggy blog: http://lars.ingebrigtsen.no
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Received: (at 32729) by debbugs.gnu.org; 12 Oct 2019 07:39:42 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sat Oct 12 03:39:42 2019 Received: from localhost ([127.0.0.1]:60677 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1iJC06-0001Wn-An for submit <at> debbugs.gnu.org; Sat, 12 Oct 2019 03:39:42 -0400 Received: from eggs.gnu.org ([209.51.188.92]:41731) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <eliz@HIDDEN>) id 1iJC03-0001WS-A7; Sat, 12 Oct 2019 03:39:40 -0400 Received: from fencepost.gnu.org ([2001:470:142:3::e]:33890) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <eliz@HIDDEN>) id 1iJBzx-0007ld-Ko; Sat, 12 Oct 2019 03:39:33 -0400 Received: from [176.228.60.248] (port=2912 helo=home-c4e4a596f7) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_256_CBC_SHA1:256) (Exim 4.82) (envelope-from <eliz@HIDDEN>) id 1iJBzx-0001eq-2w; Sat, 12 Oct 2019 03:39:33 -0400 Date: Sat, 12 Oct 2019 10:39:22 +0300 Message-Id: <83imouo1jp.fsf@HIDDEN> From: Eli Zaretskii <eliz@HIDDEN> To: Lars Ingebrigtsen <larsi@HIDDEN> In-reply-to: <871rviobu2.fsf@HIDDEN> (message from Lars Ingebrigtsen on Sat, 12 Oct 2019 05:57:09 +0200) Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN> <871rviobu2.fsf@HIDDEN> X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Spam-Score: -2.3 (--) X-Debbugs-Envelope-To: 32729 Cc: layer@HIDDEN, 32729 <at> debbugs.gnu.org, benjamin.benninghofen@HIDDEN, 32728 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -3.3 (---) > From: Lars Ingebrigtsen <larsi@HIDDEN> > Date: Sat, 12 Oct 2019 05:57:09 +0200 > Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org > > (benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000")) > => (4.703641145 0 0.0) > > (Note: Don't visit the " *zeroes*" buffer after this, because that will > hang Emacs totally. I guess the long-line display problem hasn't been > fixed after all?) It's unrelated. Did you try to insert that into a unibyte buffer instead? > So it would seem that the Emacs filter method is just unnecessarily > slow, which I've long suspected. Creating the strings before calling > the filter is probably what's taking quite a bit of this time, but the > rest is taken up by garbage collecting as it spends 13 of these 20 > seconds doing that. > > And it's a real world problem: When reading data from any network > source, you have to use filters because the protocol is usually based on > parsing the output to find out when it's over, so you can't use > sentinels. Why can't you use the default filter, and in the sentinel work on the buffer with the complete response? Please also note that, GC aside, inserting stuff of size that is unknown in advance is not free, either: we need to enlarge the buffer text each time more stuff arrives. > So for Emacs 28 I want to explore adding a new type of filter to > processes: One that doesn't take a string argument, but which just > inserts the data into the buffer, and then calls the filter with the > region positions of what was inserted, which is just as powerful, but > should allow streams to be 10x more efficient. Doesn't the default filter already do that?
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.
Received: (at 32729) by debbugs.gnu.org; 12 Oct 2019 03:57:20 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Fri Oct 11 23:57:20 2019
Received: from localhost ([127.0.0.1]:60553 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1iJ8Wt-0006lT-Je
for submit <at> debbugs.gnu.org; Fri, 11 Oct 2019 23:57:19 -0400
Received: from quimby.gnus.org ([80.91.231.51]:50562)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <larsi@HIDDEN>)
id 1iJ8Wo-0006lB-1F; Fri, 11 Oct 2019 23:57:14 -0400
Received: from cm-84.212.202.86.getinternet.no ([84.212.202.86] helo=marnie)
by quimby.gnus.org with esmtpsa (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.89) (envelope-from <larsi@HIDDEN>)
id 1iJ8Wj-0006U3-Bq; Sat, 12 Oct 2019 05:57:11 +0200
From: Lars Ingebrigtsen <larsi@HIDDEN>
To: "Benninghofen\, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
Subject: Re: bug#32729: Xemacs 23 times as fast as GNU Emacs
References: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
Date: Sat, 12 Oct 2019 05:57:09 +0200
Message-ID: <871rviobu2.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/27.0.50 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Report: Spam detection software, running on the system "quimby.gnus.org",
has NOT identified this incoming email as spam. The original
message has been attached to this so you can view it or label
similar future email. If you have any questions, see
@@CONTACT_ADDRESS@@ for details.
Content preview: The example is a bit convoluted,
but it's an interesting question
-- is Emacs' handling of process output fast or not? As a baseline, let's
read a GB of zeroes and output to /dev/null: larsi@marnie:~/src/emacs/trunk$
time dd if=/dev/zero bs=1000 count=1000000 > /dev/null 1000000+0 records
in 1000000+0 records out 1000000000 bytes (1.0 GB, 954 MiB) copied, 2.04672
s, 489 MB/s
Content analysis details: (-2.9 points, 5.0 required)
pts rule name description
---- ---------------------- --------------------------------------------------
-1.0 ALL_TRUSTED Passed through trusted hosts only via SMTP
-1.9 BAYES_00 BODY: Bayes spam probability is 0 to 1%
[score: 0.0000]
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 32729
Cc: Kevin Layer <layer@HIDDEN>, 32729 <at> debbugs.gnu.org, 32728 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
The example is a bit convoluted, but it's an interesting question -- is
Emacs' handling of process output fast or not?
As a baseline, let's read a GB of zeroes and output to /dev/null:
larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 > /dev/null
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.04672 s, 489 MB/s
real 0m2.064s
user 0m0.173s
sys 0m1.684s
(benchmark-run 1 (call-process "dd" nil nil nil "if=/dev/zero" "bs=1000" "count=1000000"))
=> (0.665899839 0 0.0)
So that's better in Emacs than in the shell, but we're just setting
stdout to /dev/zero here, so we're not actually seeing the data at all.
But let's insert the data somewhere:
(benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000"))
=> (4.703641145 0 0.0)
(Note: Don't visit the " *zeroes*" buffer after this, because that will
hang Emacs totally. I guess the long-line display problem hasn't been
fixed after all?)
4.7s isn't horrible, but it's not good, either. But most of that time
is taken up in coding system conversion, so:
(let ((coding-system-for-read 'binary))
(benchmark-run 1 (call-process "dd" nil (get-buffer-create " *zeroes*") nil "if=/dev/zero" "bs=1000" "count=1000000")))
=> (1.750549617 0 0.0)
Which is faster than writing to a file:
larsi@marnie:~/src/emacs/trunk$ time dd if=/dev/zero bs=1000 count=1000000 of=/tmp/foo
1000000+0 records in
1000000+0 records out
1000000000 bytes (1.0 GB, 954 MiB) copied, 2.21987 s, 450 MB/s
real 0m2.325s
user 0m0.168s
sys 0m1.957s
So that's OK. But what happens when we add a process filter?
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=/dev/zero" "bs=1000" "count=1000000")))
(set-process-filter proc (lambda (proc string)
))
(while (and (process-live-p proc)
(accept-process-output proc 0.01))))))
=> (16.878995199 993 12.469541476)
That's slow, and we're just discarding the data. If we output the data,
it's even slower, but not a surprising amount:
(let ((coding-system-for-read 'binary))
(kill-buffer (get-buffer-create " *zeroes*"))
(benchmark-run
1
(let ((proc (start-process "dd" (get-buffer-create " *zeroes*") "dd"
"if=/dev/zero" "bs=1000" "count=1000000")))
(set-process-filter proc (lambda (proc string)
(with-current-buffer
(get-buffer-create " *zeroes*")
(goto-char (point-max))
(insert string))))
(while (and (process-live-p proc)
(accept-process-output proc 0.01))))))
=> (19.801399562 1000 12.700370797000001)
Byte-compiling the function makes no difference.
So it would seem that the Emacs filter method is just unnecessarily
slow, which I've long suspected. Creating the strings before calling
the filter is probably what's taking quite a bit of this time, but the
rest is taken up by garbage collecting as it spends 13 of these 20
seconds doing that.
And it's a real world problem: When reading data from any network
source, you have to use filters because the protocol is usually based on
parsing the output to find out when it's over, so you can't use
sentinels.
So for Emacs 28 I want to explore adding a new type of filter to
processes: One that doesn't take a string argument, but which just
inserts the data into the buffer, and then calls the filter with the
region positions of what was inserted, which is just as powerful, but
should allow streams to be 10x more efficient.
--
(domestic pets only, the antidote for overdose, milk.)
bloggy blog: http://lars.ingebrigtsen.no
bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.Lars Ingebrigtsen <larsi@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.Glenn Morris <rgm@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at submit) by debbugs.gnu.org; 13 Sep 2018 15:13:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Sep 13 11:13:44 2018
Received: from localhost ([127.0.0.1]:39371 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1g0TJP-0006HS-IF
for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 11:13:43 -0400
Received: from eggs.gnu.org ([208.118.235.92]:39899)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <benjamin.benninghofen@HIDDEN>)
id 1g0Rkc-0003Se-60
for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 09:33:42 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <benjamin.benninghofen@HIDDEN>)
id 1g0RkQ-0007vp-RP
for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 09:33:36 -0400
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level:
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50,HTML_MESSAGE
autolearn=disabled version=3.3.2
Received: from lists.gnu.org ([2001:4830:134:3::11]:53034)
by eggs.gnu.org with esmtps (TLS1.0:RSA_AES_256_CBC_SHA1:32)
(Exim 4.71) (envelope-from <benjamin.benninghofen@HIDDEN>)
id 1g0RkQ-0007vi-NV
for submit <at> debbugs.gnu.org; Thu, 13 Sep 2018 09:33:30 -0400
Received: from eggs.gnu.org ([2001:4830:134:3::10]:60814)
by lists.gnu.org with esmtp (Exim 4.71)
(envelope-from <benjamin.benninghofen@HIDDEN>)
id 1g0RkP-0005Vj-0q
for bug-gnu-emacs@HIDDEN; Thu, 13 Sep 2018 09:33:30 -0400
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <benjamin.benninghofen@HIDDEN>)
id 1g0RkH-0007du-OE
for bug-gnu-emacs@HIDDEN; Thu, 13 Sep 2018 09:33:27 -0400
Received: from mo1.myeers.net ([87.190.7.232]:32795)
by eggs.gnu.org with esmtps (TLS1.0:DHE_RSA_AES_256_CBC_SHA1:32)
(Exim 4.71) (envelope-from <benjamin.benninghofen@HIDDEN>)
id 1g0RkE-0007El-Cx
for bug-gnu-emacs@HIDDEN; Thu, 13 Sep 2018 09:33:19 -0400
X-IronPort-AV: E=Sophos;i="5.53,369,1531778400"; d="scan'208,217";a="39201171"
Received: from unknown (HELO DE0-44HUB-P04.central.mail.corp) ([44.225.67.45])
by de0-03iro-p02-out.myeers.net with ESMTP/TLS/AES256-SHA;
13 Sep 2018 15:33:09 +0200
Received: from esa1e.demail.de.airbusds.corp (10.67.144.33) by
DE0-44HUB-P04.central.mail.corp (44.225.67.47) with Microsoft SMTP Server id
15.0.1365.1; Thu, 13 Sep 2018 15:33:04 +0200
Received: from unknown (HELO CD1-4DDAG02-P02.cdmail.common.airbusds.corp)
([10.67.164.144])
by esa1i.demail.de.airbusds.corp with ESMTP; 13 Sep 2018 15:33:04 +0200
Received: from CD1-4DDAG02-P01.cdmail.common.airbusds.corp (10.67.164.142) by
CD1-4DDAG02-P02.cdmail.common.airbusds.corp (10.67.164.144) with
Microsoft
SMTP Server (TLS) id 15.0.1365.1; Thu, 13 Sep 2018 15:32:47 +0200
Received: from CD1-4DDAG02-P01.cdmail.common.airbusds.corp ([10.67.164.142])
by CD1-4DDAG02-P01.cdmail.common.airbusds.corp ([10.67.164.142]) with mapi id
15.00.1365.000; Thu, 13 Sep 2018 15:32:48 +0200
From: "Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>
To: "bug-gnu-emacs@HIDDEN" <bug-gnu-emacs@HIDDEN>
Subject: Xemacs 23 times as fast as GNU Emacs
Thread-Topic: Xemacs 23 times as fast as GNU Emacs
Thread-Index: AdRLYcBdLvyutlnaQiCCkSAhjOwdQg==
Date: Thu, 13 Sep 2018 13:32:48 +0000
Message-ID: <c3a9871dbc0a42c799d7368e2b6457b2@HIDDEN>
Accept-Language: en-US
Content-Language: en-US
X-MS-Has-Attach:
X-MS-TNEF-Correlator:
x-ms-exchange-transport-fromentityheader: Hosted
Content-Type: multipart/alternative;
boundary="_000_c3a9871dbc0a42c799d7368e2b6457b2CD14DDAG02P01cdmailcomm_"
MIME-Version: 1.0
X-GM-Security: forwarded
X-detected-operating-system: by eggs.gnu.org: Genre and OS details not
recognized.
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 2001:4830:134:3::11
X-Spam-Score: -4.1 (----)
X-Debbugs-Envelope-To: submit
X-Mailman-Approved-At: Thu, 13 Sep 2018 11:13:41 -0400
Cc: Kevin Layer <layer@HIDDEN>
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.1 (-----)
--_000_c3a9871dbc0a42c799d7368e2b6457b2CD14DDAG02P01cdmailcomm_
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
I did send an email with the same subject earlier, but its attachment was p=
robably lost.
Now I inserted the contents of the attachments in the email text.
The versions of Xemacs and GNU Emacs are those that come with RHEL 7.5.
To reproduce the benchmark:
Evaluate this in Emacs-Lisp
(defun demo ()
(interactive)
(let ((buffer (get-buffer-create "*demo*"))
proc)
(switch-to-buffer buffer)
(erase-buffer)
(setq proc (start-process "demo" buffer "~/prog.sh"))
(set-process-filter proc 'demo-process-filter)))
(defun demo-process-filter (process output)
(let* ((xmarker (process-mark process))
(marker (if (marker-position xmarker)
xmarker
(set-marker (make-marker) 0 buffer)))
(marker-point (marker-position marker)))
(goto-char marker-point)
(insert output)
(set-marker marker (point))))
Place a file "prog.sh" with the contents below in the home directory:
#! /bin/bash
start=3D$(date +%s)
echo sleeping 5 to wait for process filter to get installed...
sleep 5
while read line; do
echo $line
done <<< "$(cat input.txt)"
end=3D$(date +%s)
echo duration: $(( end - start )) seconds
Furthermore a large text file "input.txt" is needed in the home directory.=
The file should have 1000000 lines, each line longer than 80 characters.
The file I used was created with the following ANSI-COMMON-LISP function:
(defun print-nums (&key (first 1) (last 1000000))
(check-type first fixnum)
(check-type last fixnum)
(loop for k of-type fixnum from first to last do (format t "~%~B ^3 =3D ~=
B" k (expt k 3)))
t)
Alternatively the "input.txt" file can be created as follows:
#! /bin/bash
last=3D1000000
function base2 {
echo "obase=3D2;$1" | bc
}
for k in $(seq 1 $last); do
x=3D$(( k * k * k ))
echo $(base2 $k) '^3 =3D' $(base2 $x)
done
generate the input like this:
$ ./gen.sh > input.txt
The benchmark is executed by
(M-x) demo
At the end the time is printed and I received the following results:
Xemacs : 51 seconds
GNU Emacs : 1205 seconds
So the Xemacs is more than 23 times as fast as the GNU Emacs.
---
Benjamin Benninghofen
The information in this e-mail is confidential. The contents may not be dis=
closed or used by anyone other than the addressee. Access to this e-mail by=
anyone else is unauthorised.
If you are not the intended recipient, please notify Airbus immediately and=
delete this e-mail.
Airbus cannot accept any responsibility for the accuracy or completeness of=
this e-mail as it has been sent over public networks. If you have any conc=
erns over the content of this message or its Accuracy or Integrity, please =
contact Airbus immediately.
All outgoing e-mails from Airbus are checked using regularly updated virus =
scanning software but you should take whatever measures you deem to be appr=
opriate to ensure that this message and any attachments are virus free.
--_000_c3a9871dbc0a42c799d7368e2b6457b2CD14DDAG02P01cdmailcomm_
Content-Type: text/html; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: quoted-printable
<html>
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dus-ascii">
<meta name=3D"Generator" content=3D"Microsoft Exchange Server">
<!-- converted from rtf -->
<style><!-- .EmailQuote { margin-left: 1pt; padding-left: 4pt; border-left:=
#800000 2px solid; } --></style>
</head>
<body>
<font face=3D"Calibri" size=3D"2"><span style=3D"font-size:11pt;">
<div>I did send an email with the same subject earlier, but its attachment =
was probably lost.</div>
<div> </div>
<div>Now I inserted the contents of the attachments in the email text.</div>
<div> </div>
<div> </div>
<div>The versions of Xemacs and GNU Emacs are those that come with RHEL 7.5=
.</div>
<div> </div>
<div>To reproduce the benchmark:</div>
<div> </div>
<div>Evaluate this in Emacs-Lisp</div>
<div> </div>
<div> </div>
<div>(defun demo ()</div>
<div> (interactive)</div>
<div> (let ((buffer (get-buffer-create "*demo*"=
))</div>
<div> proc)</div>
<div> (switch-to-buffer buffer)</div>
<div> (erase-buffer)</div>
<div> (setq proc (start-process "demo&qu=
ot; buffer "~/prog.sh"))</div>
<div> (set-process-filter proc 'demo-process-=
filter)))</div>
<div> </div>
<div>(defun demo-process-filter (process output)</div>
<div> (let* ((xmarker (process-mark process))</div>
<div> (marker (if (m=
arker-position xmarker)</div>
<div> &nbs=
p; xmarker</div>
<div> &nbs=
p; (set-marker (make-marker) 0 b=
uffer)))</div>
<div> (marker-point =
(marker-position marker)))</div>
<div> (goto-char marker-point)</div>
<div> (insert output)</div>
<div> (set-marker marker (point))))</div>
<div> </div>
<div>Place a file "prog.sh" with the contents below in the home d=
irectory:</div>
<div> </div>
<div>#! /bin/bash</div>
<div> start=3D$(date +%s)</div>
<div> echo sleeping 5 to wait for process filter to get installed...<=
/div>
<div> sleep 5</div>
<div> while read line; do</div>
<div> echo $line</div>
<div> done <<< "$(cat input.txt)"</div>
<div> end=3D$(date +%s)</div>
<div> echo duration: $(( end - start )) seconds</div>
<div> </div>
<div> </div>
<div> </div>
<div> </div>
<div>Furthermore a large text file "input.txt" is needed in=
the home directory. The file should have 1000000 lines, each line longer t=
han 80 characters. </div>
<div> </div>
<div> </div>
<div>The file I used was created with the following ANSI-COMMON-LISP functi=
on:</div>
<div> </div>
<div>(defun print-nums (&key (first 1) (last 1000000))</div>
<div> (check-type first fixnum)</div>
<div> (check-type last fixnum)</div>
<div> (loop for k of-type fixnum from first to last do (format t &quo=
t;~%~B ^3 =3D ~B" k (expt k 3)))</div>
<div> t)</div>
<div> </div>
<div>Alternatively the "input.txt" file can be created as follows=
:</div>
<div> </div>
<div>#! /bin/bash</div>
<div> last=3D1000000</div>
<div> function base2 {</div>
<div> echo "obase=3D2;$1" | bc</div>
<div> }</div>
<div> for k in $(seq 1 $last); do</div>
<div> x=3D$(( k * k * k ))</div>
<div> echo $(base2 $k) '^3 =3D' $(base2 $x)</=
div>
<div> done</div>
<div> </div>
<div>generate the input like this:</div>
<div> </div>
<div> $ ./gen.sh > input.txt</div>
<div> </div>
<div> </div>
<div> </div>
<div>The benchmark is executed by </div>
<div>(M-x) demo</div>
<div> </div>
<div>At the end the time is printed and I received the following results:</=
div>
<div> </div>
<div>Xemacs : 51 seconds</div>
<div>GNU Emacs : 1205 seconds</div>
<div> </div>
<div>So the Xemacs is more than 23 times as fast as the GNU Emacs.</div>
<div> </div>
<div>---</div>
<div>Benjamin Benninghofen</div>
<div> </div>
<div> </div>
</span></font>
<font style=3D"font-size: 9px;">The information in this e-mail is confident=
ial. The contents may not be disclosed or used by anyone other than the add=
ressee. Access to this e-mail by anyone else is unauthorised.<br>If you are=
not the intended recipient, please notify Airbus immediately and delete th=
is e-mail.<br>Airbus cannot accept any responsibility for the accuracy or c=
ompleteness of this e-mail as it has been sent over public networks. If you=
have any concerns over the content of this message or its Accuracy or Inte=
grity, please contact Airbus immediately.<br>All outgoing e-mails from Airb=
us are checked using regularly updated virus scanning software but you shou=
ld take whatever measures you deem to be appropriate to ensure that this me=
ssage and any attachments are virus free.</font></body>
</html>
--_000_c3a9871dbc0a42c799d7368e2b6457b2CD14DDAG02P01cdmailcomm_--
"Benninghofen, Benjamin Dr." <benjamin.benninghofen@HIDDEN>:bug-gnu-emacs@HIDDEN.
Full text available.bug-gnu-emacs@HIDDEN:bug#32729; Package emacs.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.