GNU logs - #74394, boring messages

Message sent to guix-patches@HIDDEN:

Subject: [bug#74394] [PATCH 0/2] Skip slow tests by default and run 'check' in Git pre-push hook.
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Sun, 17 Nov 2024 12:04:01 +0000
Resent-Message-ID: <handler.74394.B.173184503224362 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 74394 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Date: Sun, 17 Nov 2024 21:03:30 +0900
Message-ID: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Received-SPF: pass client-ip=2607:f8b0:4864:20::629;
 envelope-from=maxim.cournoyer@HIDDEN; helo=mail-pl1-x629.google.com
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001,
 SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.3 (-)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.3 (--)

Hello,

This is a simple change that should ensure test suite breakages are detected
as early as possible and avoid tests breaking changes to be pushed.  This is
made possible by skipping a few expensive tests suite, bringing down the total
test time to about 1 minute on a fast machine.

We could call it a "distributed CI" approach ;-).

Note: I initially pursued an Automake or Make-based approach, but it ended up
far from trivial, hitting old issues such as [0] along the way.  This solution
simply puts the skip logic in the tests that must be skipped (a one liner).

To run the complete test suite including the slow tests (as is the case prior
this change):

make check WITH_SLOW_TESTS=1

[0]  https://debbugs.gnu.org/cgi/bugreport.cgi?bug=74387

Maxim Cournoyer (2):
  build: Exclude expensive tests in check target by default.
  etc: Ensure test suite passes in pre-push git hook.

 Makefile.am                | 9 ++++++++-
 etc/git/pre-push           | 1 +
 tests/guix-home.sh         | 5 +++++
 tests/guix-package.sh      | 5 +++++
 tests/guix-system.sh       | 4 ++++
 tests/guix-time-machine.sh | 4 +++-
 6 files changed, 26 insertions(+), 2 deletions(-)

base-commit: 94133452aa49de672d69950b2e1a99432111074c
-- 
2.46.0

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Subject: bug#74394: Acknowledgement ([PATCH 0/2] Skip slow tests by
 default and run 'check' in Git pre-push hook.)
Message-ID: <handler.74394.B.173184503224362.ack <at> debbugs.gnu.org>
References: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
X-Gnu-PR-Message: ack 74394
X-Gnu-PR-Package: guix-patches
X-Gnu-PR-Keywords: patch
Reply-To: 74394 <at> debbugs.gnu.org
Date: Sun, 17 Nov 2024 12:04:02 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 guix-patches@HIDDEN

If you wish to submit further information on this problem, please
send it to 74394 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
74394: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D74394
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Message sent to maxim.cournoyer@HIDDEN, , guix-patches@HIDDEN:

Subject: [bug#74394] [PATCH 1/2] build: Exclude expensive tests in check target by default.
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: maxim.cournoyer@HIDDEN, , guix-patches@HIDDEN
Resent-Date: Sun, 17 Nov 2024 12:24:02 +0000
Resent-Message-ID: <handler.74394.B74394.173184619427708 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 74394 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer@HIDDEN>, Maxim Cournoyer <maxim.cournoyer@HIDDEN>, ( <paren@HIDDEN>, Andrew Tropin <andrew@HIDDEN>, Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>, Tanguy Le Carrour <tanguy@HIDDEN>
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Date: Sun, 17 Nov 2024 21:21:46 +0900
Message-ID: <5664cb3f6c9a888c74dd74a29912578e30697c72.1731844485.git.maxim.cournoyer@HIDDEN>
In-Reply-To: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
References: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

This reduces the test time of 'make -j check' from ~22 minutes to a bit more than one
minute on a fast machine.

* Makefile.am [CAN_RUN_TESTS] (check-local): New target.
* tests/guix-home.sh: Skip unless WITH_SLOW_TESTS is 1.
* tests/guix-package.sh: Likewise.
* tests/guix-system.sh: Likewise.
* tests/guix-time-machine.sh: Exclude one command.

Change-Id: Ieb61bf5d54aedd4bd32f93da792a4ac7dd8b9e7d
---
 Makefile.am                | 9 ++++++++-
 tests/guix-home.sh         | 5 +++++
 tests/guix-package.sh      | 5 +++++
 tests/guix-system.sh       | 4 ++++
 tests/guix-time-machine.sh | 4 +++-
 5 files changed, 25 insertions(+), 2 deletions(-)

diff --git a/Makefile.am b/Makefile.am
index e94ba87797..4bec4d4a06 100644
--- a/Makefile.am
+++ b/Makefile.am
@@ -14,7 +14,7 @@
 # Copyright © 2018 Oleg Pykhalov <go.wigust@HIDDEN>
 # Copyright © 2018 Alex Vong <alexvong1995@HIDDEN>
 # Copyright © 2019, 2023 Efraim Flashner <efraim@HIDDEN>
-# Copyright © 2020, 2021, 2023 Maxim Cournoyer <maxim.cournoyer@HIDDEN>
+# Copyright © 2020, 2021, 2023, 2024 Maxim Cournoyer <maxim.cournoyer@HIDDEN>
 # Copyright © 2021 Chris Marusich <cmmarusich@HIDDEN>
 # Copyright © 2021 Andrew Tropin <andrew@HIDDEN>
 # Copyright © 2023 Clément Lassieur <clement@HIDDEN>
@@ -668,6 +668,13 @@ tests/guix-gc.log:							\
   $(patsubst %.sh,%.log,$(filter-out tests/guix-gc.sh,$(SH_TESTS)))	\
   $(SCM_TESTS:%.scm=%.log)

+check-local:
+	@echo
+	@if [ "$$WITH_SLOW_TESTS" != 1 ]; then \
+	   echo "Skipping slow tests; set WITH_SLOW_TESTS to 1 to run them"; \
+	fi
+	@echo
+
 else !CAN_RUN_TESTS

 TESTS =
diff --git a/tests/guix-home.sh b/tests/guix-home.sh
index 649d811a0c..1c7b6da1cd 100644
--- a/tests/guix-home.sh
+++ b/tests/guix-home.sh
@@ -22,6 +22,11 @@
 # Test the 'guix home' using the external store, if any.
 #

+# This test is very expensive as guix-home appears to refer to the
+# current-guix package, which must be built the first time it runs (~20
+# minutes on a fast machine).
+test "$WITH_SLOW_TESTS" != 1 && exit 77
+
 set -e

 guix home --version
diff --git a/tests/guix-package.sh b/tests/guix-package.sh
index 945d59cdfb..5b9af19153 100644
--- a/tests/guix-package.sh
+++ b/tests/guix-package.sh
@@ -22,6 +22,11 @@
 # Test the `guix package' command-line utility.
 #

+# This test contains expensive operations such as 'guix package -A', which
+# cumulate into a total test time of multiple minutes (about 2 minutes on a
+# fast machine).
+test "$WITH_SLOW_TESTS" != 1 && exit 77
+
 guix package --version

 readlink_base ()
diff --git a/tests/guix-system.sh b/tests/guix-system.sh
index 99147cf332..bd343ec4bc 100644
--- a/tests/guix-system.sh
+++ b/tests/guix-system.sh
@@ -22,6 +22,10 @@
 # Test 'guix system', mostly error reporting.
 #

+# This test is expensive as it builds the images of multiple operating
+# systems; it takes more than a minute to run on a fast machine.
+test "$WITH_SLOW_TESTS" != 1 && exit 77
+
 set -e

 guix system --version
diff --git a/tests/guix-time-machine.sh b/tests/guix-time-machine.sh
index df75c681da..70a2443376 100644
--- a/tests/guix-time-machine.sh
+++ b/tests/guix-time-machine.sh
@@ -41,6 +41,8 @@ fi

 # Visiting a commit older than v0.16.0 must fail (this test is expensive
 # because it clones the whole repository).
-guix time-machine -q --commit=v0.15.0 $EXTRA_OPTIONS -- describe && false
+if [ "$WITH_SLOW_TESTS" = 1 ]; then
+    guix time-machine -q --commit=v0.15.0 $EXTRA_OPTIONS -- describe && false
+fi

 true
-- 
2.46.0

Message sent to maxim.cournoyer@HIDDEN, guix-patches@HIDDEN:

Subject: [bug#74394] [PATCH 2/2] etc: Ensure test suite passes in pre-push git hook.
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: maxim.cournoyer@HIDDEN, guix-patches@HIDDEN
Resent-Date: Sun, 17 Nov 2024 12:24:02 +0000
Resent-Message-ID: <handler.74394.B74394.173184619727723 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 74394 <at> debbugs.gnu.org
Cc: Maxim Cournoyer <maxim.cournoyer@HIDDEN>, Maxim Cournoyer <maxim.cournoyer@HIDDEN>
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Date: Sun, 17 Nov 2024 21:21:47 +0900
Message-ID: <9a35e1aa323f608a2a3e8da1fd23517b3b4a77f1.1731844485.git.maxim.cournoyer@HIDDEN>
In-Reply-To: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
References: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
MIME-Version: 1.0
Content-Transfer-Encoding: 8bit
Precedence: list
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>

* etc/git/pre-push: Run 'make check' on as many cores as available, which
takes about 1 minute on a fast machine.

Change-Id: I60e77d18bc885e8013daae70580126465169b1c6
---
 etc/git/pre-push | 1 +
 1 file changed, 1 insertion(+)

diff --git a/etc/git/pre-push b/etc/git/pre-push
index 325b23854b..74273ee180 100755
--- a/etc/git/pre-push
+++ b/etc/git/pre-push
@@ -33,6 +33,7 @@ do
 		case "$2" in
 		    *.gnu.org*)
 			set -e
+                        make check -j$(nproc)
 			make check-channel-news
 			exec guix git authenticate
 			exit 127
-- 
2.46.0

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#74394] [PATCH 0/2] Skip slow tests by default and run 'check' in Git pre-push hook.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Fri, 29 Nov 2024 10:07:02 +0000
Resent-Message-ID: <handler.74394.B74394.173287479427627 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 74394
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Cc: 74394 <at> debbugs.gnu.org
Received: via spool by 74394-submit <at> debbugs.gnu.org id=B74394.173287479427627
          (code B ref 74394); Fri, 29 Nov 2024 10:07:02 +0000
Received: (at 74394) by debbugs.gnu.org; 29 Nov 2024 10:06:34 +0000
Received: from localhost ([127.0.0.1]:41189 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1tGxtR-0007BW-Tf
	for submit <at> debbugs.gnu.org; Fri, 29 Nov 2024 05:06:34 -0500
Received: from eggs.gnu.org ([209.51.188.92]:46342)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1tGxtQ-0007BI-6y
 for 74394 <at> debbugs.gnu.org; Fri, 29 Nov 2024 05:06:32 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1tGxtK-0006vJ-Nh; Fri, 29 Nov 2024 05:06:26 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=Z/CWGlh/7Lx6tEoysBq/e569H3pXHYqoJPlsWQmexV8=; b=cNN/NWHtOORCMvJWtl1D
 vcuFHCIv6YB+unVCrVvIlrP86B3gJ/f5wFAc3E688NJ+u2leZ890gYbYiCMXKLSlMsfbPJ30Z6b9I
 lR05ZPs6nZnZYHt4sVYWC6MW6KzE+4Et3VWJSmbt/TA0KAHn29yOceNAF/FLSQyDs1V/xinlWy8WS
 CLJtDbTIxtX39yAA9zmYZzfYVu99ZJSOhDmi0WvD1TokF1sTLUsAauF3iAgZHPYXgLnesGRAi2EKb
 0rU9QRbxWdrPJpcs57twN2zaCWxi2CjWCtZFtP4sebssteJ7s1DYakgbNeP1w5jixI/D8P+A2DRh2
 itXvVOI6v70aGA==;
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
In-Reply-To: <cover.1731844485.git.maxim.cournoyer@HIDDEN> (Maxim
 Cournoyer's message of "Sun, 17 Nov 2024 21:03:30 +0900")
References: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
Date: Fri, 29 Nov 2024 11:05:51 +0100
Message-ID: <87y1122w3k.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

> This is a simple change that should ensure test suite breakages are detec=
ted
> as early as possible and avoid tests breaking changes to be pushed.  This=
 is
> made possible by skipping a few expensive tests suite, bringing down the =
total
> test time to about 1 minute on a fast machine.
>
> We could call it a "distributed CI" approach ;-).

I agree with the goal, of course, but not with the method: even without
expensive tests, =E2=80=9Cmake check=E2=80=9D is going to take maybe 5=E2=
=80=9310 minutes, and
having that happen when you run =E2=80=9Cgit push=E2=80=9D can be a terribl=
e development
experience (especially since the developer most likely either already
ran the test suite or part of it right before, or pushes package changes
that have infinitely small probability of breaking =E2=80=9Cmake check=E2=
=80=9D).

Back to CI and not breaking things: I think that we should have a
workflow where the forge triggers those checks and puts a green light if
it passes, red light otherwise.  (Basically what everybody else is
doing. :-))

To me this should be one of the goals for the project in 2025.

> To run the complete test suite including the slow tests (as is the case p=
rior
> this change):
>
> make check WITH_SLOW_TESTS=3D1

This variable itself may still be useful though (I=E2=80=99d call it
=E2=80=98RUN_EXPENSIVE_TESTS=E2=80=99 or something like that=E2=80=94that=
=E2=80=99s the name used in
Coreutils=E2=80=94, =E2=80=9Cexpensive=E2=80=9D being the key word).  I wou=
ld also keep it on by
default.

Thanks,
Ludo=E2=80=99.

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#74394] [PATCH 0/2] Skip slow tests by default and run 'check' in Git pre-push hook.
Resent-From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Tue, 17 Dec 2024 00:31:01 +0000
Resent-Message-ID: <handler.74394.B74394.173439541514008 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 74394
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Cc: 74394 <at> debbugs.gnu.org
Received: via spool by 74394-submit <at> debbugs.gnu.org id=B74394.173439541514008
          (code B ref 74394); Tue, 17 Dec 2024 00:31:01 +0000
Received: (at 74394) by debbugs.gnu.org; 17 Dec 2024 00:30:15 +0000
Received: from localhost ([127.0.0.1]:56779 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1tNLTa-0003ds-Ke
	for submit <at> debbugs.gnu.org; Mon, 16 Dec 2024 19:30:15 -0500
Received: from mail-pj1-f43.google.com ([209.85.216.43]:52408)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <maxim.cournoyer@HIDDEN>) id 1tNLTX-0003ck-Vi
 for 74394 <at> debbugs.gnu.org; Mon, 16 Dec 2024 19:30:13 -0500
Received: by mail-pj1-f43.google.com with SMTP id
 98e67ed59e1d1-2efded08c79so3237058a91.0
 for <74394 <at> debbugs.gnu.org>; Mon, 16 Dec 2024 16:30:11 -0800 (PST)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=gmail.com; s=20230601; t=1734395350; x=1735000150; darn=debbugs.gnu.org;
 h=content-transfer-encoding:mime-version:user-agent:message-id:date
 :references:in-reply-to:subject:cc:to:from:from:to:cc:subject:date
 :message-id:reply-to;
 bh=407peAdfE0XPtE+DMmVaQOsUNP/m/C+2xsk5zauvlaU=;
 b=KCYNcc6aFOvWh1Iqg1UZdxHJpew6XIl1BXYsDvIm4jgqFcAPNFgld5G7/5YhlVeVv8
 ur4VPb2DqG6ECeerrWfywjhtViq34HYPh4cbLqeEPTzujkyTq49EQGVVUhuF6yQUxpWB
 3m6fWasH9nlA/35Z1nt4roZGv1crj6Q6/GF6cU++HZ0NTEHdgMS2R2rfhXmQJ+EY0Jwz
 6WBqaCy2z2YZNa/RQyDVBYPWmt8S3AQDIEY470Umdpj+ds07oD85HEvboItZCdSDHfsa
 qSl6sYczcZ5D2zOd/kHRcef5rEpDAAUe8WPmRZoQNxTY5kuap3h+Cgz0iC3rXVBcaVhF
 ClKA==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
 d=1e100.net; s=20230601; t=1734395350; x=1735000150;
 h=content-transfer-encoding:mime-version:user-agent:message-id:date
 :references:in-reply-to:subject:cc:to:from:x-gm-message-state:from
 :to:cc:subject:date:message-id:reply-to;
 bh=407peAdfE0XPtE+DMmVaQOsUNP/m/C+2xsk5zauvlaU=;
 b=Zn72KeA+bz7I7wfml49fkdc6G54CqiNT/RG3g4CAMUm1VzSzBxtv54YDrzobyuT9Iu
 k2ciDUgJ5iMVw4rqltFdqWW8FeT4N4ESk5WpH4wZFW8jSbtfko2JTtYVADFe7E+cae4/
 +1KqvAQGiKIJNRhQrbQG76PO2mENayRs38ssNGUS+KuW9Fi5uy5GJ6cW6EiAYJsD8ghi
 dQN4iOwRk3tbn6KUe2+vRQWgeQF4URTb0pW77r73AgWfOarNdMmN44ufKKiv19giOob/
 bDlezDlVUIbCKbr2OBY8/k7Eo1k+A8TyGcj6qpnVZwaDTGmcCtC6j7MFnjDISLX/hqI1
 IzCw==
X-Gm-Message-State: AOJu0YxTraooxJQX0636rX/RWtFHTSVY7eM/T6TXh+TG0akUqhIpuzgt
 UbLvo/oOBNDTetgBAOdengghPY//yhUEwXaUs1dtC/zKg7CzE2Em92GeJhrP
X-Gm-Gg: ASbGncu9knMVzeeX++SdozKBfm2s3MoCnw9Rm+SYmUbGc2KFaamB5aITOs0DuWtItiy
 PcpiCTWTD7sarJz38dwQgEyZIsf7WjXqQK/0JArxBdw1hNQXR81b0+G0I6kJ7Eu/9l/zwpWNlvH
 /a4sxYmGzUY4/Zc3z0tcfhhh+XalA+8+Ug73k4RvK41hVJFj1kTFkSBH5xGqFX7fPshzVVS6VKC
 mLlKdOTdJDrmKW5WQT6FSifbVhQeKCXuuLd91FGjpyaa036Dy8MWg==
X-Google-Smtp-Source: AGHT+IH7pYWJ5W7wg7aNFShbQfjKs0b+OwoBPsXv0OKuSNB+n4dIs//38kwlIraxlOiKNHQ25sTXpw==
X-Received: by 2002:a17:90a:e70d:b0:2ee:df57:b194 with SMTP id
 98e67ed59e1d1-2f28fd6b922mr18604676a91.21.1734395350052; 
 Mon, 16 Dec 2024 16:29:10 -0800 (PST)
Received: from terra ([2405:6586:be0:0:c8ff:1707:9b9:af89])
 by smtp.gmail.com with ESMTPSA id
 98e67ed59e1d1-2f142de475bsm8723189a91.31.2024.12.16.16.29.08
 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256);
 Mon, 16 Dec 2024 16:29:09 -0800 (PST)
From: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
In-Reply-To: <87y1122w3k.fsf@HIDDEN> ("Ludovic =?UTF-8?Q?Court=C3=A8s?="'s message of "Fri, 29 Nov 2024 11:05:51 +0100")
References: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
 <87y1122w3k.fsf@HIDDEN>
Date: Tue, 17 Dec 2024 09:28:59 +0900
Message-ID: <87zfkvrw38.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)

Hi Ludovic,

Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

> Hi Maxim,
>
> Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:
>
>> This is a simple change that should ensure test suite breakages are dete=
cted
>> as early as possible and avoid tests breaking changes to be pushed.  Thi=
s is
>> made possible by skipping a few expensive tests suite, bringing down the=
 total
>> test time to about 1 minute on a fast machine.
>>
>> We could call it a "distributed CI" approach ;-).
>
> I agree with the goal, of course, but not with the method: even without
> expensive tests, =E2=80=9Cmake check=E2=80=9D is going to take maybe 5=E2=
=80=9310 minutes, and
> having that happen when you run =E2=80=9Cgit push=E2=80=9D can be a terri=
ble development
> experience (especially since the developer most likely either already
> ran the test suite or part of it right before, or pushes package changes
> that have infinitely small probability of breaking =E2=80=9Cmake check=E2=
=80=9D).

As I wrote, 'make check' with this change takes about 1 minute on my
machine; I'd be curious to know how long it takes on other people
machines; I suspect a bit more; if it's too slow, we can skip more, or
find out ways to make the tests run faster.

> Back to CI and not breaking things: I think that we should have a
> workflow where the forge triggers those checks and puts a green light if
> it passes, red light otherwise.  (Basically what everybody else is
> doing. :-))
>
> To me this should be one of the goals for the project in 2025.

That would be great.  I wonder how far QA is from making this
achievable.

>> To run the complete test suite including the slow tests (as is the case =
prior
>> this change):
>>
>> make check WITH_SLOW_TESTS=3D1
>
> This variable itself may still be useful though (I=E2=80=99d call it
> =E2=80=98RUN_EXPENSIVE_TESTS=E2=80=99 or something like that=E2=80=94that=
=E2=80=99s the name used in
> Coreutils=E2=80=94, =E2=80=9Cexpensive=E2=80=9D being the key word).  I w=
ould also keep it on by
> default.

One of the tests that was unbearably long when I measured was the
time-machine test.  It took about 20 minutes to fetch the git repository
with guile-git and run the test (which does extra work compared to the
CLI like validating each object).  I don't think we want this kind of
experience by default (because that'd probably convince people that
running the test suite often is not a reasonable thing to do).  The
other tests were more reasonable, with the longer ones in the 2-3
minutes range on my machine, IIRC.  Perhaps we could have this 20 minute
outlier skipped by default, maybe with a RUN_PROHIBITIVE_TESTS flag that
would default to 0 (false).

A long time ago I had read a blog post that argued that unit tests
should be small and fast [0], and there's a lot of good about that.
Fast tests usually translates in running the test suite more often and
catching breakage earlier.  As the author states, it also makes it
possible to determine whether a bug lies in the core logic or in the
integration of the many parts (unit tests vs integration tests), when
unit tests are decoupled from the whole system (typically by mocking all
external interfaces).

[0]  https://www.artima.com/weblogs/viewpost.jsp?thread=3D126923

--=20
Thanks,
Maxim

Message sent to guix-patches@HIDDEN:

X-Loop: help-debbugs@HIDDEN
Subject: [bug#74394] [PATCH 0/2] Skip slow tests by default and run 'check' in Git pre-push hook.
Resent-From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: guix-patches@HIDDEN
Resent-Date: Tue, 17 Dec 2024 14:52:02 +0000
Resent-Message-ID: <handler.74394.B74394.173444710614729 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
X-GNU-PR-Message: followup 74394
X-GNU-PR-Package: guix-patches
X-GNU-PR-Keywords: patch
To: Maxim Cournoyer <maxim.cournoyer@HIDDEN>
Cc: 74394 <at> debbugs.gnu.org
Received: via spool by 74394-submit <at> debbugs.gnu.org id=B74394.173444710614729
          (code B ref 74394); Tue, 17 Dec 2024 14:52:02 +0000
Received: (at 74394) by debbugs.gnu.org; 17 Dec 2024 14:51:46 +0000
Received: from localhost ([127.0.0.1]:58300 helo=debbugs.gnu.org)
	by debbugs.gnu.org with esmtp (Exim 4.84_2)
	(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
	id 1tNYvJ-0003pT-RT
	for submit <at> debbugs.gnu.org; Tue, 17 Dec 2024 09:51:46 -0500
Received: from eggs.gnu.org ([209.51.188.92]:32812)
 by debbugs.gnu.org with esmtp (Exim 4.84_2)
 (envelope-from <ludo@HIDDEN>) id 1tNYvF-0003os-Px
 for 74394 <at> debbugs.gnu.org; Tue, 17 Dec 2024 09:51:44 -0500
Received: from fencepost.gnu.org ([2001:470:142:3::e])
 by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
 (Exim 4.90_1) (envelope-from <ludo@HIDDEN>)
 id 1tNYvA-0008Bz-A5; Tue, 17 Dec 2024 09:51:36 -0500
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=gnu.org;
 s=fencepost-gnu-org; h=MIME-Version:Date:References:In-Reply-To:Subject:To:
 From; bh=thliptRBZjZnRZa8B7gPwFYaEkpgB4ec6dsU4CugelM=; b=Abqi5GTWHxrjfe3Va9tz
 tv1OVi9I3LTMj9f/6U4J/Ha9z1xVBfIK1bxo4NiBi2q7h4f+qt8k6LhPOhOAh6w8hy6TYAuQBFdVx
 FdK667rakqjQZPOW46umt70v6nyXASf255Tr5CDP4OSb1IHLZzsRqPbNbyQfTo8xcUQKISzAqpyuq
 efPHsnZp/ONXPf1G+35gVl3Wya4gYdrgUSBCYh5bq7YsS1vM3XVkhYuLVN0RTcQSFXCI/ODnc8T45
 dqo+5FjHIS9kjvLnJ/+Rnur92q3HfN9n4L3p8OnhhgIYdq+be0s+7CTq/XsNEeZ6nT/pXhknFO70l
 rEGzxqZWAMQwfQ==;
From: Ludovic =?UTF-8?Q?Court=C3=A8s?= <ludo@HIDDEN>
In-Reply-To: <87zfkvrw38.fsf@HIDDEN> (Maxim Cournoyer's message of "Tue, 17
 Dec 2024 09:28:59 +0900")
References: <cover.1731844485.git.maxim.cournoyer@HIDDEN>
 <87y1122w3k.fsf@HIDDEN> <87zfkvrw38.fsf@HIDDEN>
X-URL: http://www.fdn.fr/~lcourtes/
X-Revolutionary-Date: Septidi 27 Frimaire an 233 de la =?UTF-8?Q?R=C3=A9volution,?= jour du =?UTF-8?Q?Li=C3=A8ge?=
X-PGP-Key-ID: 0x090B11993D9AEBB5
X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc
X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4  0CFB 090B 1199 3D9A EBB5
X-OS: x86_64-pc-linux-gnu
Date: Tue, 17 Dec 2024 15:51:33 +0100
Message-ID: <87frmmnz0q.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: -2.3 (--)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -3.3 (---)

Hi Maxim,

Maxim Cournoyer <maxim.cournoyer@HIDDEN> skribis:

> Ludovic Court=C3=A8s <ludo@HIDDEN> writes:

[...]

>> I agree with the goal, of course, but not with the method: even without
>> expensive tests, =E2=80=9Cmake check=E2=80=9D is going to take maybe 5=
=E2=80=9310 minutes, and
>> having that happen when you run =E2=80=9Cgit push=E2=80=9D can be a terr=
ible development
>> experience (especially since the developer most likely either already
>> ran the test suite or part of it right before, or pushes package changes
>> that have infinitely small probability of breaking =E2=80=9Cmake check=
=E2=80=9D).
>
> As I wrote, 'make check' with this change takes about 1 minute on my
> machine;

Right now, without your patch, we have:

--8<---------------cut here---------------start------------->8---
$ wget -qO- $(guix build --log-file guix --no-grafts)|gunzip |grep "\`check=
'"
starting phase `check'
phase `check' succeeded after 2049.2 seconds
--8<---------------cut here---------------end--------------->8---

More than 30mn on the fast machines of the build farm, and with some of
the expensive tests already skipped (those that require network access:
time-machine, pack -RR, etc.).

This patch is not dividing wall-clock time by 30, is it?

>> This variable itself may still be useful though (I=E2=80=99d call it
>> =E2=80=98RUN_EXPENSIVE_TESTS=E2=80=99 or something like that=E2=80=94tha=
t=E2=80=99s the name used in
>> Coreutils=E2=80=94, =E2=80=9Cexpensive=E2=80=9D being the key word).  I =
would also keep it on by
>> default.
>
> One of the tests that was unbearably long when I measured was the
> time-machine test.  It took about 20 minutes to fetch the git repository
> with guile-git and run the test (which does extra work compared to the
> CLI like validating each object).  I don't think we want this kind of
> experience by default (because that'd probably convince people that
> running the test suite often is not a reasonable thing to do).  The
> other tests were more reasonable, with the longer ones in the 2-3
> minutes range on my machine, IIRC.  Perhaps we could have this 20 minute
> outlier skipped by default, maybe with a RUN_PROHIBITIVE_TESTS flag that
> would default to 0 (false).

Yeah okay, maybe we should skip them by default, and maybe we can find a
way to ensure developers do run them periodically.

> A long time ago I had read a blog post that argued that unit tests
> should be small and fast [0],

I actually agree.  :-)

Thanks!

Ludo=E2=80=99.

Last modified: Sun, 12 Jan 2025 05:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.