Paul Eggert <eggert@HIDDEN>
to control <at> debbugs.gnu.org.
Full text available.
Received: (at 73784) by debbugs.gnu.org; 2 Jan 2025 19:05:48 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jan 02 14:05:48 2025
Received: from localhost ([127.0.0.1]:46690 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1tTQVv-0001jT-Ug
for submit <at> debbugs.gnu.org; Thu, 02 Jan 2025 14:05:48 -0500
Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:29807)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <yamato@HIDDEN>) id 1tTQVt-0001jG-4e
for 73784 <at> debbugs.gnu.org; Thu, 02 Jan 2025 14:05:46 -0500
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
s=mimecast20190719; t=1735844744;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding:
in-reply-to:in-reply-to:references:references;
bh=1SJXAyhYfQBu0Evjcygeb87TtYvbbM/HMPrAi4gSh1c=;
b=ASKoglQcf/Hd7OHCS1iIPNbdGL1iUgxX0rm3LLBcEb96hgQNPGGrXT6YqwxCH442MEX+7V
VXSRdaBSQ7l3UkiSlJxYBD1hdX9vj9fNqRZbYoDqn1grFX1O7LHwNotmSVsQe1u6tRpUu9
KT6qeEZyol1Sk9+u02Xuf4n7R5ahk/c=
Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com
(ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by
relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,
cipher=TLS_AES_256_GCM_SHA384) id us-mta-35-RDfAMBdTMee1NZKJ5qCbhw-1; Thu,
02 Jan 2025 14:05:41 -0500
X-MC-Unique: RDfAMBdTMee1NZKJ5qCbhw-1
X-Mimecast-MFC-AGG-ID: RDfAMBdTMee1NZKJ5qCbhw
Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com
(mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS
id 006821956087; Thu, 2 Jan 2025 19:05:40 +0000 (UTC)
Received: from localhost (unknown [10.64.240.119])
by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP
id ACEB11956052; Thu, 2 Jan 2025 19:05:37 +0000 (UTC)
Date: Fri, 03 Jan 2025 04:05:35 +0900 (JST)
Message-Id: <20250103.040535.1761642872805390481.yamato@HIDDEN>
To: P@HIDDEN
Subject: Re: bug#73784: [PATCH] cp: new option --nocache-source
From: Masatake YAMATO <yamato@HIDDEN>
In-Reply-To: <fcd92e09-761c-44c3-a0e4-475142c41be9@HIDDEN>
References: <20241013045611.2572765-1-yamato@HIDDEN>
<fcd92e09-761c-44c3-a0e4-475142c41be9@HIDDEN>
Organization: Red Hat Japan, K.K.
Mime-Version: 1.0
X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17
X-Mimecast-Spam-Score: 0
X-Mimecast-MFC-PROC-ID: sJT11KwScrdFQLIlb_2yneJMXvp50Z3EzCYvU1COgpA_1735844740
X-Mimecast-Originator: redhat.com
Content-Type: Text/Plain; charset=iso-8859-1
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 73784
Cc: 73784 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -1.0 (-)
From: P=E1draig Brady <P@HIDDEN>
Subject: Re: bug#73784: [PATCH] cp: new option --nocache-source
Date: Sun, 13 Oct 2024 15:59:27 +0100
> On 13/10/2024 05:56, Masatake YAMATO wrote:
>> When copying files, the system data cache are consumed, the system
>> data cache is utilized for both the source and destination files. In
>> scenarios such as creating backup files for old, unused files, it is
>> clear to users that these files will not be needed in the near
>> future. In such cases, retaining the data for these files in the cache
>> constitutes a waste of computer resources, especially when running
>> applications that require significant memory in the foreground.
>> With the new option, users will have the ability to request the
>> discarding of the system data cache, thereby avoiding the unwanted
>> swapping out of data from foreground processes.
>> I evaluated cache consumption using a script called
>> run.bash. Initially, run.bash creates many small files, each 8 KB in
>> size. It then copies these files using the cp command, both with and
>> without the specified option. Finally, it reports the difference in
>> the total size of the caches before and after the copying process.
>> run.bash:
>> #!/bin/bash
>> CP=3D$1
>> shift
>> [[ -e "$CP" ]] || {
>> =09echo "no file found: $CP" 1>&2
>> =09exit 1
>> }
>> N=3D8
>> S=3Ddrop-src
>> D=3D${HOME}/drop-dst
>> mkdir -p $S
>> mkdir -p $D
>> start=3D
>> end=3D
>> print_cached()
>> {
>> =09grep ^Cached: /proc/meminfo
>> }
>> start()
>> {
>> =09start=3D$(print_cached | awk '{print $2}')
>> }
>> end()
>> {
>> =09end=3D$(print_cached | awk '{print $2}')
>> }
>> report()
>> {
>> =09echo -n "delta[$N:$1/$2]: "
>> =09expr "$end" - "$start"
>> }
>> cleanup()
>> {
>> =09local i
>> =09local j
>> =09for ((i =3D 0; i < 10; i++)); do
>> =09 for ((j =3D 0; j < 10; j++)); do
>> =09=09rm -f $S/F-${i}${j}*
>> =09=09rm -f $D/F-${i}${j}*
>> =09 done
>> =09done
>> =09rm -f $S/F-*
>> =09rm -f $D/F-*
>> }
>> prep()
>> {
>> =09local i
>> =09for ((i =3D 0; i < 1024 * $N; i++ )); do
>> =09 if ! dd if=3D/dev/zero of=3D$S/F-$i bs=3D4096 count=3D2 \
>> =09 status=3Dnone; then
>> =09=09echo "failed in dd of=3D$S/F-$F" 1>&2
>> =09=09exit 1
>> =09 fi
>> =09done
>> =09sync
>> }
>> run_cp()
>> {
>> =09start
>> =09local i
>> =09time for ((i =3D 0; i < 1024 * $N; i++ )); do
>> =09 if ! "${CP}" "$@" "$S/F-$i" "$D/F-$i"; then
>> =09=09echo "failed in cp " "$@" "$S/F-$i" " $D/F-$i" 1>&2
>> =09=09exit 1
>> =09 fi
>> =09done
>> =09end
>> =09report "$1" $2
>> }
>> cleanup
>> sync
>> prep
>> run_cp "$@"
>> running:
>> ~/coreutils/nocache$ ./run.bash ../src/cp
>> real=090m16.051s
>> user=090m4.249s
>> sys=090m12.437s
>> delta[8:/]: 65548
>> ~/coreutils/nocache$ ./run.bash ../src/cp --nocache-source
>> real=090m17.109s
>> user=090m4.492s
>> sys=090m13.317s
>> delta[8:--nocache-source/]: 620
>> --nocache-source option suppresses the consumption of the cache
>> massively.
>=20
> Thanks for the patch.
> I have some reservations/notes though...
>=20
> There is nothing particularly special about cp, that it might need
> this option.
> I.e. it would be nice to be able to wrap any program so that it
> streamed
> data through the cache, rather than aggressively cached. I'm not sure
> how to do that,
> but also I'd be reluctant to start adding such options to individual
> commands though.
> Perhaps Linux' open() may gain an O_STREAM flag in future that might
> be
> more generally applied with a wrapper or something.
I found an interesting article: https://www.phoronix.com/news/Uncached-Buff=
ered-IO-Linux-6.14
It seems that RWF_DONTCACHE flag of pwritev and preadv implements
what we need.
When Linux-6.14 is released, I will rewrite my patch based on
RWF_DONTCACHE.
Masatake YAMATO
> For single (large) files, one already has this functionality in dd.
>=20
> On the write side, you'd also have to worry about syncing, to make the
> drop cache advisory effective, and this could impact performance.
>=20
> Might this drop caches for already cached files,
> which cp may just happen to be copying,
> thus potentially impacting performance for other programs.
>=20
> If reflinking we probably would not want to do this operation,
> since we're not reading the source.
>=20
> thanks,
> P=E1draig
>
bug-coreutils@HIDDEN:bug#73784; Package coreutils.
Full text available.
Received: (at 73784) by debbugs.gnu.org; 13 Oct 2024 15:00:49 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 11:00:49 2024
Received: from localhost ([127.0.0.1]:52056 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1t005Q-0007nA-UO
for submit <at> debbugs.gnu.org; Sun, 13 Oct 2024 11:00:49 -0400
Received: from mail-wm1-f46.google.com ([209.85.128.46]:47394)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <pixelbeat@HIDDEN>) id 1t005O-0007mz-EY
for 73784 <at> debbugs.gnu.org; Sun, 13 Oct 2024 11:00:47 -0400
Received: by mail-wm1-f46.google.com with SMTP id
5b1f17b1804b1-4311e470fdaso20198035e9.3
for <73784 <at> debbugs.gnu.org>; Sun, 13 Oct 2024 08:00:30 -0700 (PDT)
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=gmail.com; s=20230601; t=1728831570; x=1729436370; darn=debbugs.gnu.org;
h=content-transfer-encoding:in-reply-to:from:content-language
:references:to:subject:user-agent:mime-version:date:message-id
:sender:from:to:cc:subject:date:message-id:reply-to;
bh=cux9yVtmYgMWxuT/XMWiv+jmZiy3n1EJaY+NoT0mxX4=;
b=SNo1PhWckh1EYSFWJuxDc7Lrlm2s/G7Nh3v3SdPdrrJCl81YUjeOYfeiE5ucUMZZpu
rESRTMa6385QPosY19DtonEIGvrk0QoFm/1b8yoiStIbRioFERBApX7WbC8OIjLuwrjb
MilxCoV48tdJzrkr5wMlOudAJUEvvrE9GfWPHm07gWAmGOkI3tk1aMEHwhh8rJ22BfJ2
xok2DDz0QcySn06YHOrpQfnYrEU6bndDxfVRTPkuJvOE4e4VIwajio/7NF5AcIFYzLbw
bEMrP1WfWkMjIKRvYkJI10yFoJDW3kb2ORw+Oy1fPNc2oHSmABYgq01T6yMSJkRuQjYe
4XSw==
X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;
d=1e100.net; s=20230601; t=1728831570; x=1729436370;
h=content-transfer-encoding:in-reply-to:from:content-language
:references:to:subject:user-agent:mime-version:date:message-id
:sender:x-gm-message-state:from:to:cc:subject:date:message-id
:reply-to;
bh=cux9yVtmYgMWxuT/XMWiv+jmZiy3n1EJaY+NoT0mxX4=;
b=G8ST8JBm6syPcKFufloa3sLmcoMs2f5MAC6d2qqJwYiNlMDCnWVeaU3dCgZY+1Xl9p
llQiaGAsecRYk0xLMBcr7WgI1JRMN37fnlbRBVdR+M8OBX0q+ayV6nV2ba40YW2JIPJ4
oQ/0rqZ0V5gQgklVFYEeVeobtz2KuocE63/2t14o61BX47uhUvsk+8yq1uQcjetfoMJs
mTpmRVDRkelqabQ6PPBRLOKKA5J+yXbwTif+t7ML2EJ5sOzDD/sW8+z0DP8DQ7ZUSgYP
A0zSVGwyP2Sx4FyRCnMLuAOZ8lxW2wlKlVxS5F0HtQ2CMUFANtAqonpdHCglnhA/jeP2
DIMQ==
X-Forwarded-Encrypted: i=1;
AJvYcCX+ameheG7DIqJ54XqFj3rxqYHWnJTde3fnTgS1xYlH+Me0CHK7wYoIaiqp683EnIpBOWbf1Q==@debbugs.gnu.org
X-Gm-Message-State: AOJu0YyRPui01BXrJR9HdyiDJzPChddM9AeFDtvti6d6qaCDlMHbg/Kg
y4D2bKCEry+i+dHfmwMdewB2Y+HIpKUrBZ0ApiZXjpcDbztwpI00
X-Google-Smtp-Source: AGHT+IFss1bEYk6z2SfXrT+BFR0iQIZraqhWJq/6U5ey3gqyBt1GsSnGcz+RGuWCPZItLj9B8ubm6w==
X-Received: by 2002:a05:600c:190d:b0:430:53f6:f20b with SMTP id
5b1f17b1804b1-4311df56f1amr78940785e9.23.1728831569816;
Sun, 13 Oct 2024 07:59:29 -0700 (PDT)
Received: from [192.168.1.156]
(86-44-211-146-dynamic.agg2.lod.rsl-rtd.eircom.net. [86.44.211.146])
by smtp.googlemail.com with ESMTPSA id
5b1f17b1804b1-430ccf1f797sm128401055e9.4.2024.10.13.07.59.28
(version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128);
Sun, 13 Oct 2024 07:59:28 -0700 (PDT)
Message-ID: <fcd92e09-761c-44c3-a0e4-475142c41be9@HIDDEN>
Date: Sun, 13 Oct 2024 15:59:27 +0100
MIME-Version: 1.0
User-Agent: Mozilla Thunderbird Beta
Subject: Re: bug#73784: [PATCH] cp: new option --nocache-source
To: Masatake YAMATO <yamato@HIDDEN>, 73784 <at> debbugs.gnu.org
References: <20241013045611.2572765-1-yamato@HIDDEN>
Content-Language: en-US
From: =?UTF-8?Q?P=C3=A1draig_Brady?= <P@HIDDEN>
In-Reply-To: <20241013045611.2572765-1-yamato@HIDDEN>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Spam-Score: 0.3 (/)
X-Debbugs-Envelope-To: 73784
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.7 (/)
On 13/10/2024 05:56, Masatake YAMATO wrote:
> When copying files, the system data cache are consumed, the system
> data cache is utilized for both the source and destination files. In
> scenarios such as creating backup files for old, unused files, it is
> clear to users that these files will not be needed in the near
> future. In such cases, retaining the data for these files in the cache
> constitutes a waste of computer resources, especially when running
> applications that require significant memory in the foreground.
>
> With the new option, users will have the ability to request the
> discarding of the system data cache, thereby avoiding the unwanted
> swapping out of data from foreground processes.
>
> I evaluated cache consumption using a script called
> run.bash. Initially, run.bash creates many small files, each 8 KB in
> size. It then copies these files using the cp command, both with and
> without the specified option. Finally, it reports the difference in
> the total size of the caches before and after the copying process.
>
> run.bash:
>
> #!/bin/bash
> CP=$1
> shift
>
> [[ -e "$CP" ]] || {
> echo "no file found: $CP" 1>&2
> exit 1
> }
>
> N=8
> S=drop-src
> D=${HOME}/drop-dst
>
> mkdir -p $S
> mkdir -p $D
>
> start=
> end=
> print_cached()
> {
> grep ^Cached: /proc/meminfo
> }
>
> start()
> {
> start=$(print_cached | awk '{print $2}')
> }
>
> end()
> {
> end=$(print_cached | awk '{print $2}')
> }
>
> report()
> {
> echo -n "delta[$N:$1/$2]: "
> expr "$end" - "$start"
> }
>
> cleanup()
> {
> local i
> local j
> for ((i = 0; i < 10; i++)); do
> for ((j = 0; j < 10; j++)); do
> rm -f $S/F-${i}${j}*
> rm -f $D/F-${i}${j}*
> done
> done
> rm -f $S/F-*
> rm -f $D/F-*
> }
>
> prep()
> {
> local i
> for ((i = 0; i < 1024 * $N; i++ )); do
> if ! dd if=/dev/zero of=$S/F-$i bs=4096 count=2 \
> status=none; then
> echo "failed in dd of=$S/F-$F" 1>&2
> exit 1
> fi
> done
> sync
> }
>
> run_cp()
> {
> start
>
> local i
> time for ((i = 0; i < 1024 * $N; i++ )); do
> if ! "${CP}" "$@" "$S/F-$i" "$D/F-$i"; then
> echo "failed in cp " "$@" "$S/F-$i" " $D/F-$i" 1>&2
> exit 1
> fi
> done
>
> end
> report "$1" $2
> }
>
> cleanup
> sync
>
> prep
> run_cp "$@"
>
> running:
>
> ~/coreutils/nocache$ ./run.bash ../src/cp
>
> real 0m16.051s
> user 0m4.249s
> sys 0m12.437s
> delta[8:/]: 65548
> ~/coreutils/nocache$ ./run.bash ../src/cp --nocache-source
>
> real 0m17.109s
> user 0m4.492s
> sys 0m13.317s
> delta[8:--nocache-source/]: 620
>
> --nocache-source option suppresses the consumption of the cache
> massively.
Thanks for the patch.
I have some reservations/notes though...
There is nothing particularly special about cp, that it might need this option.
I.e. it would be nice to be able to wrap any program so that it streamed
data through the cache, rather than aggressively cached. I'm not sure how to do that,
but also I'd be reluctant to start adding such options to individual commands though.
Perhaps Linux' open() may gain an O_STREAM flag in future that might be
more generally applied with a wrapper or something.
For single (large) files, one already has this functionality in dd.
On the write side, you'd also have to worry about syncing, to make the
drop cache advisory effective, and this could impact performance.
Might this drop caches for already cached files,
which cp may just happen to be copying,
thus potentially impacting performance for other programs.
If reflinking we probably would not want to do this operation,
since we're not reading the source.
thanks,
Pádraig
bug-coreutils@HIDDEN:bug#73784; Package coreutils.
Full text available.
Received: (at submit) by debbugs.gnu.org; 13 Oct 2024 04:56:44 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Oct 13 00:56:44 2024
Received: from localhost ([127.0.0.1]:49864 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1szqep-0007Yp-6d
for submit <at> debbugs.gnu.org; Sun, 13 Oct 2024 00:56:43 -0400
Received: from lists.gnu.org ([209.51.188.17]:47082)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <yamato@HIDDEN>) id 1szqem-0007Yh-Fg
for submit <at> debbugs.gnu.org; Sun, 13 Oct 2024 00:56:41 -0400
Received: from eggs.gnu.org ([2001:470:142:3::10])
by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <yamato@HIDDEN>) id 1szqeX-0004yf-4r
for bug-coreutils@HIDDEN; Sun, 13 Oct 2024 00:56:25 -0400
Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124])
by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256)
(Exim 4.90_1) (envelope-from <yamato@HIDDEN>) id 1szqeV-0000aF-36
for bug-coreutils@HIDDEN; Sun, 13 Oct 2024 00:56:24 -0400
DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com;
s=mimecast20190719; t=1728795381;
h=from:from:reply-to:subject:subject:date:date:message-id:message-id:
to:to:cc:cc:mime-version:mime-version:content-type:content-type:
content-transfer-encoding:content-transfer-encoding;
bh=z4vt6IRHCoCjKvN1mVBNFgFCiUI0C2x4gFhSZbIi6FA=;
b=LVfM4JZlH2pYEZlMSkowD54mWRl1MZRUJEwQPrV933YBO14e5xeQe4FJ3r6Agr13XoiY8B
YbziJXaE9KFx4XSMD/y+zf2RKWfoYtmSy1kSCKm2hdQAW224iyIp5piygwRpmOmg/qOOpr
yuup1F12y6foabC9qL9ERXL8PB8O0Ec=
Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com
(ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by
relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3,
cipher=TLS_AES_256_GCM_SHA384) id us-mta-626-8wsuHBTSOuSlU1U4L6YVag-1; Sun,
13 Oct 2024 00:56:20 -0400
X-MC-Unique: 8wsuHBTSOuSlU1U4L6YVag-1
Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com
(mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15])
(using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits)
key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256)
(No client certificate requested)
by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS
id 29F5419560A7
for <bug-coreutils@HIDDEN>; Sun, 13 Oct 2024 04:56:19 +0000 (UTC)
Received: from dev64.com (unknown [10.64.240.9])
by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP
id C07881956089; Sun, 13 Oct 2024 04:56:17 +0000 (UTC)
From: Masatake YAMATO <yamato@HIDDEN>
To: bug-coreutils@HIDDEN
Subject: [PATCH] cp: new option --nocache-source
Date: Sun, 13 Oct 2024 13:56:11 +0900
Message-ID: <20241013045611.2572765-1-yamato@HIDDEN>
MIME-Version: 1.0
X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15
X-Mimecast-Spam-Score: 0
X-Mimecast-Originator: redhat.com
Content-Transfer-Encoding: 8bit
Content-Type: text/plain; charset="US-ASCII"; x-default=true
Received-SPF: pass client-ip=170.10.133.124; envelope-from=yamato@HIDDEN;
helo=us-smtp-delivery-124.mimecast.com
X-Spam_score_int: -21
X-Spam_score: -2.2
X-Spam_bar: --
X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.151,
DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1,
RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001,
SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: -1.4 (-)
X-Debbugs-Envelope-To: submit
Cc: yamato@HIDDEN
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -2.4 (--)
When copying files, the system data cache are consumed, the system
data cache is utilized for both the source and destination files. In
scenarios such as creating backup files for old, unused files, it is
clear to users that these files will not be needed in the near
future. In such cases, retaining the data for these files in the cache
constitutes a waste of computer resources, especially when running
applications that require significant memory in the foreground.
With the new option, users will have the ability to request the
discarding of the system data cache, thereby avoiding the unwanted
swapping out of data from foreground processes.
I evaluated cache consumption using a script called
run.bash. Initially, run.bash creates many small files, each 8 KB in
size. It then copies these files using the cp command, both with and
without the specified option. Finally, it reports the difference in
the total size of the caches before and after the copying process.
run.bash:
#!/bin/bash
CP=$1
shift
[[ -e "$CP" ]] || {
echo "no file found: $CP" 1>&2
exit 1
}
N=8
S=drop-src
D=${HOME}/drop-dst
mkdir -p $S
mkdir -p $D
start=
end=
print_cached()
{
grep ^Cached: /proc/meminfo
}
start()
{
start=$(print_cached | awk '{print $2}')
}
end()
{
end=$(print_cached | awk '{print $2}')
}
report()
{
echo -n "delta[$N:$1/$2]: "
expr "$end" - "$start"
}
cleanup()
{
local i
local j
for ((i = 0; i < 10; i++)); do
for ((j = 0; j < 10; j++)); do
rm -f $S/F-${i}${j}*
rm -f $D/F-${i}${j}*
done
done
rm -f $S/F-*
rm -f $D/F-*
}
prep()
{
local i
for ((i = 0; i < 1024 * $N; i++ )); do
if ! dd if=/dev/zero of=$S/F-$i bs=4096 count=2 \
status=none; then
echo "failed in dd of=$S/F-$F" 1>&2
exit 1
fi
done
sync
}
run_cp()
{
start
local i
time for ((i = 0; i < 1024 * $N; i++ )); do
if ! "${CP}" "$@" "$S/F-$i" "$D/F-$i"; then
echo "failed in cp " "$@" "$S/F-$i" " $D/F-$i" 1>&2
exit 1
fi
done
end
report "$1" $2
}
cleanup
sync
prep
run_cp "$@"
running:
~/coreutils/nocache$ ./run.bash ../src/cp
real 0m16.051s
user 0m4.249s
sys 0m12.437s
delta[8:/]: 65548
~/coreutils/nocache$ ./run.bash ../src/cp --nocache-source
real 0m17.109s
user 0m4.492s
sys 0m13.317s
delta[8:--nocache-source/]: 620
--nocache-source option suppresses the consumption of the cache
massively.
* src/copy.h (struct cp_options): New member 'src_nocache'.
* src/copy.c (copy_reg): Call 'fdadvise (..., FADVISE_DONTNEED)'
if the new member is true.
* src/cp.c (NOCACHE_SOURCE_OPTION):
New enumerator.
(long_opts): Add the option.
(usage): Document the option.
(main): Support the option.
* doc/coreutils.texi (mv invocation): Document the option.
* NEWS: Mention the option.
---
NEWS | 3 +++
doc/coreutils.texi | 4 ++++
src/copy.c | 2 ++
src/copy.h | 3 +++
src/cp.c | 13 +++++++++++++
5 files changed, 25 insertions(+)
diff --git a/NEWS b/NEWS
index 8fea2657b..483c79e91 100644
--- a/NEWS
+++ b/NEWS
@@ -46,6 +46,9 @@ GNU coreutils NEWS -*- outline -*-
%<i>$ format, where '<i>' is an integer referencing a particular argument,
thus allowing repetition or reordering of printf arguments.
+ cp now accepts --nocache-source option, which causes cp to request to discard
+ the system data cache for the source files.
+
** Improvements
'head -c NUM', 'head -n NUM', 'nl -l NUM', 'nproc --ignore NUM',
diff --git a/doc/coreutils.texi b/doc/coreutils.texi
index d5dd55092..e5bdc70ff 100644
--- a/doc/coreutils.texi
+++ b/doc/coreutils.texi
@@ -10305,6 +10305,10 @@ skip existing files but not fail.
If a file cannot be renamed because the destination file system differs,
fail with a diagnostic instead of copying and then removing the file.
+@item --nocache-source
+@opindex --nocache-source
+Request to discard the system data cache for the source files.
+
@item --exchange
@opindex --exchange
Exchange source and destination instead of renaming source to destination.
diff --git a/src/copy.c b/src/copy.c
index b1ac52c79..1c2ed367d 100644
--- a/src/copy.c
+++ b/src/copy.c
@@ -1705,6 +1705,8 @@ close_src_and_dst_desc:
return_val = false;
}
close_src_desc:
+ if (x->src_nocache)
+ fdadvise (source_desc, 0, 0, FADVISE_DONTNEED);
if (close (source_desc) < 0)
{
error (0, errno, _("failed to close %s"), quoteaf (src_name));
diff --git a/src/copy.h b/src/copy.h
index ab89c75fd..1042ecf7b 100644
--- a/src/copy.h
+++ b/src/copy.h
@@ -302,6 +302,9 @@ struct cp_options
/* FIXME */
Hash_table *src_info;
+
+ /* Request to discard cache for the source files. */
+ bool src_nocache;
};
bool copy (char const *src_name, char const *dst_name,
diff --git a/src/cp.c b/src/cp.c
index 127b5603f..10d66fd80 100644
--- a/src/cp.c
+++ b/src/cp.c
@@ -63,6 +63,8 @@ enum
COPY_CONTENTS_OPTION,
DEBUG_OPTION,
NO_PRESERVE_ATTRIBUTES_OPTION,
+ NOCACHE_DESTINATION_OPTION,
+ NOCACHE_SOURCE_OPTION,
PARENTS_OPTION,
PRESERVE_ATTRIBUTES_OPTION,
REFLINK_OPTION,
@@ -127,6 +129,8 @@ static struct option const long_opts[] =
{"no-dereference", no_argument, nullptr, 'P'},
{"no-preserve", required_argument, nullptr, NO_PRESERVE_ATTRIBUTES_OPTION},
{"no-target-directory", no_argument, nullptr, 'T'},
+ {"nocache-destination", no_argument, nullptr, NOCACHE_DESTINATION_OPTION},
+ {"nocache-source", no_argument, nullptr, NOCACHE_SOURCE_OPTION},
{"one-file-system", no_argument, nullptr, 'x'},
{"parents", no_argument, nullptr, PARENTS_OPTION},
{"path", no_argument, nullptr, PARENTS_OPTION}, /* Deprecated. */
@@ -197,6 +201,9 @@ Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.\n\
fputs (_("\
-n, --no-clobber (deprecated) silently skip existing files.\n\
See also --update\n\
+"), stdout);
+ fputs (_("\
+ --nocache-source request to discard the cache for the source files.\n\
"), stdout);
fputs (_("\
-P, --no-dereference never follow symbolic links in SOURCE\n\
@@ -875,6 +882,8 @@ cp_option_init (struct cp_options *x)
x->dest_info = nullptr;
x->src_info = nullptr;
+
+ x->src_nocache = false;
}
/* Given a string, ARG, containing a comma-separated list of arguments
@@ -1088,6 +1097,10 @@ main (int argc, char **argv)
decode_preserve_arg (optarg, &x, false);
break;
+ case NOCACHE_SOURCE_OPTION:
+ x.src_nocache = true;
+ break;
+
case PRESERVE_ATTRIBUTES_OPTION:
if (optarg == nullptr)
{
--
2.46.2
Masatake YAMATO <yamato@HIDDEN>:bug-coreutils@HIDDEN.
Full text available.bug-coreutils@HIDDEN:bug#73784; Package coreutils.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.