X-Loop: help-debbugs@HIDDEN Subject: bug#73784: [PATCH] cp: new option --nocache-source Resent-From: Masatake YAMATO <yamato@HIDDEN> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-coreutils@HIDDEN Resent-Date: Sun, 13 Oct 2024 04:57:02 +0000 Resent-Message-ID: <handler.73784.B.172879540429074 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: report 73784 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: 73784 <at> debbugs.gnu.org Cc: yamato@HIDDEN X-Debbugs-Original-To: bug-coreutils@HIDDEN Received: via spool by submit <at> debbugs.gnu.org id=B.172879540429074 (code B ref -1); Sun, 13 Oct 2024 04:57:02 +0000 Received: (at submit) by debbugs.gnu.org; 13 Oct 2024 04:56:44 +0000 Received: from localhost ([127.0.0.1]:49864 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1szqep-0007Yp-6d for submit <at> debbugs.gnu.org; Sun, 13 Oct 2024 00:56:43 -0400 Received: from lists.gnu.org ([209.51.188.17]:47082) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <yamato@HIDDEN>) id 1szqem-0007Yh-Fg for submit <at> debbugs.gnu.org; Sun, 13 Oct 2024 00:56:41 -0400 Received: from eggs.gnu.org ([2001:470:142:3::10]) by lists.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <yamato@HIDDEN>) id 1szqeX-0004yf-4r for bug-coreutils@HIDDEN; Sun, 13 Oct 2024 00:56:25 -0400 Received: from us-smtp-delivery-124.mimecast.com ([170.10.133.124]) by eggs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.90_1) (envelope-from <yamato@HIDDEN>) id 1szqeV-0000aF-36 for bug-coreutils@HIDDEN; Sun, 13 Oct 2024 00:56:24 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1728795381; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding; bh=z4vt6IRHCoCjKvN1mVBNFgFCiUI0C2x4gFhSZbIi6FA=; b=LVfM4JZlH2pYEZlMSkowD54mWRl1MZRUJEwQPrV933YBO14e5xeQe4FJ3r6Agr13XoiY8B YbziJXaE9KFx4XSMD/y+zf2RKWfoYtmSy1kSCKm2hdQAW224iyIp5piygwRpmOmg/qOOpr yuup1F12y6foabC9qL9ERXL8PB8O0Ec= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-626-8wsuHBTSOuSlU1U4L6YVag-1; Sun, 13 Oct 2024 00:56:20 -0400 X-MC-Unique: 8wsuHBTSOuSlU1U4L6YVag-1 Received: from mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.15]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 29F5419560A7 for <bug-coreutils@HIDDEN>; Sun, 13 Oct 2024 04:56:19 +0000 (UTC) Received: from dev64.com (unknown [10.64.240.9]) by mx-prod-int-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id C07881956089; Sun, 13 Oct 2024 04:56:17 +0000 (UTC) From: Masatake YAMATO <yamato@HIDDEN> Date: Sun, 13 Oct 2024 13:56:11 +0900 Message-ID: <20241013045611.2572765-1-yamato@HIDDEN> MIME-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.15 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Transfer-Encoding: 8bit Content-Type: text/plain; charset="US-ASCII"; x-default=true Received-SPF: pass client-ip=170.10.133.124; envelope-from=yamato@HIDDEN; helo=us-smtp-delivery-124.mimecast.com X-Spam_score_int: -21 X-Spam_score: -2.2 X-Spam_bar: -- X-Spam_report: (-2.2 / 5.0 requ) BAYES_00=-1.9, DKIMWL_WL_HIGH=-0.151, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no X-Spam_action: no action X-Spam-Score: -1.4 (-) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -2.4 (--) When copying files, the system data cache are consumed, the system data cache is utilized for both the source and destination files. In scenarios such as creating backup files for old, unused files, it is clear to users that these files will not be needed in the near future. In such cases, retaining the data for these files in the cache constitutes a waste of computer resources, especially when running applications that require significant memory in the foreground. With the new option, users will have the ability to request the discarding of the system data cache, thereby avoiding the unwanted swapping out of data from foreground processes. I evaluated cache consumption using a script called run.bash. Initially, run.bash creates many small files, each 8 KB in size. It then copies these files using the cp command, both with and without the specified option. Finally, it reports the difference in the total size of the caches before and after the copying process. run.bash: #!/bin/bash CP=$1 shift [[ -e "$CP" ]] || { echo "no file found: $CP" 1>&2 exit 1 } N=8 S=drop-src D=${HOME}/drop-dst mkdir -p $S mkdir -p $D start= end= print_cached() { grep ^Cached: /proc/meminfo } start() { start=$(print_cached | awk '{print $2}') } end() { end=$(print_cached | awk '{print $2}') } report() { echo -n "delta[$N:$1/$2]: " expr "$end" - "$start" } cleanup() { local i local j for ((i = 0; i < 10; i++)); do for ((j = 0; j < 10; j++)); do rm -f $S/F-${i}${j}* rm -f $D/F-${i}${j}* done done rm -f $S/F-* rm -f $D/F-* } prep() { local i for ((i = 0; i < 1024 * $N; i++ )); do if ! dd if=/dev/zero of=$S/F-$i bs=4096 count=2 \ status=none; then echo "failed in dd of=$S/F-$F" 1>&2 exit 1 fi done sync } run_cp() { start local i time for ((i = 0; i < 1024 * $N; i++ )); do if ! "${CP}" "$@" "$S/F-$i" "$D/F-$i"; then echo "failed in cp " "$@" "$S/F-$i" " $D/F-$i" 1>&2 exit 1 fi done end report "$1" $2 } cleanup sync prep run_cp "$@" running: ~/coreutils/nocache$ ./run.bash ../src/cp real 0m16.051s user 0m4.249s sys 0m12.437s delta[8:/]: 65548 ~/coreutils/nocache$ ./run.bash ../src/cp --nocache-source real 0m17.109s user 0m4.492s sys 0m13.317s delta[8:--nocache-source/]: 620 --nocache-source option suppresses the consumption of the cache massively. * src/copy.h (struct cp_options): New member 'src_nocache'. * src/copy.c (copy_reg): Call 'fdadvise (..., FADVISE_DONTNEED)' if the new member is true. * src/cp.c (NOCACHE_SOURCE_OPTION): New enumerator. (long_opts): Add the option. (usage): Document the option. (main): Support the option. * doc/coreutils.texi (mv invocation): Document the option. * NEWS: Mention the option. --- NEWS | 3 +++ doc/coreutils.texi | 4 ++++ src/copy.c | 2 ++ src/copy.h | 3 +++ src/cp.c | 13 +++++++++++++ 5 files changed, 25 insertions(+) diff --git a/NEWS b/NEWS index 8fea2657b..483c79e91 100644 --- a/NEWS +++ b/NEWS @@ -46,6 +46,9 @@ GNU coreutils NEWS -*- outline -*- %<i>$ format, where '<i>' is an integer referencing a particular argument, thus allowing repetition or reordering of printf arguments. + cp now accepts --nocache-source option, which causes cp to request to discard + the system data cache for the source files. + ** Improvements 'head -c NUM', 'head -n NUM', 'nl -l NUM', 'nproc --ignore NUM', diff --git a/doc/coreutils.texi b/doc/coreutils.texi index d5dd55092..e5bdc70ff 100644 --- a/doc/coreutils.texi +++ b/doc/coreutils.texi @@ -10305,6 +10305,10 @@ skip existing files but not fail. If a file cannot be renamed because the destination file system differs, fail with a diagnostic instead of copying and then removing the file. +@item --nocache-source +@opindex --nocache-source +Request to discard the system data cache for the source files. + @item --exchange @opindex --exchange Exchange source and destination instead of renaming source to destination. diff --git a/src/copy.c b/src/copy.c index b1ac52c79..1c2ed367d 100644 --- a/src/copy.c +++ b/src/copy.c @@ -1705,6 +1705,8 @@ close_src_and_dst_desc: return_val = false; } close_src_desc: + if (x->src_nocache) + fdadvise (source_desc, 0, 0, FADVISE_DONTNEED); if (close (source_desc) < 0) { error (0, errno, _("failed to close %s"), quoteaf (src_name)); diff --git a/src/copy.h b/src/copy.h index ab89c75fd..1042ecf7b 100644 --- a/src/copy.h +++ b/src/copy.h @@ -302,6 +302,9 @@ struct cp_options /* FIXME */ Hash_table *src_info; + + /* Request to discard cache for the source files. */ + bool src_nocache; }; bool copy (char const *src_name, char const *dst_name, diff --git a/src/cp.c b/src/cp.c index 127b5603f..10d66fd80 100644 --- a/src/cp.c +++ b/src/cp.c @@ -63,6 +63,8 @@ enum COPY_CONTENTS_OPTION, DEBUG_OPTION, NO_PRESERVE_ATTRIBUTES_OPTION, + NOCACHE_DESTINATION_OPTION, + NOCACHE_SOURCE_OPTION, PARENTS_OPTION, PRESERVE_ATTRIBUTES_OPTION, REFLINK_OPTION, @@ -127,6 +129,8 @@ static struct option const long_opts[] = {"no-dereference", no_argument, nullptr, 'P'}, {"no-preserve", required_argument, nullptr, NO_PRESERVE_ATTRIBUTES_OPTION}, {"no-target-directory", no_argument, nullptr, 'T'}, + {"nocache-destination", no_argument, nullptr, NOCACHE_DESTINATION_OPTION}, + {"nocache-source", no_argument, nullptr, NOCACHE_SOURCE_OPTION}, {"one-file-system", no_argument, nullptr, 'x'}, {"parents", no_argument, nullptr, PARENTS_OPTION}, {"path", no_argument, nullptr, PARENTS_OPTION}, /* Deprecated. */ @@ -197,6 +201,9 @@ Copy SOURCE to DEST, or multiple SOURCE(s) to DIRECTORY.\n\ fputs (_("\ -n, --no-clobber (deprecated) silently skip existing files.\n\ See also --update\n\ +"), stdout); + fputs (_("\ + --nocache-source request to discard the cache for the source files.\n\ "), stdout); fputs (_("\ -P, --no-dereference never follow symbolic links in SOURCE\n\ @@ -875,6 +882,8 @@ cp_option_init (struct cp_options *x) x->dest_info = nullptr; x->src_info = nullptr; + + x->src_nocache = false; } /* Given a string, ARG, containing a comma-separated list of arguments @@ -1088,6 +1097,10 @@ main (int argc, char **argv) decode_preserve_arg (optarg, &x, false); break; + case NOCACHE_SOURCE_OPTION: + x.src_nocache = true; + break; + case PRESERVE_ATTRIBUTES_OPTION: if (optarg == nullptr) { -- 2.46.2
Content-Disposition: inline Content-Transfer-Encoding: quoted-printable MIME-Version: 1.0 X-Mailer: MIME-tools 5.505 (Entity 5.505) Content-Type: text/plain; charset=utf-8 X-Loop: help-debbugs@HIDDEN From: help-debbugs@HIDDEN (GNU bug Tracking System) To: Masatake YAMATO <yamato@HIDDEN> Subject: bug#73784: Acknowledgement ([PATCH] cp: new option --nocache-source) Message-ID: <handler.73784.B.172879540429074.ack <at> debbugs.gnu.org> References: <20241013045611.2572765-1-yamato@HIDDEN> X-Gnu-PR-Message: ack 73784 X-Gnu-PR-Package: coreutils X-Gnu-PR-Keywords: patch Reply-To: 73784 <at> debbugs.gnu.org Date: Sun, 13 Oct 2024 04:57:02 +0000 Thank you for filing a new bug report with debbugs.gnu.org. This is an automatically generated reply to let you know your message has been received. Your message is being forwarded to the package maintainers and other interested parties for their attention; they will reply in due course. Your message has been sent to the package maintainer(s): bug-coreutils@HIDDEN If you wish to submit further information on this problem, please send it to 73784 <at> debbugs.gnu.org. Please do not send mail to help-debbugs@HIDDEN unless you wish to report a problem with the Bug-tracking system. --=20 73784: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D73784 GNU Bug Tracking System Contact help-debbugs@HIDDEN with problems
X-Loop: help-debbugs@HIDDEN Subject: bug#73784: [PATCH] cp: new option --nocache-source Resent-From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-coreutils@HIDDEN Resent-Date: Sun, 13 Oct 2024 15:01:01 +0000 Resent-Message-ID: <handler.73784.B73784.172883164929960 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: followup 73784 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: Masatake YAMATO <yamato@HIDDEN>, 73784 <at> debbugs.gnu.org Received: via spool by 73784-submit <at> debbugs.gnu.org id=B73784.172883164929960 (code B ref 73784); Sun, 13 Oct 2024 15:01:01 +0000 Received: (at 73784) by debbugs.gnu.org; 13 Oct 2024 15:00:49 +0000 Received: from localhost ([127.0.0.1]:52056 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1t005Q-0007nA-UO for submit <at> debbugs.gnu.org; Sun, 13 Oct 2024 11:00:49 -0400 Received: from mail-wm1-f46.google.com ([209.85.128.46]:47394) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <pixelbeat@HIDDEN>) id 1t005O-0007mz-EY for 73784 <at> debbugs.gnu.org; Sun, 13 Oct 2024 11:00:47 -0400 Received: by mail-wm1-f46.google.com with SMTP id 5b1f17b1804b1-4311e470fdaso20198035e9.3 for <73784 <at> debbugs.gnu.org>; Sun, 13 Oct 2024 08:00:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1728831570; x=1729436370; darn=debbugs.gnu.org; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id :sender:from:to:cc:subject:date:message-id:reply-to; bh=cux9yVtmYgMWxuT/XMWiv+jmZiy3n1EJaY+NoT0mxX4=; b=SNo1PhWckh1EYSFWJuxDc7Lrlm2s/G7Nh3v3SdPdrrJCl81YUjeOYfeiE5ucUMZZpu rESRTMa6385QPosY19DtonEIGvrk0QoFm/1b8yoiStIbRioFERBApX7WbC8OIjLuwrjb MilxCoV48tdJzrkr5wMlOudAJUEvvrE9GfWPHm07gWAmGOkI3tk1aMEHwhh8rJ22BfJ2 xok2DDz0QcySn06YHOrpQfnYrEU6bndDxfVRTPkuJvOE4e4VIwajio/7NF5AcIFYzLbw bEMrP1WfWkMjIKRvYkJI10yFoJDW3kb2ORw+Oy1fPNc2oHSmABYgq01T6yMSJkRuQjYe 4XSw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1728831570; x=1729436370; h=content-transfer-encoding:in-reply-to:from:content-language :references:to:subject:user-agent:mime-version:date:message-id :sender:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=cux9yVtmYgMWxuT/XMWiv+jmZiy3n1EJaY+NoT0mxX4=; b=G8ST8JBm6syPcKFufloa3sLmcoMs2f5MAC6d2qqJwYiNlMDCnWVeaU3dCgZY+1Xl9p llQiaGAsecRYk0xLMBcr7WgI1JRMN37fnlbRBVdR+M8OBX0q+ayV6nV2ba40YW2JIPJ4 oQ/0rqZ0V5gQgklVFYEeVeobtz2KuocE63/2t14o61BX47uhUvsk+8yq1uQcjetfoMJs mTpmRVDRkelqabQ6PPBRLOKKA5J+yXbwTif+t7ML2EJ5sOzDD/sW8+z0DP8DQ7ZUSgYP A0zSVGwyP2Sx4FyRCnMLuAOZ8lxW2wlKlVxS5F0HtQ2CMUFANtAqonpdHCglnhA/jeP2 DIMQ== X-Forwarded-Encrypted: i=1; AJvYcCX+ameheG7DIqJ54XqFj3rxqYHWnJTde3fnTgS1xYlH+Me0CHK7wYoIaiqp683EnIpBOWbf1Q==@debbugs.gnu.org X-Gm-Message-State: AOJu0YyRPui01BXrJR9HdyiDJzPChddM9AeFDtvti6d6qaCDlMHbg/Kg y4D2bKCEry+i+dHfmwMdewB2Y+HIpKUrBZ0ApiZXjpcDbztwpI00 X-Google-Smtp-Source: AGHT+IFss1bEYk6z2SfXrT+BFR0iQIZraqhWJq/6U5ey3gqyBt1GsSnGcz+RGuWCPZItLj9B8ubm6w== X-Received: by 2002:a05:600c:190d:b0:430:53f6:f20b with SMTP id 5b1f17b1804b1-4311df56f1amr78940785e9.23.1728831569816; Sun, 13 Oct 2024 07:59:29 -0700 (PDT) Received: from [192.168.1.156] (86-44-211-146-dynamic.agg2.lod.rsl-rtd.eircom.net. [86.44.211.146]) by smtp.googlemail.com with ESMTPSA id 5b1f17b1804b1-430ccf1f797sm128401055e9.4.2024.10.13.07.59.28 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Sun, 13 Oct 2024 07:59:28 -0700 (PDT) Message-ID: <fcd92e09-761c-44c3-a0e4-475142c41be9@HIDDEN> Date: Sun, 13 Oct 2024 15:59:27 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Beta References: <20241013045611.2572765-1-yamato@HIDDEN> Content-Language: en-US From: =?UTF-8?Q?P=C3=A1draig?= Brady <P@HIDDEN> In-Reply-To: <20241013045611.2572765-1-yamato@HIDDEN> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Spam-Score: 0.3 (/) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -0.7 (/) On 13/10/2024 05:56, Masatake YAMATO wrote: > When copying files, the system data cache are consumed, the system > data cache is utilized for both the source and destination files. In > scenarios such as creating backup files for old, unused files, it is > clear to users that these files will not be needed in the near > future. In such cases, retaining the data for these files in the cache > constitutes a waste of computer resources, especially when running > applications that require significant memory in the foreground. > > With the new option, users will have the ability to request the > discarding of the system data cache, thereby avoiding the unwanted > swapping out of data from foreground processes. > > I evaluated cache consumption using a script called > run.bash. Initially, run.bash creates many small files, each 8 KB in > size. It then copies these files using the cp command, both with and > without the specified option. Finally, it reports the difference in > the total size of the caches before and after the copying process. > > run.bash: > > #!/bin/bash > CP=$1 > shift > > [[ -e "$CP" ]] || { > echo "no file found: $CP" 1>&2 > exit 1 > } > > N=8 > S=drop-src > D=${HOME}/drop-dst > > mkdir -p $S > mkdir -p $D > > start= > end= > print_cached() > { > grep ^Cached: /proc/meminfo > } > > start() > { > start=$(print_cached | awk '{print $2}') > } > > end() > { > end=$(print_cached | awk '{print $2}') > } > > report() > { > echo -n "delta[$N:$1/$2]: " > expr "$end" - "$start" > } > > cleanup() > { > local i > local j > for ((i = 0; i < 10; i++)); do > for ((j = 0; j < 10; j++)); do > rm -f $S/F-${i}${j}* > rm -f $D/F-${i}${j}* > done > done > rm -f $S/F-* > rm -f $D/F-* > } > > prep() > { > local i > for ((i = 0; i < 1024 * $N; i++ )); do > if ! dd if=/dev/zero of=$S/F-$i bs=4096 count=2 \ > status=none; then > echo "failed in dd of=$S/F-$F" 1>&2 > exit 1 > fi > done > sync > } > > run_cp() > { > start > > local i > time for ((i = 0; i < 1024 * $N; i++ )); do > if ! "${CP}" "$@" "$S/F-$i" "$D/F-$i"; then > echo "failed in cp " "$@" "$S/F-$i" " $D/F-$i" 1>&2 > exit 1 > fi > done > > end > report "$1" $2 > } > > cleanup > sync > > prep > run_cp "$@" > > running: > > ~/coreutils/nocache$ ./run.bash ../src/cp > > real 0m16.051s > user 0m4.249s > sys 0m12.437s > delta[8:/]: 65548 > ~/coreutils/nocache$ ./run.bash ../src/cp --nocache-source > > real 0m17.109s > user 0m4.492s > sys 0m13.317s > delta[8:--nocache-source/]: 620 > > --nocache-source option suppresses the consumption of the cache > massively. Thanks for the patch. I have some reservations/notes though... There is nothing particularly special about cp, that it might need this option. I.e. it would be nice to be able to wrap any program so that it streamed data through the cache, rather than aggressively cached. I'm not sure how to do that, but also I'd be reluctant to start adding such options to individual commands though. Perhaps Linux' open() may gain an O_STREAM flag in future that might be more generally applied with a wrapper or something. For single (large) files, one already has this functionality in dd. On the write side, you'd also have to worry about syncing, to make the drop cache advisory effective, and this could impact performance. Might this drop caches for already cached files, which cp may just happen to be copying, thus potentially impacting performance for other programs. If reflinking we probably would not want to do this operation, since we're not reading the source. thanks, Pádraig
X-Loop: help-debbugs@HIDDEN Subject: bug#73784: [PATCH] cp: new option --nocache-source Resent-From: Masatake YAMATO <yamato@HIDDEN> Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> Resent-CC: bug-coreutils@HIDDEN Resent-Date: Thu, 02 Jan 2025 19:06:03 +0000 Resent-Message-ID: <handler.73784.B73784.17358447486668 <at> debbugs.gnu.org> Resent-Sender: help-debbugs@HIDDEN X-GNU-PR-Message: followup 73784 X-GNU-PR-Package: coreutils X-GNU-PR-Keywords: patch To: P@HIDDEN Cc: 73784 <at> debbugs.gnu.org Received: via spool by 73784-submit <at> debbugs.gnu.org id=B73784.17358447486668 (code B ref 73784); Thu, 02 Jan 2025 19:06:03 +0000 Received: (at 73784) by debbugs.gnu.org; 2 Jan 2025 19:05:48 +0000 Received: from localhost ([127.0.0.1]:46690 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1tTQVv-0001jT-Ug for submit <at> debbugs.gnu.org; Thu, 02 Jan 2025 14:05:48 -0500 Received: from us-smtp-delivery-124.mimecast.com ([170.10.129.124]:29807) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <yamato@HIDDEN>) id 1tTQVt-0001jG-4e for 73784 <at> debbugs.gnu.org; Thu, 02 Jan 2025 14:05:46 -0500 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1735844744; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=1SJXAyhYfQBu0Evjcygeb87TtYvbbM/HMPrAi4gSh1c=; b=ASKoglQcf/Hd7OHCS1iIPNbdGL1iUgxX0rm3LLBcEb96hgQNPGGrXT6YqwxCH442MEX+7V VXSRdaBSQ7l3UkiSlJxYBD1hdX9vj9fNqRZbYoDqn1grFX1O7LHwNotmSVsQe1u6tRpUu9 KT6qeEZyol1Sk9+u02Xuf4n7R5ahk/c= Received: from mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (ec2-54-186-198-63.us-west-2.compute.amazonaws.com [54.186.198.63]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-35-RDfAMBdTMee1NZKJ5qCbhw-1; Thu, 02 Jan 2025 14:05:41 -0500 X-MC-Unique: RDfAMBdTMee1NZKJ5qCbhw-1 X-Mimecast-MFC-AGG-ID: RDfAMBdTMee1NZKJ5qCbhw Received: from mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com [10.30.177.17]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mx-prod-mc-02.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTPS id 006821956087; Thu, 2 Jan 2025 19:05:40 +0000 (UTC) Received: from localhost (unknown [10.64.240.119]) by mx-prod-int-05.mail-002.prod.us-west-2.aws.redhat.com (Postfix) with ESMTP id ACEB11956052; Thu, 2 Jan 2025 19:05:37 +0000 (UTC) Date: Fri, 03 Jan 2025 04:05:35 +0900 (JST) Message-Id: <20250103.040535.1761642872805390481.yamato@HIDDEN> From: Masatake YAMATO <yamato@HIDDEN> In-Reply-To: <fcd92e09-761c-44c3-a0e4-475142c41be9@HIDDEN> References: <20241013045611.2572765-1-yamato@HIDDEN> <fcd92e09-761c-44c3-a0e4-475142c41be9@HIDDEN> Organization: Red Hat Japan, K.K. Mime-Version: 1.0 X-Scanned-By: MIMEDefang 3.0 on 10.30.177.17 X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: sJT11KwScrdFQLIlb_2yneJMXvp50Z3EzCYvU1COgpA_1735844740 X-Mimecast-Originator: redhat.com Content-Type: Text/Plain; charset=iso-8859-1 Content-Transfer-Encoding: quoted-printable X-Spam-Score: 0.0 (/) X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) From: P=E1draig Brady <P@HIDDEN> Subject: Re: bug#73784: [PATCH] cp: new option --nocache-source Date: Sun, 13 Oct 2024 15:59:27 +0100 > On 13/10/2024 05:56, Masatake YAMATO wrote: >> When copying files, the system data cache are consumed, the system >> data cache is utilized for both the source and destination files. In >> scenarios such as creating backup files for old, unused files, it is >> clear to users that these files will not be needed in the near >> future. In such cases, retaining the data for these files in the cache >> constitutes a waste of computer resources, especially when running >> applications that require significant memory in the foreground. >> With the new option, users will have the ability to request the >> discarding of the system data cache, thereby avoiding the unwanted >> swapping out of data from foreground processes. >> I evaluated cache consumption using a script called >> run.bash. Initially, run.bash creates many small files, each 8 KB in >> size. It then copies these files using the cp command, both with and >> without the specified option. Finally, it reports the difference in >> the total size of the caches before and after the copying process. >> run.bash: >> #!/bin/bash >> CP=3D$1 >> shift >> [[ -e "$CP" ]] || { >> =09echo "no file found: $CP" 1>&2 >> =09exit 1 >> } >> N=3D8 >> S=3Ddrop-src >> D=3D${HOME}/drop-dst >> mkdir -p $S >> mkdir -p $D >> start=3D >> end=3D >> print_cached() >> { >> =09grep ^Cached: /proc/meminfo >> } >> start() >> { >> =09start=3D$(print_cached | awk '{print $2}') >> } >> end() >> { >> =09end=3D$(print_cached | awk '{print $2}') >> } >> report() >> { >> =09echo -n "delta[$N:$1/$2]: " >> =09expr "$end" - "$start" >> } >> cleanup() >> { >> =09local i >> =09local j >> =09for ((i =3D 0; i < 10; i++)); do >> =09 for ((j =3D 0; j < 10; j++)); do >> =09=09rm -f $S/F-${i}${j}* >> =09=09rm -f $D/F-${i}${j}* >> =09 done >> =09done >> =09rm -f $S/F-* >> =09rm -f $D/F-* >> } >> prep() >> { >> =09local i >> =09for ((i =3D 0; i < 1024 * $N; i++ )); do >> =09 if ! dd if=3D/dev/zero of=3D$S/F-$i bs=3D4096 count=3D2 \ >> =09 status=3Dnone; then >> =09=09echo "failed in dd of=3D$S/F-$F" 1>&2 >> =09=09exit 1 >> =09 fi >> =09done >> =09sync >> } >> run_cp() >> { >> =09start >> =09local i >> =09time for ((i =3D 0; i < 1024 * $N; i++ )); do >> =09 if ! "${CP}" "$@" "$S/F-$i" "$D/F-$i"; then >> =09=09echo "failed in cp " "$@" "$S/F-$i" " $D/F-$i" 1>&2 >> =09=09exit 1 >> =09 fi >> =09done >> =09end >> =09report "$1" $2 >> } >> cleanup >> sync >> prep >> run_cp "$@" >> running: >> ~/coreutils/nocache$ ./run.bash ../src/cp >> real=090m16.051s >> user=090m4.249s >> sys=090m12.437s >> delta[8:/]: 65548 >> ~/coreutils/nocache$ ./run.bash ../src/cp --nocache-source >> real=090m17.109s >> user=090m4.492s >> sys=090m13.317s >> delta[8:--nocache-source/]: 620 >> --nocache-source option suppresses the consumption of the cache >> massively. >=20 > Thanks for the patch. > I have some reservations/notes though... >=20 > There is nothing particularly special about cp, that it might need > this option. > I.e. it would be nice to be able to wrap any program so that it > streamed > data through the cache, rather than aggressively cached. I'm not sure > how to do that, > but also I'd be reluctant to start adding such options to individual > commands though. > Perhaps Linux' open() may gain an O_STREAM flag in future that might > be > more generally applied with a wrapper or something. I found an interesting article: https://www.phoronix.com/news/Uncached-Buff= ered-IO-Linux-6.14 It seems that RWF_DONTCACHE flag of pwritev and preadv implements what we need. When Linux-6.14 is released, I will rewrite my patch based on RWF_DONTCACHE. Masatake YAMATO > For single (large) files, one already has this functionality in dd. >=20 > On the write side, you'd also have to worry about syncing, to make the > drop cache advisory effective, and this could impact performance. >=20 > Might this drop caches for already cached files, > which cp may just happen to be copying, > thus potentially impacting performance for other programs. >=20 > If reflinking we probably would not want to do this operation, > since we're not reading the source. >=20 > thanks, > P=E1draig >
Received: (at control) by debbugs.gnu.org; 16 Feb 2025 06:58:51 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Feb 16 01:58:51 2025 Received: from localhost ([127.0.0.1]:59698 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1tjYc6-0000Xc-Mw for submit <at> debbugs.gnu.org; Sun, 16 Feb 2025 01:58:50 -0500 Received: from mail.cs.ucla.edu ([131.179.128.66]:41272) by debbugs.gnu.org with esmtps (TLS1.2:ECDHE_RSA_AES_256_GCM_SHA384:256) (Exim 4.84_2) (envelope-from <eggert@HIDDEN>) id 1tjYc4-0000XJ-AU for control <at> debbugs.gnu.org; Sun, 16 Feb 2025 01:58:48 -0500 Received: from localhost (localhost [127.0.0.1]) by mail.cs.ucla.edu (Postfix) with ESMTP id B42513C00E415 for <control <at> debbugs.gnu.org>; Sat, 15 Feb 2025 22:58:42 -0800 (PST) Received: from mail.cs.ucla.edu ([127.0.0.1]) by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavis, port 10032) with ESMTP id UNS52wPzlaiO for <control <at> debbugs.gnu.org>; Sat, 15 Feb 2025 22:58:42 -0800 (PST) Received: from localhost (localhost [127.0.0.1]) by mail.cs.ucla.edu (Postfix) with ESMTP id 544723C00E41C for <control <at> debbugs.gnu.org>; Sat, 15 Feb 2025 22:58:42 -0800 (PST) DKIM-Filter: OpenDKIM Filter v2.10.3 mail.cs.ucla.edu 544723C00E41C DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cs.ucla.edu; s=9D0B346E-2AEB-11ED-9476-E14B719DCE6C; t=1739689122; bh=SVQ/gcc45eTT73KzgPWfKuaPFT5aSqflTr9MAI/aVS4=; h=Message-ID:Date:MIME-Version:To:From; b=JrHvtUHs1RtuxZXmG+lLRj13bunbeqow9OFro+5oQ674s/0BCvwQYfYgcCW2XGicG dSFFQGOAFFbejXGkXm+HWSOSbV4DV38Bbs9pORQXY/L+Hd0LOs65r7OdXmHZzd7e21 23rceNiZYqeYumYcpdmFkuGGUW9Zu5tUW2WL8FWabXYRNyPzqF+HDiu6l307k91U7W OL0ET+MgmpFbAIT7BbbjHuVM7nzPcwSZRc1jt3hQ35NwJcZeI0yDnTs8YdlPhiFuut ybX4H09imLgTFYBXEurk8qMwBA3+2NuYnEYGjLeaqIIluWPhXIUM8+xDICuwwnKA2R oK3L7NDIeKwug== X-Virus-Scanned: amavis at mail.cs.ucla.edu Received: from mail.cs.ucla.edu ([127.0.0.1]) by localhost (mail.cs.ucla.edu [127.0.0.1]) (amavis, port 10026) with ESMTP id w3IJTT58v7eV for <control <at> debbugs.gnu.org>; Sat, 15 Feb 2025 22:58:42 -0800 (PST) Received: from [192.168.254.12] (unknown [47.147.225.25]) by mail.cs.ucla.edu (Postfix) with ESMTPSA id 3C6763C00E415 for <control <at> debbugs.gnu.org>; Sat, 15 Feb 2025 22:58:42 -0800 (PST) Message-ID: <d3313a51-ea4c-4bd1-ae8b-ef173a233eb9@HIDDEN> Date: Sat, 15 Feb 2025 22:58:42 -0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Content-Language: en-US To: control <at> debbugs.gnu.org From: Paul Eggert <eggert@HIDDEN> Subject: coreutils bug maintenance Organization: UCLA Computer Science Department Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: control X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.0 (-) severity 76060 wishlist severity 75994 wishlist severity 75649 wishlist severity 73510 wishlist severity 73331 wishlist severity 71307 wishlist severity 73784 wishlist tags 73194 notabug close 74194 close 73194 close 73134 close 72756 severity 72024 wishlist tags 71895 moreinfo unreproducible tags 69636 notabug close 69636 severity 69368 wishlist
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.