GNU logs - #77150, boring messages

Message sent to bug-gnu-emacs@HIDDEN:

Subject: bug#77150: pdumper: reduce pdmp size for 64-bit systems
Resent-From: Pip Cet <pipcet@HIDDEN>
Original-Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
Resent-CC: bug-gnu-emacs@HIDDEN
Resent-Date: Fri, 21 Mar 2025 11:43:03 +0000
Resent-Message-ID: <handler.77150.B.17425573342913 <at> debbugs.gnu.org>
Resent-Sender: help-debbugs@HIDDEN
To: 77150 <at> debbugs.gnu.org, dancol@HIDDEN
Date: Fri, 21 Mar 2025 11:41:43 +0000
From: Pip Cet <pipcet@HIDDEN>
Message-ID: <87h63mzk4o.fsf@HIDDEN>
Feedback-ID: 112775352:user:proton
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
Received-SPF: pass client-ip=185.70.40.133; envelope-from=pipcet@HIDDEN;
 helo=mail-40133.protonmail.ch
X-Spam_score_int: -20
X-Spam_score: -2.1
X-Spam_bar: --
X-Spam_report: (-2.1 / 5.0 requ) BAYES_00=-1.9, DKIM_SIGNED=0.1,
 DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_FROM=0.001,
 RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001,
 RCVD_IN_VALIDITY_RPBL_BLOCKED=0.001, RCVD_IN_VALIDITY_SAFE_BLOCKED=0.001,
 SPF_HELO_PASS=-0.001, SPF_PASS=-0.001 autolearn=ham autolearn_force=no
X-Spam_action: no action
X-Spam-Score: 1.0 (+)
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, 
 <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -0.0 (/)

This is a wishlist item (with a PoC patch) for improving pdumper on
64-bit systems.

On such systems, we can save about 2 MB of the pdmp file (reducing it
from 14 MB to 12 MB without nativecomp, or 17 MB to 15 MB with
nativecomp) by storing dump-to-dump relocations for Lisp_Objects and
pointers in the heap image part of the dump, and omitting the
corresponding explicit 32-bit relocations in the relocation part of the
dump.

What makes this possible is that a 64-bit Lisp_Object or pointer is
usually large enough to contain two dump offsets: one to indicate which
object should be referred to after relocation, and another one linking
to the next in-place relocation, allowing us to find all of them by
walking the linked list.

There is no measurable performance difference on my system, though I
suspect that older systems will see performance gains because of the
reduction in pdmp size.  (In theory, it's easier to prefetch relocation
data with the old format, but in practice, using offsets rather than
pointers makes this impossible for GCC to achieve on current CPUs.)

Here's a proof of concept.  Is this worth pursuing?

My TODO list:

1. put the initial pointer into a relocation rather than the header
2. investigate possibility of a 32-bit version
3. compress dump-to-emacs relocations
4. use bitfields rather than direct bit manipulation
5. investigate reversing the chain so we don't go through the heap backward=
s

Note that the code is written to fall back to the old functions if a
relocation doesn't fit, so unusual dumping scenarios should continue to
work, they just won't benefit as much, and there will be fewer conflicts
with other changes to the pdumper code.

In particular, if there is still interest in working on a version of
pdumper which loads the dump at a fixed address and pre-fills the heap
image with the right pointers/Lisp_Objects (unexec-style), there'd be no
conflict: we just couldn't do both at the same time.

Similarly, the disabled DANGEROUS optimization in
dump_field_lv_or_rawptr would reduce the amount of memory saved.  It's
probably time to remove that code.

From 50712c9a349e97a3f9fdd1daf08e5765799645b1 Mon Sep 17 00:00:00 2001
From: Pip Cet <pipcet@HIDDEN>
Subject: [PATCH] Reduce pdump size by using in-place relocations on 64-bit
 systems

A 64-bit Lisp_Object is enough to store the dump offsets of both the
Lisp_Object that is meant to appear in a given location and the dump
offset of the next compressed relocation in the list.

* src/pdumper.c (dump_do_fixup): Compress a relocation plus a link to
the previous one into a single Lisp_Object, if it fits.
(dump_do_fixups):
(Fdump_emacs_portable): Keep track of the list of in-place fixup
relocations.
(dump_do_fixup_chain): Replace compressed in-place relocations by
their fixups.
(pdumper_load): Call 'dump_do_fixup_chain' before applying explicit
relocs.
---
 src/pdumper.c | 67 +++++++++++++++++++++++++++++++++++++++++++++------
 1 file changed, 59 insertions(+), 8 deletions(-)

diff --git a/src/pdumper.c b/src/pdumper.c
index de213130756..5ccac63637a 100644
--- a/src/pdumper.c
+++ b/src/pdumper.c
@@ -403,6 +403,9 @@ dump_fingerprint (FILE *output, char const *label,
=20
   /* Offset of a vector of the dumped hash tables.  */
   dump_off hash_list;
+
+  /* Offset of the last relocation compressed into a Lisp_Object.  */
+  dump_off fixup_chain;
 };
=20
 /* Double-ended singly linked list.  */
@@ -3991,10 +3994,11 @@ drain_reloc_list (struct dump_context *ctx,
   ctx->flags =3D old_flags;
 }
=20
-static void
+static dump_off
 dump_do_fixup (struct dump_context *ctx,
                Lisp_Object fixup,
-               Lisp_Object prev_fixup)
+               Lisp_Object prev_fixup,
+=09       dump_off prev_fixup_chain)
 {
   enum dump_fixup_type type
     =3D (enum dump_fixup_type) XFIXNUM (dump_pop (&fixup));
@@ -4013,6 +4017,7 @@ dump_do_fixup (struct dump_context *ctx,
   dump_seek (ctx, dump_fixup_offset);
   intptr_t dump_value;
   bool do_write =3D true;
+  bool do_link =3D false;
   switch (type)
     {
     case DUMP_FIXUP_LISP_OBJECT:
@@ -4051,7 +4056,18 @@ dump_do_fixup (struct dump_context *ctx,
           dump_value =3D dump_recall_object (ctx, arg);
           if (dump_value <=3D 0)
             error ("fixup object not dumped");
-          if (type =3D=3D DUMP_FIXUP_LISP_OBJECT)
+=09replace:
+=09  if (sizeof (Lisp_Object) =3D=3D 8
+=09      && ctx->offset - prev_fixup_chain < 0x10000000
+=09      && dump_value < 0x10000000)
+=09    {
+=09      uint64_t new_dump_value =3D
+=09=09((dump_value << 36LL) + ((long long)((type =3D=3D DUMP_FIXUP_LISP_OB=
JECT) ? (int) XTYPE (arg) : (int) 8) << 32LL) +
+=09=09 (ctx->offset - prev_fixup_chain));
+=09      do_link =3D true;
+=09      dump_value =3D new_dump_value;
+=09    }
+          else if (type =3D=3D DUMP_FIXUP_LISP_OBJECT)
             dump_reloc_dump_to_dump_lv (ctx, ctx->offset, XTYPE (arg));
           else
             dump_reloc_dump_to_dump_ptr_raw (ctx, ctx->offset);
@@ -4062,8 +4078,7 @@ dump_do_fixup (struct dump_context *ctx,
          object.  It knows the exact location it wants, so just
          believe it.  */
       dump_value =3D dump_off_from_lisp (arg);
-      dump_reloc_dump_to_dump_ptr_raw (ctx, ctx->offset);
-      break;
+      goto replace;
     case DUMP_FIXUP_BIGNUM_DATA:
       {
         eassert (BIGNUMP (arg));
@@ -4081,11 +4096,13 @@ dump_do_fixup (struct dump_context *ctx,
     default:
       emacs_abort ();
     }
+  dump_off ret =3D do_link ? ctx->offset : prev_fixup_chain;
   if (do_write)
     dump_write (ctx, &dump_value, sizeof (dump_value));
+  return ret;
 }
=20
-static void
+static dump_off
 dump_do_fixups (struct dump_context *ctx)
 {
   dump_off saved_offset =3D ctx->offset;
@@ -4094,13 +4111,15 @@ dump_do_fixups (struct dump_context *ctx)
 =09=09=09      Qdump_emacs_portable__sort_predicate);
   Lisp_Object prev_fixup =3D Qnil;
   ctx->fixups =3D Qnil;
+  dump_off prev_fixup_chain =3D 0;
   while (!NILP (fixups))
     {
       Lisp_Object fixup =3D dump_pop (&fixups);
-      dump_do_fixup (ctx, fixup, prev_fixup);
+      prev_fixup_chain =3D dump_do_fixup (ctx, fixup, prev_fixup, prev_fix=
up_chain);
       prev_fixup =3D fixup;
     }
   dump_seek (ctx, saved_offset);
+  return prev_fixup_chain;
 }
=20
 static void
@@ -4371,7 +4390,7 @@ DEFUN ("dump-emacs-portable",
   ctx->end_heap =3D ctx->offset;
=20
   /* Make remembered modifications to the dump file itself.  */
-  dump_do_fixups (ctx);
+  dump_off fixup_chain =3D dump_do_fixups (ctx);
=20
   drain_reloc_merger emacs_reloc_merger =3D
 #ifdef ENABLE_CHECKING
@@ -4410,6 +4429,7 @@ DEFUN ("dump-emacs-portable",
=20
   /* Dump is complete.  Go back to the header and write the magic
      indicating that the dump is complete and can be loaded.  */
+  ctx->header.fixup_chain =3D fixup_chain;
   ctx->header.magic[0] =3D dump_magic[0];
   dump_seek (ctx, 0);
   dump_write (ctx, &ctx->header, sizeof (ctx->header));
@@ -5636,6 +5656,36 @@ dump_do_emacs_relocation (const uintptr_t dump_base,
     }
 }
=20
+static void
+dump_do_fixup_chain (const struct dump_header *const header,
+=09=09     const uintptr_t dump_base)
+{
+  const dump_off fixup_chain =3D header->fixup_chain;
+  dump_off curr_off =3D fixup_chain;
+  while (curr_off)
+    {
+      uint64_t dump_value;
+      memcpy (&dump_value,
+              dump_ptr (dump_base, curr_off),
+              sizeof (dump_value));
+      int type =3D (dump_value >> 32LL) & 15;
+      uintptr_t value =3D (dump_value >> 36LL) & 0xfffffff;
+      uintptr_t delta =3D dump_value & 0xfffffff;
+      void *obj_ptr =3D dump_ptr (dump_base, value);
+      Lisp_Object lv =3D Qnil;
+      if (type =3D=3D 8)
+=09lv =3D XIL ((uintptr_t)obj_ptr);
+      else if (type =3D=3D Lisp_Symbol)
+=09lv =3D make_lisp_symbol (obj_ptr);
+      else
+=09lv =3D make_lisp_ptr (obj_ptr, type);
+      memcpy (dump_ptr (dump_base, curr_off),
+=09      &lv,
+=09      sizeof lv);
+      curr_off -=3D delta;
+    }
+}
+
 static void
 dump_do_all_emacs_relocations (const struct dump_header *const header,
 =09=09=09       const uintptr_t dump_base)
@@ -5831,6 +5881,7 @@ pdumper_load (const char *dump_filename, char *argv0)
   dump_public.start =3D dump_base;
   dump_public.end =3D dump_public.start + dump_size;
=20
+  dump_do_fixup_chain (header, dump_base);
   dump_do_all_dump_reloc_for_phase (header, dump_base, EARLY_RELOCS);
   dump_do_all_emacs_relocations (header, dump_base);
=20
--=20
2.48.1

Message sent:

Content-Disposition: inline
Content-Transfer-Encoding: quoted-printable
MIME-Version: 1.0
X-Mailer: MIME-tools 5.505 (Entity 5.505)
Content-Type: text/plain; charset=utf-8
X-Loop: help-debbugs@HIDDEN
From: help-debbugs@HIDDEN (GNU bug Tracking System)
To: Pip Cet <pipcet@HIDDEN>
Subject: bug#77150: Acknowledgement (pdumper: reduce pdmp size for 64-bit
 systems)
Message-ID: <handler.77150.B.17425573342913.ack <at> debbugs.gnu.org>
References: <87h63mzk4o.fsf@HIDDEN>
X-Gnu-PR-Message: ack 77150
X-Gnu-PR-Package: emacs
Reply-To: 77150 <at> debbugs.gnu.org
Date: Fri, 21 Mar 2025 11:43:04 +0000

Thank you for filing a new bug report with debbugs.gnu.org.

This is an automatically generated reply to let you know your message
has been received.

Your message is being forwarded to the package maintainers and other
interested parties for their attention; they will reply in due course.

Your message has been sent to the package maintainer(s):
 bug-gnu-emacs@HIDDEN

If you wish to submit further information on this problem, please
send it to 77150 <at> debbugs.gnu.org.

Please do not send mail to help-debbugs@HIDDEN unless you wish
to report a problem with the Bug-tracking system.

--=20
77150: https://debbugs.gnu.org/cgi/bugreport.cgi?bug=3D77150
GNU Bug Tracking System
Contact help-debbugs@HIDDEN with problems

Last modified: Fri, 21 Mar 2025 11:45:02 UTC

GNU bug tracking system
Copyright (C) 1999 Darren O. Benham, 1997 nCipher Corporation Ltd, 1994-97 Ian Jackson.