Received: (at 19235) by debbugs.gnu.org; 26 Jun 2016 00:51:17 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sat Jun 25 20:51:17 2016
Received: from localhost ([127.0.0.1]:55844 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1bGyI9-0001pE-DU
for submit <at> debbugs.gnu.org; Sat, 25 Jun 2016 20:51:17 -0400
Received: from world.peace.net ([50.252.239.5]:58617)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mhw@HIDDEN>)
id 1bGyI7-0001ow-Cc; Sat, 25 Jun 2016 20:51:15 -0400
Received: from pool-71-174-35-80.bstnma.east.verizon.net ([71.174.35.80]
helo=jojen)
by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.84_2) (envelope-from <mhw@HIDDEN>)
id 1bGyI1-0001EP-6c; Sat, 25 Jun 2016 20:51:09 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=)
Subject: Re: bug#19235: make-fresh-user-module procedure leaks memory
References: <20141130232834.32cbf5b2@HIDDEN>
<87iohn51dc.fsf@HIDDEN> <20141207141903.1347c764@HIDDEN>
<20141226182608.24c7dabf@HIDDEN> <87por9ru0u.fsf@HIDDEN>
<878txwrnub.fsf@HIDDEN> <8760szau7r.fsf@HIDDEN>
Date: Sat, 25 Jun 2016 20:50:56 -0400
In-Reply-To: <8760szau7r.fsf@HIDDEN> ("Ludovic
\=\?utf-8\?Q\?Court\=C3\=A8s\=22'\?\=
\=\?utf-8\?Q\?s\?\= message of "Fri, 24 Jun 2016 10:04:24 +0200")
Message-ID: <878txsrcwf.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.0.95 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 19235
Cc: Andy Wingo <wingo@HIDDEN>, 15602 <at> debbugs.gnu.org,
Chris Vine <chris@HIDDEN>, 19235 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)
ludo@HIDDEN (Ludovic Court=C3=A8s) writes:
> Mark H Weaver <mhw@HIDDEN> skribis:
>
>> One potential issue that has been troubling me is that in Guile's model,
>> there's no guarantee that a module will _ever_ finish loading.
>
> I think the fact that we evaluate all the top-level forms is
> problematic. The R6RS phases were a great idea. :-)
>
>> The main program itself could simply run from the auto-load. That's
>> why I think it's important to propagate permission to threads created
>> during the auto-load, but maybe there will still be problems.
>
> I=E2=80=99m not sure what you mean by =E2=80=9Cpropagate permission=E2=80=
=9D?
I mean: propagate permission to access the not-yet-committed module.
For example, suppose a program loads a module that runs the main event
loop as a top-level form in its body. This module will never be
committed to the global module table, because it never finishes loading.
Now suppose that it spawns some new threads. Those threads should have
access to the module.
Similarly, if a module uses 'par-for-each' to initialize some tables,
the spawned threads should have access to the module being loaded.
Mark
bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.Received: (at 19235) by debbugs.gnu.org; 24 Jun 2016 08:04:45 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Jun 24 04:04:45 2016 Received: from localhost ([127.0.0.1]:53319 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1bGM6X-0001H4-1x for submit <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:04:45 -0400 Received: from eggs.gnu.org ([208.118.235.92]:38775) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <ludo@HIDDEN>) id 1bGM6V-0001Gi-IN for 19235 <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:04:43 -0400 Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1bGM6P-0006ps-Mj for 19235 <at> debbugs.gnu.org; Fri, 24 Jun 2016 04:04:38 -0400 X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=5.0 tests=BAYES_50,RP_MATCHES_RCVD autolearn=disabled version=3.3.2 Received: from fencepost.gnu.org ([2001:4830:134:3::e]:54148) by eggs.gnu.org with esmtp (Exim 4.71) (envelope-from <ludo@HIDDEN>) id 1bGM6H-0006oP-0x; Fri, 24 Jun 2016 04:04:29 -0400 Received: from pluto.bordeaux.inria.fr ([193.50.110.57]:47396 helo=pluto) by fencepost.gnu.org with esmtpsa (TLS1.2:RSA_AES_128_CBC_SHA1:128) (Exim 4.82) (envelope-from <ludo@HIDDEN>) id 1bGM6F-0003tR-VS; Fri, 24 Jun 2016 04:04:28 -0400 From: ludo@HIDDEN (Ludovic =?utf-8?Q?Court=C3=A8s?=) To: Mark H Weaver <mhw@HIDDEN> Subject: Re: bug#19235: make-fresh-user-module procedure leaks memory References: <20141130232834.32cbf5b2@HIDDEN> <87iohn51dc.fsf@HIDDEN> <20141207141903.1347c764@HIDDEN> <20141226182608.24c7dabf@HIDDEN> <87por9ru0u.fsf@HIDDEN> <878txwrnub.fsf@HIDDEN> X-URL: http://www.fdn.fr/~lcourtes/ X-Revolutionary-Date: 7 Messidor an 224 de la =?utf-8?Q?R=C3=A9volution?= X-PGP-Key-ID: 0x090B11993D9AEBB5 X-PGP-Key: http://www.fdn.fr/~lcourtes/ludovic.asc X-PGP-Fingerprint: 3CE4 6455 8A84 FDC6 9DB4 0CFB 090B 1199 3D9A EBB5 X-OS: x86_64-unknown-linux-gnu Date: Fri, 24 Jun 2016 10:04:24 +0200 In-Reply-To: <878txwrnub.fsf@HIDDEN> (Mark H. Weaver's message of "Thu, 23 Jun 2016 10:17:48 -0400") Message-ID: <8760szau7r.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.2.x-3.x [generic] X-Received-From: 2001:4830:134:3::e X-Spam-Score: -6.4 (------) X-Debbugs-Envelope-To: 19235 Cc: Andy Wingo <wingo@HIDDEN>, 15602 <at> debbugs.gnu.org, Chris Vine <chris@HIDDEN>, 19235 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -6.4 (------) Mark H Weaver <mhw@HIDDEN> skribis: > Andy Wingo <wingo@HIDDEN> writes: > >> In many ways I think Ludovic was right in #15602 -- we should allow >> excursions to isolate changes to the module tree. Sometimes you want an >> excursion to never add a module to the tree. Sometimes you do, but >> maybe all in one go and with a mutex, to avoid races -- like, you could >> load a file or evaluate some code in a private fork of the module tree, >> but then commit it to the main tree afterwards. Is that a sensible >> thing? > > Yes, I agree. In fact, I'd been thinking of something along those lines > to enable thread-safe module loading. More specifically, I was thinking > that there should be a fluid variable that contains some additional > modules that are not yet committed to the global module tree. > > Briefly, when a module is auto-loaded by a thread, the new module would > initially be visible only to that thread, and also to any threads > spawned by that thread during the auto-load. Any attempts to access the > module from other threads would block until the module is either fully > loaded. That sounds like a nice idea. In the current state of things, perhaps this behavior could be emulated by running =E2=80=98compile-file=E2=80=99 in a module excursion, and passin= g it a root module that=E2=80=99s a copy of =E2=80=98the-root-module=E2=80=99, somethin= g like that (though that would probably perform badly.) > One potential issue that has been troubling me is that in Guile's model, > there's no guarantee that a module will _ever_ finish loading. I think the fact that we evaluate all the top-level forms is problematic. The R6RS phases were a great idea. :-) > The main program itself could simply run from the auto-load. That's > why I think it's important to propagate permission to threads created > during the auto-load, but maybe there will still be problems. I=E2=80=99m not sure what you mean by =E2=80=9Cpropagate permission=E2=80= =9D? Ludo=E2=80=99.
bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.
Received: (at 19235) by debbugs.gnu.org; 23 Jun 2016 14:18:09 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Thu Jun 23 10:18:09 2016
Received: from localhost ([127.0.0.1]:52761 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1bG5SK-00036p-Qs
for submit <at> debbugs.gnu.org; Thu, 23 Jun 2016 10:18:09 -0400
Received: from world.peace.net ([50.252.239.5]:32984)
by debbugs.gnu.org with esmtp (Exim 4.84_2)
(envelope-from <mhw@HIDDEN>)
id 1bG5SJ-000363-0T; Thu, 23 Jun 2016 10:18:07 -0400
Received: from c-73-253-48-168.hsd1.ma.comcast.net ([73.253.48.168] helo=jojen)
by world.peace.net with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128)
(Exim 4.84_2) (envelope-from <mhw@HIDDEN>)
id 1bG5SC-0007iY-6c; Thu, 23 Jun 2016 10:18:00 -0400
From: Mark H Weaver <mhw@HIDDEN>
To: Andy Wingo <wingo@HIDDEN>
Subject: Re: bug#19235: make-fresh-user-module procedure leaks memory
References: <20141130232834.32cbf5b2@HIDDEN>
<87iohn51dc.fsf@HIDDEN> <20141207141903.1347c764@HIDDEN>
<20141226182608.24c7dabf@HIDDEN> <87por9ru0u.fsf@HIDDEN>
Date: Thu, 23 Jun 2016 10:17:48 -0400
In-Reply-To: <87por9ru0u.fsf@HIDDEN> (Andy Wingo's message of "Wed, 22 Jun
2016 19:52:01 +0200")
Message-ID: <878txwrnub.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/25.0.95 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 19235
Cc: 15602 <at> debbugs.gnu.org, ludo@HIDDEN,
Chris Vine <chris@HIDDEN>, 19235 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.18
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)
Andy Wingo <wingo@HIDDEN> writes:
> In many ways I think Ludovic was right in #15602 -- we should allow
> excursions to isolate changes to the module tree. Sometimes you want an
> excursion to never add a module to the tree. Sometimes you do, but
> maybe all in one go and with a mutex, to avoid races -- like, you could
> load a file or evaluate some code in a private fork of the module tree,
> but then commit it to the main tree afterwards. Is that a sensible
> thing?
Yes, I agree. In fact, I'd been thinking of something along those lines
to enable thread-safe module loading. More specifically, I was thinking
that there should be a fluid variable that contains some additional
modules that are not yet committed to the global module tree.
Briefly, when a module is auto-loaded by a thread, the new module would
initially be visible only to that thread, and also to any threads
spawned by that thread during the auto-load. Any attempts to access the
module from other threads would block until the module is either fully
loaded.
One potential issue that has been troubling me is that in Guile's model,
there's no guarantee that a module will _ever_ finish loading. The main
program itself could simply run from the auto-load. That's why I think
it's important to propagate permission to threads created during the
auto-load, but maybe there will still be problems.
Thoughts?
Mark
bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.Received: (at 19235) by debbugs.gnu.org; 22 Jun 2016 17:52:15 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Wed Jun 22 13:52:15 2016 Received: from localhost ([127.0.0.1]:51415 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1bFmJz-0000b4-Fn for submit <at> debbugs.gnu.org; Wed, 22 Jun 2016 13:52:15 -0400 Received: from pb-sasl1.pobox.com ([64.147.108.66]:58329 helo=sasl.smtp.pobox.com) by debbugs.gnu.org with esmtp (Exim 4.84_2) (envelope-from <wingo@HIDDEN>) id 1bFmJv-0000ao-DG; Wed, 22 Jun 2016 13:52:13 -0400 Received: from sasl.smtp.pobox.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id B73731FFC6; Wed, 22 Jun 2016 13:52:09 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha1; c=relaxed; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; s=sasl; bh=uHz8mIV11CKPJO+Gvjt3WnWbTK0=; b=uDFhaM YpKzW04+r+tQpmZHEnji5PdvR3LCIYoTc1X3ozbQSRYQWXiSuQcOYoOQ/TnNjTP2 fAUbvrY+WqSqt3hqVDmB2V74VTQDXlt0XdZON1N6eKt9HMDHtbIVpRq2ivLrbM8o E8AWig0NCN5XnW2zSXETulJds2wuu6CjsTcXc= DomainKey-Signature: a=rsa-sha1; c=nofws; d=pobox.com; h=from:to:cc :subject:references:date:in-reply-to:message-id:mime-version :content-type; q=dns; s=sasl; b=rIDWMjJzFBxMwmSAj560Gdt2NWWqaQ9L BYnBccZ6mTZqxTpKc2nGNKTe4hHCCA1kHoTQbW7x9Qb0OQ/9Fgm8RpD766yBJLTz Ro2ULFoZt3sBK2l08aYbL2gZd/1s8Zd/UrKz9tQlLItpkinBMRZPXHhyvnDzK4mc HoZlIsW5BNo= Received: from pb-sasl1.nyi.icgroup.com (unknown [127.0.0.1]) by pb-sasl1.pobox.com (Postfix) with ESMTP id AFEC91FFC5; Wed, 22 Jun 2016 13:52:09 -0400 (EDT) Received: from clucks (unknown [88.160.190.192]) (using TLSv1 with cipher ECDHE-RSA-AES256-SHA (256/256 bits)) (No client certificate requested) by pb-sasl1.pobox.com (Postfix) with ESMTPSA id CBAC11FFC4; Wed, 22 Jun 2016 13:52:08 -0400 (EDT) From: Andy Wingo <wingo@HIDDEN> To: Chris Vine <chris@HIDDEN> Subject: Re: bug#19235: make-fresh-user-module procedure leaks memory References: <20141130232834.32cbf5b2@HIDDEN> <87iohn51dc.fsf@HIDDEN> <20141207141903.1347c764@HIDDEN> <20141226182608.24c7dabf@HIDDEN> Date: Wed, 22 Jun 2016 19:52:01 +0200 In-Reply-To: <20141226182608.24c7dabf@HIDDEN> (Chris Vine's message of "Fri, 26 Dec 2014 18:26:08 +0000") Message-ID: <87por9ru0u.fsf@HIDDEN> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.5 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain X-Pobox-Relay-ID: 0A5E99C2-38A2-11E6-BF3F-C1836462E9F6-02397024!pb-sasl1.pobox.com X-Spam-Score: -1.4 (-) X-Debbugs-Envelope-To: 19235 Cc: Mark H Weaver <mhw@HIDDEN>, ludo@HIDDEN, 15602 <at> debbugs.gnu.org, 19235 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.18 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: -1.4 (-) In many ways I think Ludovic was right in #15602 -- we should allow excursions to isolate changes to the module tree. Sometimes you want an excursion to never add a module to the tree. Sometimes you do, but maybe all in one go and with a mutex, to avoid races -- like, you could load a file or evaluate some code in a private fork of the module tree, but then commit it to the main tree afterwards. Is that a sensible thing? Andy On Fri 26 Dec 2014 19:26, Chris Vine <chris@HIDDEN> writes: > As far as I can tell the make-fresh-user-module procedure is not called > by guile itself, and providing a global mutex for it with a binding > enabling it to be called from scheme code seems to work fine. > > This also makes it straightforward to incorporate in a thread-safe > way the code you suggested to free stale user modules. However, as I > mentioned, I am a bit reluctant to incorporate code which might break > in the future. Is there any possibility that a "delete-module!" > procedure could be included within the public guile API for the next > release of guile? It seems like something that could be useful to > anyone using non-default user modules in their code. > > Chris
bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.Received: (at 19235) by debbugs.gnu.org; 26 Dec 2014 18:26:14 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Fri Dec 26 13:26:14 2014 Received: from localhost ([127.0.0.1]:58382 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1Y4ZaY-0001w6-B7 for submit <at> debbugs.gnu.org; Fri, 26 Dec 2014 13:26:14 -0500 Received: from smtpout5.wanadoo.co.uk ([80.12.242.80]:24047 helo=smtpout.wanadoo.co.uk) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <chris@HIDDEN>) id 1Y4ZaU-0001vs-JH for 19235 <at> debbugs.gnu.org; Fri, 26 Dec 2014 13:26:13 -0500 Received: from laptop.homenet ([95.146.111.203]) by mwinf5d66 with ME id YJS81p00E4PMKgF03JS8Ln; Fri, 26 Dec 2014 19:26:08 +0100 X-ME-Helo: laptop.homenet X-ME-Date: Fri, 26 Dec 2014 19:26:08 +0100 X-ME-IP: 95.146.111.203 Received: from bother.homenet (localhost [127.0.0.1]) by laptop.homenet (Postfix) with ESMTP id 7513E8C1F0; Fri, 26 Dec 2014 18:26:08 +0000 (GMT) Date: Fri, 26 Dec 2014 18:26:08 +0000 From: Chris Vine <chris@HIDDEN> To: Mark H Weaver <mhw@HIDDEN> Subject: Re: bug#19235: make-fresh-user-module procedure leaks memory Message-ID: <20141226182608.24c7dabf@HIDDEN> In-Reply-To: <20141207141903.1347c764@HIDDEN> References: <20141130232834.32cbf5b2@HIDDEN> <87iohn51dc.fsf@HIDDEN> <20141207141903.1347c764@HIDDEN> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; i686-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 19235 Cc: 19235 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: 0.0 (/) As far as I can tell the make-fresh-user-module procedure is not called by guile itself, and providing a global mutex for it with a binding enabling it to be called from scheme code seems to work fine. This also makes it straightforward to incorporate in a thread-safe way the code you suggested to free stale user modules. However, as I mentioned, I am a bit reluctant to incorporate code which might break in the future. Is there any possibility that a "delete-module!" procedure could be included within the public guile API for the next release of guile? It seems like something that could be useful to anyone using non-default user modules in their code. Chris
bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.Received: (at 19235) by debbugs.gnu.org; 7 Dec 2014 14:19:09 +0000 From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 07 09:19:09 2014 Received: from localhost ([127.0.0.1]:56076 helo=debbugs.gnu.org) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>) id 1Xxcg0-00078c-MV for submit <at> debbugs.gnu.org; Sun, 07 Dec 2014 09:19:09 -0500 Received: from smtpout5.wanadoo.co.uk ([80.12.242.80]:53392 helo=smtpout.wanadoo.co.uk) by debbugs.gnu.org with esmtp (Exim 4.80) (envelope-from <chris@HIDDEN>) id 1Xxcfx-00078P-9e for 19235 <at> debbugs.gnu.org; Sun, 07 Dec 2014 09:19:06 -0500 Received: from laptop.homenet ([95.146.110.225]) by mwinf5d67 with ME id QeK21p0044rpotr03eK2hu; Sun, 07 Dec 2014 15:19:03 +0100 X-ME-Helo: laptop.homenet X-ME-Date: Sun, 07 Dec 2014 15:19:03 +0100 X-ME-IP: 95.146.110.225 Received: from bother.homenet (localhost [127.0.0.1]) by laptop.homenet (Postfix) with ESMTP id CDDC88BCBE; Sun, 7 Dec 2014 14:19:03 +0000 (GMT) Date: Sun, 7 Dec 2014 14:19:03 +0000 From: Chris Vine <chris@HIDDEN> To: Mark H Weaver <mhw@HIDDEN> Subject: Re: bug#19235: make-fresh-user-module procedure leaks memory Message-ID: <20141207141903.1347c764@HIDDEN> In-Reply-To: <87iohn51dc.fsf@HIDDEN> References: <20141130232834.32cbf5b2@HIDDEN> <87iohn51dc.fsf@HIDDEN> X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; i686-pc-linux-gnu) MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII Content-Transfer-Encoding: 7bit X-Spam-Score: 0.0 (/) X-Debbugs-Envelope-To: 19235 Cc: 19235 <at> debbugs.gnu.org X-BeenThere: debbugs-submit <at> debbugs.gnu.org X-Mailman-Version: 2.1.15 Precedence: list List-Id: <debbugs-submit.debbugs.gnu.org> List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe> List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/> List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org> List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help> List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>, <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe> Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org> X-Spam-Score: 0.0 (/) On Sun, 07 Dec 2014 03:07:43 -0500 Mark H Weaver <mhw@HIDDEN> wrote: > Chris Vine <chris@HIDDEN> writes: > > > The make-fresh-user-module procedure leaks memory in guile-2.0.11 as > > demonstrated by the attached test case. [...] > > > > The question which might be asked is "Would any sane person ever > > want to invoke the make-fresh-user-module procedure more than a few > > times in a practical program?". The answer to this question is > > "Yes", if guile is being used as an extension framework for a C or > > C++ program, and it executes guile extensions as individual tasks, > > and it is necessary that top levels should not to be shared. The > > execution of tasks concurrently is one such case, but there can be > > many cases where isolated top levels are desirable for tasks > > executed serially also. > > Unfortunately, Guile modules cannot be garbage collected. The problem > is that modules are usually referenced by name, not by direct > pointers. Every module must have a name, due to the way our macro > expander works. Modules created by 'make-module' or > 'make-fresh-user-module' are named using gensyms. > > We maintain a global map from module names to module objects, and we > can never safely delete from this map, because we cannot prove that > the module name won't be looked up in the future. > > I agree that your use case is reasonable. I'll think about how we > might allow unnamed modules to be collected, but I'm afraid it might > be quite difficult. > > In the meantime, I wrote a procedure that uses undocumented interfaces > to forcefully delete a module from the name->module map. However, I > must emphasize that this procedure is likely to break in a future > version of Guile. However, it should work in the 2.0.x series. [snip] > I should mention that creating new modules with > 'make-fresh-user-module' is not thread safe, nor is the procedure > above. Both of them mutate the same name->module map. For now, I > recommend protecting calls to both of them with a mutex. Thanks for that. Having make-fresh-user-module (and possibly set-current-module ??) not thread safe should be relatively easy to deal with in my use case, subject to the next paragraph. I would need to call these procedures in C/C++ code before calling scm_eval_string() to execute a scheme task, but presumably I can obtain a C variable reference for make-fresh-user-module by calling scm_c_lookup() - set-current-module already has a C interface provided by libguile. I would also need to use a POSIX mutex, but that is fine as guile uses native threads. However, that would not be completely effective if guile might also call make-fresh-user-module internally, since that would be unprotected. Are there any circumstances in which guile might do this? I know from what you said some time ago that guile module loading is not thread safe. Without asking you to exercise powers of clairvoyance, can you think of any other thread safety problems I should be on the look out for? The documentation says that "multiple threads can call scm_with_guile concurrently" and whereas that might literally be true there seem a number of other things that one might reasonably expect to do with scm_with_guile that are not. On the memory leak, the usage case involves a library so I do not think I can include code which might be liable to breakage at some indeterminate time. It would also be quite difficult to implement by reference to the same mutex as used to protect make-fresh-user-module because that would mean reimplementing your suggested code on the C/C++ side. So I think it best just to document the leakage in the documentation for the library (my one, not guile), and hope that at some time it will be possible to deal with it in guile. Chris
bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.
Received: (at 19235) by debbugs.gnu.org; 7 Dec 2014 08:09:34 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Dec 07 03:09:34 2014
Received: from localhost ([127.0.0.1]:55987 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.80)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1XxWuM-0003ri-DI
for submit <at> debbugs.gnu.org; Sun, 07 Dec 2014 03:09:34 -0500
Received: from world.peace.net ([50.252.239.5]:59930)
by debbugs.gnu.org with esmtp (Exim 4.80)
(envelope-from <mhw@HIDDEN>) id 1XxWuK-0003ra-K8
for 19235 <at> debbugs.gnu.org; Sun, 07 Dec 2014 03:09:33 -0500
Received: from c-24-62-95-23.hsd1.ma.comcast.net ([24.62.95.23]
helo=yeeloong.lan)
by world.peace.net with esmtpsa (TLS1.0:RSA_AES_128_CBC_SHA1:16)
(Exim 4.72) (envelope-from <mhw@HIDDEN>)
id 1XxWuD-00087n-CR; Sun, 07 Dec 2014 03:09:25 -0500
From: Mark H Weaver <mhw@HIDDEN>
To: Chris Vine <chris@HIDDEN>
Subject: Re: bug#19235: make-fresh-user-module procedure leaks memory
References: <20141130232834.32cbf5b2@HIDDEN>
Date: Sun, 07 Dec 2014 03:07:43 -0500
In-Reply-To: <20141130232834.32cbf5b2@HIDDEN> (Chris Vine's message of
"Sun, 30 Nov 2014 23:28:34 +0000")
Message-ID: <87iohn51dc.fsf@HIDDEN>
User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/24.4 (gnu/linux)
MIME-Version: 1.0
Content-Type: text/plain
X-Spam-Score: 0.0 (/)
X-Debbugs-Envelope-To: 19235
Cc: 19235 <at> debbugs.gnu.org
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: 0.0 (/)
Chris Vine <chris@HIDDEN> writes:
> The make-fresh-user-module procedure leaks memory in guile-2.0.11 as
> demonstrated by the attached test case. [...]
>
> The question which might be asked is "Would any sane person ever want
> to invoke the make-fresh-user-module procedure more than a few times
> in a practical program?". The answer to this question is "Yes", if
> guile is being used as an extension framework for a C or C++ program,
> and it executes guile extensions as individual tasks, and it is
> necessary that top levels should not to be shared. The execution of
> tasks concurrently is one such case, but there can be many cases where
> isolated top levels are desirable for tasks executed serially also.
Unfortunately, Guile modules cannot be garbage collected. The problem
is that modules are usually referenced by name, not by direct pointers.
Every module must have a name, due to the way our macro expander works.
Modules created by 'make-module' or 'make-fresh-user-module' are named
using gensyms.
We maintain a global map from module names to module objects, and we can
never safely delete from this map, because we cannot prove that the
module name won't be looked up in the future.
I agree that your use case is reasonable. I'll think about how we might
allow unnamed modules to be collected, but I'm afraid it might be quite
difficult.
In the meantime, I wrote a procedure that uses undocumented interfaces
to forcefully delete a module from the name->module map. However, I
must emphasize that this procedure is likely to break in a future
version of Guile. However, it should work in the 2.0.x series.
--8<---------------cut here---------------start------------->8---
;; WARNING: Uses undocumented interfaces; NOT FUTURE PROOF!!
;; This needs (srfi srfi-1)
(define (delete-module! module)
(let* ((name (module-name module))
(last-name (last name))
(parent-name (drop-right name 1))
(parent (resolve-module parent-name #f))
(var (module-variable parent last-name)))
(when (and var
(variable-bound? var)
(eqv? (variable-ref var) module))
(hashq-remove! (module-obarray parent) last-name))
(hashq-remove! (module-submodules parent) last-name)
#f))
--8<---------------cut here---------------end--------------->8---
I should mention that creating new modules with 'make-fresh-user-module'
is not thread safe, nor is the procedure above. Both of them mutate the
same name->module map. For now, I recommend protecting calls to both of
them with a mutex.
Mark
bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.
Received: (at submit) by debbugs.gnu.org; 30 Nov 2014 23:29:08 +0000
From debbugs-submit-bounces <at> debbugs.gnu.org Sun Nov 30 18:29:08 2014
Received: from localhost ([127.0.0.1]:50210 helo=debbugs.gnu.org)
by debbugs.gnu.org with esmtp (Exim 4.80)
(envelope-from <debbugs-submit-bounces <at> debbugs.gnu.org>)
id 1XvDvP-00026s-MO
for submit <at> debbugs.gnu.org; Sun, 30 Nov 2014 18:29:07 -0500
Received: from eggs.gnu.org ([208.118.235.92]:48829)
by debbugs.gnu.org with esmtp (Exim 4.80)
(envelope-from <chris@HIDDEN>) id 1XvDvN-00026f-5A
for submit <at> debbugs.gnu.org; Sun, 30 Nov 2014 18:29:05 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <chris@HIDDEN>) id 1XvDvD-00052W-5z
for submit <at> debbugs.gnu.org; Sun, 30 Nov 2014 18:29:05 -0500
X-Spam-Checker-Version: SpamAssassin 3.3.2 (2011-06-06) on eggs.gnu.org
X-Spam-Level:
X-Spam-Status: No, score=0.8 required=5.0 tests=BAYES_50 autolearn=disabled
version=3.3.2
Received: from lists.gnu.org ([208.118.235.17]:56536)
by eggs.gnu.org with esmtp (Exim 4.71)
(envelope-from <chris@HIDDEN>) id 1XvDvD-00052Q-2c
for submit <at> debbugs.gnu.org; Sun, 30 Nov 2014 18:28:55 -0500
Received: from eggs.gnu.org ([2001:4830:134:3::10]:50417)
by lists.gnu.org with esmtp (Exim 4.71)
(envelope-from <chris@HIDDEN>) id 1XvDv5-00082v-Bg
for bug-guile@HIDDEN; Sun, 30 Nov 2014 18:28:54 -0500
Received: from Debian-exim by eggs.gnu.org with spam-scanned (Exim 4.71)
(envelope-from <chris@HIDDEN>) id 1XvDuw-0004vv-LN
for bug-guile@HIDDEN; Sun, 30 Nov 2014 18:28:47 -0500
Received: from smtpout3.wanadoo.co.uk ([80.12.242.59]:60772
helo=smtpout.wanadoo.co.uk) by eggs.gnu.org with esmtp (Exim 4.71)
(envelope-from <chris@HIDDEN>) id 1XvDuw-0004v3-CE
for bug-guile@HIDDEN; Sun, 30 Nov 2014 18:28:38 -0500
Received: from laptop.homenet ([95.146.110.225]) by mwinf5d37 with ME
id MzUb1p0024rpotr03zUbWR; Mon, 01 Dec 2014 00:28:35 +0100
X-ME-Helo: laptop.homenet
X-ME-Date: Mon, 01 Dec 2014 00:28:35 +0100
X-ME-IP: 95.146.110.225
Received: from bother.homenet (localhost [127.0.0.1])
by laptop.homenet (Postfix) with ESMTP id CCF938B70A
for <bug-guile@HIDDEN>; Sun, 30 Nov 2014 23:28:34 +0000 (GMT)
Date: Sun, 30 Nov 2014 23:28:34 +0000
From: Chris Vine <chris@HIDDEN>
To: bug-guile@HIDDEN
Subject: make-fresh-user-module procedure leaks memory
Message-ID: <20141130232834.32cbf5b2@HIDDEN>
X-Mailer: Claws Mail 3.11.1 (GTK+ 2.24.25; i686-pc-linux-gnu)
MIME-Version: 1.0
Content-Type: text/plain; charset=US-ASCII
Content-Transfer-Encoding: 7bit
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-detected-operating-system: by eggs.gnu.org: GNU/Linux 2.6.x
X-Received-From: 208.118.235.17
X-Spam-Score: -5.0 (-----)
X-Debbugs-Envelope-To: submit
X-BeenThere: debbugs-submit <at> debbugs.gnu.org
X-Mailman-Version: 2.1.15
Precedence: list
List-Id: <debbugs-submit.debbugs.gnu.org>
List-Unsubscribe: <https://debbugs.gnu.org/cgi-bin/mailman/options/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=unsubscribe>
List-Archive: <https://debbugs.gnu.org/cgi-bin/mailman/private/debbugs-submit/>
List-Post: <mailto:debbugs-submit <at> debbugs.gnu.org>
List-Help: <mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=help>
List-Subscribe: <https://debbugs.gnu.org/cgi-bin/mailman/listinfo/debbugs-submit>,
<mailto:debbugs-submit-request <at> debbugs.gnu.org?subject=subscribe>
Errors-To: debbugs-submit-bounces <at> debbugs.gnu.org
Sender: "Debbugs-submit" <debbugs-submit-bounces <at> debbugs.gnu.org>
X-Spam-Score: -5.0 (-----)
The make-fresh-user-module procedure leaks memory in guile-2.0.11 as
demonstrated by the attached test case. This test case should be
invoked either with the "shared" or "fresh" option. If invoked with
the "fresh" option, it will call make-fresh-user-module on each
iteration through the loop. On my 32-bit machine it will steadily
accumulate a memory leak before running out of memory on consuming
approximately 2.2G memory, after about 180,000 iterations. If called
with the "shared" option, it will accumulate no additional memory while
executing, and will execute normally to the end of its iterations.
The question which might be asked is "Would any sane person ever want
to invoke the make-fresh-user-module procedure more than a few times
in a practical program?". The answer to this question is "Yes", if
guile is being used as an extension framework for a C or C++ program,
and it executes guile extensions as individual tasks, and it is
necessary that top levels should not to be shared. The execution of
tasks concurrently is one such case, but there can be many cases where
isolated top levels are desirable for tasks executed serially also.
Test case:
----------------------------- snip -----------------------------
/* compile with 'gcc -O2 -Wall `pkg-config --cflags --libs guile-2.0` -o test-guile' */
#include <unistd.h>
#include <libguile.h>
#include <stdio.h>
#include <string.h>
int fresh;
void* func (void* data)
{
switch (fresh)
{
case 0:
scm_c_eval_string("");
break;
default:
scm_c_eval_string("(set-current-module (make-fresh-user-module))");
}
return NULL;
}
int main (int argc, char *argv[])
{
int count;
if (argc != 2
|| (strcmp (argv[1], "shared") &&
strcmp (argv[1], "fresh")))
{
puts ("Usage: test-guile shared | fresh");
exit (1);
}
if (!strcmp (argv[1], "fresh"))
{
puts("Invoking make-fresh-user-module");
fresh = 1;
}
else
puts("Using shared top level");
for (count = 0; count < 256000; ++count)
{
scm_with_guile(func, NULL);
if (!(count % 100)) {
printf("%d ", count);
fflush(stdout);
}
usleep(1);
}
puts("");
return 0;
}
Chris Vine <chris@HIDDEN>:bug-guile@HIDDEN.
Full text available.bug-guile@HIDDEN:bug#19235; Package guile.
Full text available.
GNU bug tracking system
Copyright (C) 1999 Darren O. Benham,
1997 nCipher Corporation Ltd,
1994-97 Ian Jackson.