Shared state in multiprocessing Processes
Please consider this code:
import time
from multiprocessing import Process
class Host(object):
def __init__(self):
self.id = None
def callback(self):
print "self.id = %s" % self.id
def bind(self, event_source):
event_source.callback = self.callback
class Event(object):
def __init__(self):
self.callback = None
def trigger(self):
self.callback()
h = Host()
h.id = "A"
e = Event()
h.bind(e)
e.trigger()
def delayed_trigger(f, delay):
time.sleep(delay)
f()
p = Process(target = delayed_trigger, args = (e.trigger, 3,))
p.start()
h.id = "B"
e.trigger()
This gives in output
self.id = A
self.id = B
self.id = A
However, I expected it to give
self.id = A
self.id = B
self.id = B
..because the h.id was already changed to "B" by the time the trigger method was called.
It seems that a copy of host instance is created at the moment when the separate Process is started, so the changes in the original host do not influence that copy.
In my project (more elaborate, of course), the host instance fields are altered time to time, and it is important that the events that are triggered by the code running in a separate process, have access to those changes.
python multiprocessing
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
add a comment |
Please consider this code:
import time
from multiprocessing import Process
class Host(object):
def __init__(self):
self.id = None
def callback(self):
print "self.id = %s" % self.id
def bind(self, event_source):
event_source.callback = self.callback
class Event(object):
def __init__(self):
self.callback = None
def trigger(self):
self.callback()
h = Host()
h.id = "A"
e = Event()
h.bind(e)
e.trigger()
def delayed_trigger(f, delay):
time.sleep(delay)
f()
p = Process(target = delayed_trigger, args = (e.trigger, 3,))
p.start()
h.id = "B"
e.trigger()
This gives in output
self.id = A
self.id = B
self.id = A
However, I expected it to give
self.id = A
self.id = B
self.id = B
..because the h.id was already changed to "B" by the time the trigger method was called.
It seems that a copy of host instance is created at the moment when the separate Process is started, so the changes in the original host do not influence that copy.
In my project (more elaborate, of course), the host instance fields are altered time to time, and it is important that the events that are triggered by the code running in a separate process, have access to those changes.
python multiprocessing
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
def __init(self):? Aren't you missing__at the end?
– thefourtheye
May 15 '15 at 16:48
2
You are correct: Multiprocessing runs those threads in separate threads and each thread has it's own instance of the Host class and they don't communicate with each other. You should check out this answer: stackoverflow.com/questions/16244745/…
– theodox
May 15 '15 at 16:48
1
multiprocessing does not share memory ... it is effectively 2 totally separate programs. usemultiprocessing.Pipeto communicate between processes, or use threading if you need shared memory space (there is a way to share memory with multiprocessing iirc ... but it makes everything slow ... unbearably so)
– Joran Beasley
May 15 '15 at 16:52
1
@theodox - each thread has it's own instance of the Host class - you've mixed up threads and processes. threads share the memory, processes do not.
– tdelaney
May 15 '15 at 16:58
true, sloppy vocab on my part. The main point is that they separate hosts are running in isolation.
– theodox
May 15 '15 at 17:14
add a comment |
Please consider this code:
import time
from multiprocessing import Process
class Host(object):
def __init__(self):
self.id = None
def callback(self):
print "self.id = %s" % self.id
def bind(self, event_source):
event_source.callback = self.callback
class Event(object):
def __init__(self):
self.callback = None
def trigger(self):
self.callback()
h = Host()
h.id = "A"
e = Event()
h.bind(e)
e.trigger()
def delayed_trigger(f, delay):
time.sleep(delay)
f()
p = Process(target = delayed_trigger, args = (e.trigger, 3,))
p.start()
h.id = "B"
e.trigger()
This gives in output
self.id = A
self.id = B
self.id = A
However, I expected it to give
self.id = A
self.id = B
self.id = B
..because the h.id was already changed to "B" by the time the trigger method was called.
It seems that a copy of host instance is created at the moment when the separate Process is started, so the changes in the original host do not influence that copy.
In my project (more elaborate, of course), the host instance fields are altered time to time, and it is important that the events that are triggered by the code running in a separate process, have access to those changes.
python multiprocessing
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
Please consider this code:
import time
from multiprocessing import Process
class Host(object):
def __init__(self):
self.id = None
def callback(self):
print "self.id = %s" % self.id
def bind(self, event_source):
event_source.callback = self.callback
class Event(object):
def __init__(self):
self.callback = None
def trigger(self):
self.callback()
h = Host()
h.id = "A"
e = Event()
h.bind(e)
e.trigger()
def delayed_trigger(f, delay):
time.sleep(delay)
f()
p = Process(target = delayed_trigger, args = (e.trigger, 3,))
p.start()
h.id = "B"
e.trigger()
This gives in output
self.id = A
self.id = B
self.id = A
However, I expected it to give
self.id = A
self.id = B
self.id = B
..because the h.id was already changed to "B" by the time the trigger method was called.
It seems that a copy of host instance is created at the moment when the separate Process is started, so the changes in the original host do not influence that copy.
In my project (more elaborate, of course), the host instance fields are altered time to time, and it is important that the events that are triggered by the code running in a separate process, have access to those changes.
python multiprocessing
python multiprocessing
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
edited Nov 25 '18 at 11:15
martineau
68.2k1090183
68.2k1090183
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
asked May 15 '15 at 16:44
PassidayPassiday
1,73152542
1,73152542
def __init(self):? Aren't you missing__at the end?
– thefourtheye
May 15 '15 at 16:48
2
You are correct: Multiprocessing runs those threads in separate threads and each thread has it's own instance of the Host class and they don't communicate with each other. You should check out this answer: stackoverflow.com/questions/16244745/…
– theodox
May 15 '15 at 16:48
1
multiprocessing does not share memory ... it is effectively 2 totally separate programs. usemultiprocessing.Pipeto communicate between processes, or use threading if you need shared memory space (there is a way to share memory with multiprocessing iirc ... but it makes everything slow ... unbearably so)
– Joran Beasley
May 15 '15 at 16:52
1
@theodox - each thread has it's own instance of the Host class - you've mixed up threads and processes. threads share the memory, processes do not.
– tdelaney
May 15 '15 at 16:58
true, sloppy vocab on my part. The main point is that they separate hosts are running in isolation.
– theodox
May 15 '15 at 17:14
add a comment |
def __init(self):? Aren't you missing__at the end?
– thefourtheye
May 15 '15 at 16:48
2
You are correct: Multiprocessing runs those threads in separate threads and each thread has it's own instance of the Host class and they don't communicate with each other. You should check out this answer: stackoverflow.com/questions/16244745/…
– theodox
May 15 '15 at 16:48
1
multiprocessing does not share memory ... it is effectively 2 totally separate programs. usemultiprocessing.Pipeto communicate between processes, or use threading if you need shared memory space (there is a way to share memory with multiprocessing iirc ... but it makes everything slow ... unbearably so)
– Joran Beasley
May 15 '15 at 16:52
1
@theodox - each thread has it's own instance of the Host class - you've mixed up threads and processes. threads share the memory, processes do not.
– tdelaney
May 15 '15 at 16:58
true, sloppy vocab on my part. The main point is that they separate hosts are running in isolation.
– theodox
May 15 '15 at 17:14
def __init(self):? Aren't you missing __ at the end?– thefourtheye
May 15 '15 at 16:48
def __init(self):? Aren't you missing __ at the end?– thefourtheye
May 15 '15 at 16:48
2
2
You are correct: Multiprocessing runs those threads in separate threads and each thread has it's own instance of the Host class and they don't communicate with each other. You should check out this answer: stackoverflow.com/questions/16244745/…
– theodox
May 15 '15 at 16:48
You are correct: Multiprocessing runs those threads in separate threads and each thread has it's own instance of the Host class and they don't communicate with each other. You should check out this answer: stackoverflow.com/questions/16244745/…
– theodox
May 15 '15 at 16:48
1
1
multiprocessing does not share memory ... it is effectively 2 totally separate programs. use
multiprocessing.Pipe to communicate between processes, or use threading if you need shared memory space (there is a way to share memory with multiprocessing iirc ... but it makes everything slow ... unbearably so)– Joran Beasley
May 15 '15 at 16:52
multiprocessing does not share memory ... it is effectively 2 totally separate programs. use
multiprocessing.Pipe to communicate between processes, or use threading if you need shared memory space (there is a way to share memory with multiprocessing iirc ... but it makes everything slow ... unbearably so)– Joran Beasley
May 15 '15 at 16:52
1
1
@theodox - each thread has it's own instance of the Host class - you've mixed up threads and processes. threads share the memory, processes do not.
– tdelaney
May 15 '15 at 16:58
@theodox - each thread has it's own instance of the Host class - you've mixed up threads and processes. threads share the memory, processes do not.
– tdelaney
May 15 '15 at 16:58
true, sloppy vocab on my part. The main point is that they separate hosts are running in isolation.
– theodox
May 15 '15 at 17:14
true, sloppy vocab on my part. The main point is that they separate hosts are running in isolation.
– theodox
May 15 '15 at 17:14
add a comment |
1 Answer
1
active
oldest
votes
multiprocessing runs stuff in separate processes. It is almost inconceivable that things are not copied as they're sent, as sharing stuff between processes requires shared memory or communication.
In fact, if you peruse the module, you can see the amount of effort it takes to actually share anything between the processes after the diverge, either through explicit communication, or through explicitly-shared objects (which are of a very limited subset of the language, and have to be managed by a Manager).
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
Thanks, I've managed to find a way how to run those event sources in separate threads rather than processes. If I could not do that perhaps I could use Redis to get the data across then.
– Passiday
May 15 '15 at 18:55
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f30264699%2fshared-state-in-multiprocessing-processes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
multiprocessing runs stuff in separate processes. It is almost inconceivable that things are not copied as they're sent, as sharing stuff between processes requires shared memory or communication.
In fact, if you peruse the module, you can see the amount of effort it takes to actually share anything between the processes after the diverge, either through explicit communication, or through explicitly-shared objects (which are of a very limited subset of the language, and have to be managed by a Manager).
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
Thanks, I've managed to find a way how to run those event sources in separate threads rather than processes. If I could not do that perhaps I could use Redis to get the data across then.
– Passiday
May 15 '15 at 18:55
add a comment |
multiprocessing runs stuff in separate processes. It is almost inconceivable that things are not copied as they're sent, as sharing stuff between processes requires shared memory or communication.
In fact, if you peruse the module, you can see the amount of effort it takes to actually share anything between the processes after the diverge, either through explicit communication, or through explicitly-shared objects (which are of a very limited subset of the language, and have to be managed by a Manager).
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
Thanks, I've managed to find a way how to run those event sources in separate threads rather than processes. If I could not do that perhaps I could use Redis to get the data across then.
– Passiday
May 15 '15 at 18:55
add a comment |
multiprocessing runs stuff in separate processes. It is almost inconceivable that things are not copied as they're sent, as sharing stuff between processes requires shared memory or communication.
In fact, if you peruse the module, you can see the amount of effort it takes to actually share anything between the processes after the diverge, either through explicit communication, or through explicitly-shared objects (which are of a very limited subset of the language, and have to be managed by a Manager).
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
multiprocessing runs stuff in separate processes. It is almost inconceivable that things are not copied as they're sent, as sharing stuff between processes requires shared memory or communication.
In fact, if you peruse the module, you can see the amount of effort it takes to actually share anything between the processes after the diverge, either through explicit communication, or through explicitly-shared objects (which are of a very limited subset of the language, and have to be managed by a Manager).
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
answered May 15 '15 at 16:53
Ami TavoryAmi Tavory
52.3k867107
52.3k867107
Thanks, I've managed to find a way how to run those event sources in separate threads rather than processes. If I could not do that perhaps I could use Redis to get the data across then.
– Passiday
May 15 '15 at 18:55
add a comment |
Thanks, I've managed to find a way how to run those event sources in separate threads rather than processes. If I could not do that perhaps I could use Redis to get the data across then.
– Passiday
May 15 '15 at 18:55
Thanks, I've managed to find a way how to run those event sources in separate threads rather than processes. If I could not do that perhaps I could use Redis to get the data across then.
– Passiday
May 15 '15 at 18:55
Thanks, I've managed to find a way how to run those event sources in separate threads rather than processes. If I could not do that perhaps I could use Redis to get the data across then.
– Passiday
May 15 '15 at 18:55
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f30264699%2fshared-state-in-multiprocessing-processes%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
def __init(self):? Aren't you missing__at the end?– thefourtheye
May 15 '15 at 16:48
2
You are correct: Multiprocessing runs those threads in separate threads and each thread has it's own instance of the Host class and they don't communicate with each other. You should check out this answer: stackoverflow.com/questions/16244745/…
– theodox
May 15 '15 at 16:48
1
multiprocessing does not share memory ... it is effectively 2 totally separate programs. use
multiprocessing.Pipeto communicate between processes, or use threading if you need shared memory space (there is a way to share memory with multiprocessing iirc ... but it makes everything slow ... unbearably so)– Joran Beasley
May 15 '15 at 16:52
1
@theodox - each thread has it's own instance of the Host class - you've mixed up threads and processes. threads share the memory, processes do not.
– tdelaney
May 15 '15 at 16:58
true, sloppy vocab on my part. The main point is that they separate hosts are running in isolation.
– theodox
May 15 '15 at 17:14