multiprocessing: map vs map_async
What's the difference between using map and map_async? Are they not running the same function after distributing the items from the list to 4 processes?
So is it wrong to presume both are running asynchronous and parallel?
def f(x):
return 2*x
p=Pool(4)
l=[1,2,3,4]
out1=p.map(f,l)
#vs
out2=p.map_async(f,l)
python-2.7 python-multiprocessing
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
asked Mar 10 '16 at 6:15
amanaman
5891727
add a comment |
What's the difference between using map and map_async? Are they not running the same function after distributing the items from the list to 4 processes?
So is it wrong to presume both are running asynchronous and parallel?
def f(x):
return 2*x
p=Pool(4)
l=[1,2,3,4]
out1=p.map(f,l)
#vs
out2=p.map_async(f,l)
python-2.7 python-multiprocessing
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
asked Mar 10 '16 at 6:15
amanaman
5891727
1
Doesn'tmapreturn only once the map is done (ie synchronously but in parallel), whilemap_asyncreturns right away and allows the mapping to be done in the background (ie asynchronously and in parallel)?
– Joachim Isaksson
Mar 10 '16 at 6:37
add a comment |
What's the difference between using map and map_async? Are they not running the same function after distributing the items from the list to 4 processes?
So is it wrong to presume both are running asynchronous and parallel?
def f(x):
return 2*x
p=Pool(4)
l=[1,2,3,4]
out1=p.map(f,l)
#vs
out2=p.map_async(f,l)
python-2.7 python-multiprocessing
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
asked Mar 10 '16 at 6:15
amanaman
5891727
What's the difference between using map and map_async? Are they not running the same function after distributing the items from the list to 4 processes?
So is it wrong to presume both are running asynchronous and parallel?
def f(x):
return 2*x
p=Pool(4)
l=[1,2,3,4]
out1=p.map(f,l)
#vs
out2=p.map_async(f,l)
python-2.7 python-multiprocessing
python-2.7 python-multiprocessing
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
asked Mar 10 '16 at 6:15
amanaman
5891727
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
asked Mar 10 '16 at 6:15
amanaman
5891727
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
edited Nov 25 '18 at 11:17
martineau
68.2k1090183
68.2k1090183
asked Mar 10 '16 at 6:15
amanaman
5891727
asked Mar 10 '16 at 6:15
amanaman
5891727
asked Mar 10 '16 at 6:15
amanaman
5891727
5891727
1
Doesn'tmapreturn only once the map is done (ie synchronously but in parallel), whilemap_asyncreturns right away and allows the mapping to be done in the background (ie asynchronously and in parallel)?
– Joachim Isaksson
Mar 10 '16 at 6:37
add a comment |
1
Doesn'tmapreturn only once the map is done (ie synchronously but in parallel), whilemap_asyncreturns right away and allows the mapping to be done in the background (ie asynchronously and in parallel)?
– Joachim Isaksson
Mar 10 '16 at 6:37
1
1
Doesn't
map return only once the map is done (ie synchronously but in parallel), while map_async returns right away and allows the mapping to be done in the background (ie asynchronously and in parallel)?– Joachim Isaksson
Mar 10 '16 at 6:37
Doesn't
map return only once the map is done (ie synchronously but in parallel), while map_async returns right away and allows the mapping to be done in the background (ie asynchronously and in parallel)?– Joachim Isaksson
Mar 10 '16 at 6:37
add a comment |
1 Answer
1
active
oldest
votes
There are four choices to mapping jobs to processes. You have to consider multi-args, concurrency, blocking, and ordering. map and map_asnyc only differ with respect to blocking. map_async is non-blocking where as map is blocking
So let's say you had a function
from multiprocessing import Pool
import time
def f(x):
print x*x
if __name__ == '__main__':
pool = Pool(processes=4)
pool.map(f, range(10))
r = pool.map_async(f, range(10))
# DO STUFF
print 'HERE'
print 'MORE'
r.wait()
print 'DONE'
Example output:
0
1
9
4
16
25
36
49
64
81
0
HERE
1
4
MORE
16
25
36
9
49
64
81
DONE
pool.map(f, range(10)) will wait for all 10 of those function calls to finish so we see all the prints in a row.
r = pool.map_async(f, range(10)) will execute them asynchronously and only block when r.wait() is called so we see HERE and MORE in between but DONE will always be at the end.
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
2
ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
– aman
Mar 10 '16 at 6:59
4
Not quite. You'll notice map will execute in order, but map_async doesn't
– quikst3r
Mar 10 '16 at 23:35
2
Should there be aprint 'DONE'afterr.wait()?
– HBeel
Nov 1 '16 at 14:32
Yes there should be!
– quikst3r
Nov 2 '16 at 19:32
If above example doesn't return different results formapandmap_asyncon first run, try settingrange(500)or something large.
– webelo
Oct 28 '17 at 15:22
|
show 2 more comments
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f35908987%2fmultiprocessing-map-vs-map-async%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
There are four choices to mapping jobs to processes. You have to consider multi-args, concurrency, blocking, and ordering. map and map_asnyc only differ with respect to blocking. map_async is non-blocking where as map is blocking
So let's say you had a function
from multiprocessing import Pool
import time
def f(x):
print x*x
if __name__ == '__main__':
pool = Pool(processes=4)
pool.map(f, range(10))
r = pool.map_async(f, range(10))
# DO STUFF
print 'HERE'
print 'MORE'
r.wait()
print 'DONE'
Example output:
0
1
9
4
16
25
36
49
64
81
0
HERE
1
4
MORE
16
25
36
9
49
64
81
DONE
pool.map(f, range(10)) will wait for all 10 of those function calls to finish so we see all the prints in a row.
r = pool.map_async(f, range(10)) will execute them asynchronously and only block when r.wait() is called so we see HERE and MORE in between but DONE will always be at the end.
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
2
ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
– aman
Mar 10 '16 at 6:59
4
Not quite. You'll notice map will execute in order, but map_async doesn't
– quikst3r
Mar 10 '16 at 23:35
2
Should there be aprint 'DONE'afterr.wait()?
– HBeel
Nov 1 '16 at 14:32
Yes there should be!
– quikst3r
Nov 2 '16 at 19:32
If above example doesn't return different results formapandmap_asyncon first run, try settingrange(500)or something large.
– webelo
Oct 28 '17 at 15:22
|
show 2 more comments
There are four choices to mapping jobs to processes. You have to consider multi-args, concurrency, blocking, and ordering. map and map_asnyc only differ with respect to blocking. map_async is non-blocking where as map is blocking
So let's say you had a function
from multiprocessing import Pool
import time
def f(x):
print x*x
if __name__ == '__main__':
pool = Pool(processes=4)
pool.map(f, range(10))
r = pool.map_async(f, range(10))
# DO STUFF
print 'HERE'
print 'MORE'
r.wait()
print 'DONE'
Example output:
0
1
9
4
16
25
36
49
64
81
0
HERE
1
4
MORE
16
25
36
9
49
64
81
DONE
pool.map(f, range(10)) will wait for all 10 of those function calls to finish so we see all the prints in a row.
r = pool.map_async(f, range(10)) will execute them asynchronously and only block when r.wait() is called so we see HERE and MORE in between but DONE will always be at the end.
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
2
ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
– aman
Mar 10 '16 at 6:59
4
Not quite. You'll notice map will execute in order, but map_async doesn't
– quikst3r
Mar 10 '16 at 23:35
2
Should there be aprint 'DONE'afterr.wait()?
– HBeel
Nov 1 '16 at 14:32
Yes there should be!
– quikst3r
Nov 2 '16 at 19:32
If above example doesn't return different results formapandmap_asyncon first run, try settingrange(500)or something large.
– webelo
Oct 28 '17 at 15:22
|
show 2 more comments
There are four choices to mapping jobs to processes. You have to consider multi-args, concurrency, blocking, and ordering. map and map_asnyc only differ with respect to blocking. map_async is non-blocking where as map is blocking
So let's say you had a function
from multiprocessing import Pool
import time
def f(x):
print x*x
if __name__ == '__main__':
pool = Pool(processes=4)
pool.map(f, range(10))
r = pool.map_async(f, range(10))
# DO STUFF
print 'HERE'
print 'MORE'
r.wait()
print 'DONE'
Example output:
0
1
9
4
16
25
36
49
64
81
0
HERE
1
4
MORE
16
25
36
9
49
64
81
DONE
pool.map(f, range(10)) will wait for all 10 of those function calls to finish so we see all the prints in a row.
r = pool.map_async(f, range(10)) will execute them asynchronously and only block when r.wait() is called so we see HERE and MORE in between but DONE will always be at the end.
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
There are four choices to mapping jobs to processes. You have to consider multi-args, concurrency, blocking, and ordering. map and map_asnyc only differ with respect to blocking. map_async is non-blocking where as map is blocking
So let's say you had a function
from multiprocessing import Pool
import time
def f(x):
print x*x
if __name__ == '__main__':
pool = Pool(processes=4)
pool.map(f, range(10))
r = pool.map_async(f, range(10))
# DO STUFF
print 'HERE'
print 'MORE'
r.wait()
print 'DONE'
Example output:
0
1
9
4
16
25
36
49
64
81
0
HERE
1
4
MORE
16
25
36
9
49
64
81
DONE
pool.map(f, range(10)) will wait for all 10 of those function calls to finish so we see all the prints in a row.
r = pool.map_async(f, range(10)) will execute them asynchronously and only block when r.wait() is called so we see HERE and MORE in between but DONE will always be at the end.
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
edited Nov 25 '18 at 11:35
martineau
68.2k1090183
68.2k1090183
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
answered Mar 10 '16 at 6:41
quikst3rquikst3r
990514
990514
2
ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
– aman
Mar 10 '16 at 6:59
4
Not quite. You'll notice map will execute in order, but map_async doesn't
– quikst3r
Mar 10 '16 at 23:35
2
Should there be aprint 'DONE'afterr.wait()?
– HBeel
Nov 1 '16 at 14:32
Yes there should be!
– quikst3r
Nov 2 '16 at 19:32
If above example doesn't return different results formapandmap_asyncon first run, try settingrange(500)or something large.
– webelo
Oct 28 '17 at 15:22
|
show 2 more comments
2
ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
– aman
Mar 10 '16 at 6:59
4
Not quite. You'll notice map will execute in order, but map_async doesn't
– quikst3r
Mar 10 '16 at 23:35
2
Should there be aprint 'DONE'afterr.wait()?
– HBeel
Nov 1 '16 at 14:32
Yes there should be!
– quikst3r
Nov 2 '16 at 19:32
If above example doesn't return different results formapandmap_asyncon first run, try settingrange(500)or something large.
– webelo
Oct 28 '17 at 15:22
2
2
ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
– aman
Mar 10 '16 at 6:59
ok so if i dont have other tasks to do beside executing the function f over the list, then map and map_async are same
– aman
Mar 10 '16 at 6:59
4
4
Not quite. You'll notice map will execute in order, but map_async doesn't
– quikst3r
Mar 10 '16 at 23:35
Not quite. You'll notice map will execute in order, but map_async doesn't
– quikst3r
Mar 10 '16 at 23:35
2
2
Should there be a
print 'DONE' after r.wait()?– HBeel
Nov 1 '16 at 14:32
Should there be a
print 'DONE' after r.wait()?– HBeel
Nov 1 '16 at 14:32
Yes there should be!
– quikst3r
Nov 2 '16 at 19:32
Yes there should be!
– quikst3r
Nov 2 '16 at 19:32
If above example doesn't return different results for
map and map_async on first run, try setting range(500) or something large.– webelo
Oct 28 '17 at 15:22
If above example doesn't return different results for
map and map_async on first run, try setting range(500) or something large.– webelo
Oct 28 '17 at 15:22
|
show 2 more comments
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f35908987%2fmultiprocessing-map-vs-map-async%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Doesn't
mapreturn only once the map is done (ie synchronously but in parallel), whilemap_asyncreturns right away and allows the mapping to be done in the background (ie asynchronously and in parallel)?– Joachim Isaksson
Mar 10 '16 at 6:37