Python asyncio callback from another thread
0
There are many question on threads and asyncio on stackoverflow but I have not yet found my answer. I have a callback in another thread than the asyncio loop, that callback can't be a coroutine due to external library define the signature and calls it. I would like to call a coroutine from that callback, but I do NOT want to wait or block for that call to finish, since that will hold up the external library. In short how to schedule a coroutine from another thread, to run as soon as possible. Some test code, if the call to f.result() is removed the on_message is not scheduled or we get complaints on that it is not awaited.
def _on_message(self, consumer, message):
f = asyncio.run_coroutine_threadsafe(self.on_message(consumer, message), self.loop)
# don't want this blocking call to result
f.result()
async def on_message(self, consumer, message):
# do real work
pass
python-asyncio
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
|
show 4 more comments
0
There are many question on threads and asyncio on stackoverflow but I have not yet found my answer. I have a callback in another thread than the asyncio loop, that callback can't be a coroutine due to external library define the signature and calls it. I would like to call a coroutine from that callback, but I do NOT want to wait or block for that call to finish, since that will hold up the external library. In short how to schedule a coroutine from another thread, to run as soon as possible. Some test code, if the call to f.result() is removed the on_message is not scheduled or we get complaints on that it is not awaited.
def _on_message(self, consumer, message):
f = asyncio.run_coroutine_threadsafe(self.on_message(consumer, message), self.loop)
# don't want this blocking call to result
f.result()
async def on_message(self, consumer, message):
# do real work
pass
python-asyncio
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
1
Ifon_messageis not scheduled (i.e.on_messagedoesn't run at all), that likely means that something is wrong with your event loop, e.g. it's not running or is blocked by something else. Can you create a minimal runnable example that shows the issue?
– user4815162342
Nov 21 '18 at 15:04
Have you tried torun_in_executor? docs.python.org/3/library/… . What kind of external library is this ? Is it a web framework ?
– JoseKilo
Nov 22 '18 at 10:10
@JoseKilorun_in_executorwill not help the OP because they want to schedule a coroutine to execute from a sync callback.run_in_executoris when you need the inverse, executing sync code from inside a coroutine.run_coroutine_threadsafeis designed for exactly the OP's use case.
– user4815162342
Nov 22 '18 at 13:56
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result(). The asyncio loop is very busy, so it might be that it would get scheduled if not a lot of other coroutines were waited for, which are scheduled. The external libraries I have these issues with are for example paho-mqtt and pulsar-client, both allow registering callbacks for subscriptions.
– Harald Gustafsson
Nov 22 '18 at 18:50
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result() Are you sure of that? Have you tried replacingf.result()withtime.sleep(5)(or however long it takes to get to the result)?
– user4815162342
Nov 24 '18 at 8:36
|
show 4 more comments
0
0
0
There are many question on threads and asyncio on stackoverflow but I have not yet found my answer. I have a callback in another thread than the asyncio loop, that callback can't be a coroutine due to external library define the signature and calls it. I would like to call a coroutine from that callback, but I do NOT want to wait or block for that call to finish, since that will hold up the external library. In short how to schedule a coroutine from another thread, to run as soon as possible. Some test code, if the call to f.result() is removed the on_message is not scheduled or we get complaints on that it is not awaited.
def _on_message(self, consumer, message):
f = asyncio.run_coroutine_threadsafe(self.on_message(consumer, message), self.loop)
# don't want this blocking call to result
f.result()
async def on_message(self, consumer, message):
# do real work
pass
python-asyncio
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
There are many question on threads and asyncio on stackoverflow but I have not yet found my answer. I have a callback in another thread than the asyncio loop, that callback can't be a coroutine due to external library define the signature and calls it. I would like to call a coroutine from that callback, but I do NOT want to wait or block for that call to finish, since that will hold up the external library. In short how to schedule a coroutine from another thread, to run as soon as possible. Some test code, if the call to f.result() is removed the on_message is not scheduled or we get complaints on that it is not awaited.
def _on_message(self, consumer, message):
f = asyncio.run_coroutine_threadsafe(self.on_message(consumer, message), self.loop)
# don't want this blocking call to result
f.result()
async def on_message(self, consumer, message):
# do real work
pass
python-asyncio
python-asyncio
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
asked Nov 21 '18 at 13:43
Harald Gustafsson
344
344
1
Ifon_messageis not scheduled (i.e.on_messagedoesn't run at all), that likely means that something is wrong with your event loop, e.g. it's not running or is blocked by something else. Can you create a minimal runnable example that shows the issue?
– user4815162342
Nov 21 '18 at 15:04
Have you tried torun_in_executor? docs.python.org/3/library/… . What kind of external library is this ? Is it a web framework ?
– JoseKilo
Nov 22 '18 at 10:10
@JoseKilorun_in_executorwill not help the OP because they want to schedule a coroutine to execute from a sync callback.run_in_executoris when you need the inverse, executing sync code from inside a coroutine.run_coroutine_threadsafeis designed for exactly the OP's use case.
– user4815162342
Nov 22 '18 at 13:56
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result(). The asyncio loop is very busy, so it might be that it would get scheduled if not a lot of other coroutines were waited for, which are scheduled. The external libraries I have these issues with are for example paho-mqtt and pulsar-client, both allow registering callbacks for subscriptions.
– Harald Gustafsson
Nov 22 '18 at 18:50
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result() Are you sure of that? Have you tried replacingf.result()withtime.sleep(5)(or however long it takes to get to the result)?
– user4815162342
Nov 24 '18 at 8:36
|
show 4 more comments
1
Ifon_messageis not scheduled (i.e.on_messagedoesn't run at all), that likely means that something is wrong with your event loop, e.g. it's not running or is blocked by something else. Can you create a minimal runnable example that shows the issue?
– user4815162342
Nov 21 '18 at 15:04
Have you tried torun_in_executor? docs.python.org/3/library/… . What kind of external library is this ? Is it a web framework ?
– JoseKilo
Nov 22 '18 at 10:10
@JoseKilorun_in_executorwill not help the OP because they want to schedule a coroutine to execute from a sync callback.run_in_executoris when you need the inverse, executing sync code from inside a coroutine.run_coroutine_threadsafeis designed for exactly the OP's use case.
– user4815162342
Nov 22 '18 at 13:56
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result(). The asyncio loop is very busy, so it might be that it would get scheduled if not a lot of other coroutines were waited for, which are scheduled. The external libraries I have these issues with are for example paho-mqtt and pulsar-client, both allow registering callbacks for subscriptions.
– Harald Gustafsson
Nov 22 '18 at 18:50
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result() Are you sure of that? Have you tried replacingf.result()withtime.sleep(5)(or however long it takes to get to the result)?
– user4815162342
Nov 24 '18 at 8:36
1
1
If
on_message is not scheduled (i.e. on_message doesn't run at all), that likely means that something is wrong with your event loop, e.g. it's not running or is blocked by something else. Can you create a minimal runnable example that shows the issue?– user4815162342
Nov 21 '18 at 15:04
If
on_message is not scheduled (i.e. on_message doesn't run at all), that likely means that something is wrong with your event loop, e.g. it's not running or is blocked by something else. Can you create a minimal runnable example that shows the issue?– user4815162342
Nov 21 '18 at 15:04
Have you tried to
run_in_executor ? docs.python.org/3/library/… . What kind of external library is this ? Is it a web framework ?– JoseKilo
Nov 22 '18 at 10:10
Have you tried to
run_in_executor ? docs.python.org/3/library/… . What kind of external library is this ? Is it a web framework ?– JoseKilo
Nov 22 '18 at 10:10
@JoseKilo
run_in_executor will not help the OP because they want to schedule a coroutine to execute from a sync callback. run_in_executor is when you need the inverse, executing sync code from inside a coroutine. run_coroutine_threadsafe is designed for exactly the OP's use case.– user4815162342
Nov 22 '18 at 13:56
@JoseKilo
run_in_executor will not help the OP because they want to schedule a coroutine to execute from a sync callback. run_in_executor is when you need the inverse, executing sync code from inside a coroutine. run_coroutine_threadsafe is designed for exactly the OP's use case.– user4815162342
Nov 22 '18 at 13:56
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result(). The asyncio loop is very busy, so it might be that it would get scheduled if not a lot of other coroutines were waited for, which are scheduled. The external libraries I have these issues with are for example paho-mqtt and pulsar-client, both allow registering callbacks for subscriptions.
– Harald Gustafsson
Nov 22 '18 at 18:50
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result(). The asyncio loop is very busy, so it might be that it would get scheduled if not a lot of other coroutines were waited for, which are scheduled. The external libraries I have these issues with are for example paho-mqtt and pulsar-client, both allow registering callbacks for subscriptions.
– Harald Gustafsson
Nov 22 '18 at 18:50
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result() Are you sure of that? Have you tried replacing
f.result() with time.sleep(5) (or however long it takes to get to the result)?– user4815162342
Nov 24 '18 at 8:36
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result() Are you sure of that? Have you tried replacing
f.result() with time.sleep(5) (or however long it takes to get to the result)?– user4815162342
Nov 24 '18 at 8:36
|
show 4 more comments
0
active
oldest
votes
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53413444%2fpython-asyncio-callback-from-another-thread%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
0
active
oldest
votes
0
active
oldest
votes
active
oldest
votes
active
oldest
votes
draft saved
draft discarded
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
draft saved
draft discarded
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53413444%2fpython-asyncio-callback-from-another-thread%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
If
on_messageis not scheduled (i.e.on_messagedoesn't run at all), that likely means that something is wrong with your event loop, e.g. it's not running or is blocked by something else. Can you create a minimal runnable example that shows the issue?– user4815162342
Nov 21 '18 at 15:04
Have you tried to
run_in_executor? docs.python.org/3/library/… . What kind of external library is this ? Is it a web framework ?– JoseKilo
Nov 22 '18 at 10:10
@JoseKilo
run_in_executorwill not help the OP because they want to schedule a coroutine to execute from a sync callback.run_in_executoris when you need the inverse, executing sync code from inside a coroutine.run_coroutine_threadsafeis designed for exactly the OP's use case.– user4815162342
Nov 22 '18 at 13:56
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result(). The asyncio loop is very busy, so it might be that it would get scheduled if not a lot of other coroutines were waited for, which are scheduled. The external libraries I have these issues with are for example paho-mqtt and pulsar-client, both allow registering callbacks for subscriptions.
– Harald Gustafsson
Nov 22 '18 at 18:50
Sorry currently I don't have a small example, but from what I can see the on_message does not even get scheduled without the call to result() Are you sure of that? Have you tried replacing
f.result()withtime.sleep(5)(or however long it takes to get to the result)?– user4815162342
Nov 24 '18 at 8:36