Why would CPython logging use a Lock for each handler rather than one Lock per Logger?
While developing my own logging library, I studied the source code of the standard logging module of CPython.
One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.
I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.
What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?
I see two main advantages:
- Better performances as there is only one lock acquired and released
- Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence
H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1)to happen)
In my personal logging library, can I safely do something like this per logger:
with self.lock:
for handler in self.handlers:
handler.handle(message) # No lock used in "handle()"
Or am I overlooking something else?
python multithreading logging locks
asked 3 hours ago
Delgan
1166
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
add a comment |
While developing my own logging library, I studied the source code of the standard logging module of CPython.
One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.
I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.
What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?
I see two main advantages:
- Better performances as there is only one lock acquired and released
- Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence
H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1)to happen)
In my personal logging library, can I safely do something like this per logger:
with self.lock:
for handler in self.handlers:
handler.handle(message) # No lock used in "handle()"
Or am I overlooking something else?
python multithreading logging locks
asked 3 hours ago
Delgan
1166
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago
add a comment |
While developing my own logging library, I studied the source code of the standard logging module of CPython.
One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.
I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.
What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?
I see two main advantages:
- Better performances as there is only one lock acquired and released
- Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence
H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1)to happen)
In my personal logging library, can I safely do something like this per logger:
with self.lock:
for handler in self.handlers:
handler.handle(message) # No lock used in "handle()"
Or am I overlooking something else?
python multithreading logging locks
asked 3 hours ago
Delgan
1166
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
While developing my own logging library, I studied the source code of the standard logging module of CPython.
One of its features is that handlers are thread-safe. One can write logs to a file from multiple threads without the risk that the lines are corrupted.
I digged into the sources of the logging module and I figured out that to ensure thread-safety, each handler uses its own Lock that it acquire and release at each logging call. You can see this here.
What is the point of doing this rather than using a single lock surrounding the call of handlers globally? For example, why not acquire the lock in the callHandlers() function?
I see two main advantages:
- Better performances as there is only one lock acquired and released
- Ensure that logged messages are displayed in the same order in all handlers (thinks for example of two handlers H1 and H2 and two threads logging respectively M1 and M2, this avoid the possibility of the sequence
H1.handle(M1) -> H1.handle(M2) -> H2.handle(M2) -> H2.handle(M1)to happen)
In my personal logging library, can I safely do something like this per logger:
with self.lock:
for handler in self.handlers:
handler.handle(message) # No lock used in "handle()"
Or am I overlooking something else?
python multithreading logging locks
python multithreading logging locks
asked 3 hours ago
Delgan
1166
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 hours ago
Delgan
1166
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
edited 1 hour ago
asked 3 hours ago
Delgan
1166
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
asked 3 hours ago
Delgan
1166
asked 3 hours ago
Delgan
1166
1166
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
New contributor
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Delgan is a new contributor to this site. Take care in asking for clarification, commenting, and answering.
Check out our Code of Conduct.
Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago
add a comment |
Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago
Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago
Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago
add a comment |
1 Answer
1
active
oldest
votes
Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).
Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.
answered 2 hours ago
Lie Ryan
6,45811825
I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doinglogger.log()anyway, right?
– Delgan
2 hours ago
Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
2 hours ago
Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago
And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago
add a comment |
Your Answer
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "131"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: false,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: null,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Delgan is a new contributor. Be nice, and check out our Code of Conduct.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsoftwareengineering.stackexchange.com%2fquestions%2f384505%2fwhy-would-cpython-logging-use-a-lock-for-each-handler-rather-than-one-lock-per-l%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).
Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.
answered 2 hours ago
Lie Ryan
6,45811825
I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doinglogger.log()anyway, right?
– Delgan
2 hours ago
Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
2 hours ago
Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago
And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago
add a comment |
Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).
Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.
answered 2 hours ago
Lie Ryan
6,45811825
I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doinglogger.log()anyway, right?
– Delgan
2 hours ago
Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
2 hours ago
Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago
And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago
add a comment |
Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).
Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.
answered 2 hours ago
Lie Ryan
6,45811825
Logging handlers can be slow, especially if you do remote logging to another machine, using for example SocketHandler/SMTPHandler/HTTPHandler, or to third party service logging handlers. If the entire logging library uses a single lock, then logging to slow handlers (e.g. email to admin on exception) will block logging to fast handlers (e.g. local files logs).
Additionally, some logging handlers may not require locks at all, for example external logging may use timestamp to order logs instead of implicitly by serializing the handling.
answered 2 hours ago
Lie Ryan
6,45811825
answered 2 hours ago
Lie Ryan
6,45811825
answered 2 hours ago
Lie Ryan
6,45811825
answered 2 hours ago
Lie Ryan
6,45811825
6,45811825
I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doinglogger.log()anyway, right?
– Delgan
2 hours ago
Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
2 hours ago
Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago
And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago
add a comment |
I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doinglogger.log()anyway, right?
– Delgan
2 hours ago
Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
2 hours ago
Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago
And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago
I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing
logger.log() anyway, right?– Delgan
2 hours ago
I don't mean using one lock for "the entire logging library", I mean using one lock per logger. If H1 is slow and H2 is fast, but both are attached to the same logger, then H1 will block H2 while doing
logger.log() anyway, right?– Delgan
2 hours ago
Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
2 hours ago
Also, locking on the loggers would probably not achieve thread safety as a single handler can be configured as a handle target for multiple loggers. If you lock on the loggers, then multiple threads logging to multiple logger that logs to the same handler will be indeterministic.
– Lie Ryan
2 hours ago
Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago
Oh, yeah, I did not think of the fact that handlers could be re-used across multiple loggers.
– Delgan
1 hour ago
And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago
And putting your first argument in others words: one lock per handler allows "parallelization" of logs handlings. For example, if using only one lock, if H1 and H2 are both slow, and if two threads need to log a message at the same time, the second thread to acquire the lock will need to wait for the first thread to emit on both H1 and H2, before logging itself to H1 + H2. Using one lock per handler, this allows second thread to emit on H1 as soon as possible, while in the same time, the other thread log to H2.
– Delgan
1 hour ago
add a comment |
Delgan is a new contributor. Be nice, and check out our Code of Conduct.
Delgan is a new contributor. Be nice, and check out our Code of Conduct.
Delgan is a new contributor. Be nice, and check out our Code of Conduct.
Delgan is a new contributor. Be nice, and check out our Code of Conduct.
Thanks for contributing an answer to Software Engineering Stack Exchange!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fsoftwareengineering.stackexchange.com%2fquestions%2f384505%2fwhy-would-cpython-logging-use-a-lock-for-each-handler-rather-than-one-lock-per-l%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Please, explain downvote so I can improve my questions in the future. What did I do wrong? Thanks!
– Delgan
1 hour ago