Are Django settings shared across uwsgi workers?
up vote
4
down vote
favorite
I have a Django app, with a setting (in my settings.py file) that's populated dynamically in my App Config's ready() function. Ie in settings.py I have:
POPULATE_THIS = None
and then in apps.py in ready I have:
def ready(self):
if POPULATE_THIS is None:
POPULATE_THIS = ... some code which instantiates an object I need that's effectively a singleton ...
This seems to work ok. But I'm now at the point where rather than just running the dev server locally (ie python manage.py runserver), I'm now running the Django app through uwsgi (proxied behind nginx), and uwsgi is configured to run 10 worker processes (ie my uwsgi ini file has processes = 10 and threads = 1).
I'm seeing evidence that even though there are 10 uwsgi processes , ready() is still called exactly once on app startup and the value of POPULATE_THIS is the same across all workers (doing a str on it is giving the same memory address).
My question: How is that value shared across the uwsgi processes, as I thought separate processes are distinct and do not share any memory? And am I correct in assuming that ready() is going to be called once per app startup (ie when uwsgi itself spins up), and not once per uwsgi worker process startup?
This answer (Multiple server processes using nginx and uWSGI) on a different question seems to indicate that some data is shared across workers, but I can't seem to find any official docs that indicate what exactly is shared and how, specifically with respect to Django settings, so some explanation/details would be much appreciated.
django python-3.x uwsgi
add a comment |
up vote
4
down vote
favorite
I have a Django app, with a setting (in my settings.py file) that's populated dynamically in my App Config's ready() function. Ie in settings.py I have:
POPULATE_THIS = None
and then in apps.py in ready I have:
def ready(self):
if POPULATE_THIS is None:
POPULATE_THIS = ... some code which instantiates an object I need that's effectively a singleton ...
This seems to work ok. But I'm now at the point where rather than just running the dev server locally (ie python manage.py runserver), I'm now running the Django app through uwsgi (proxied behind nginx), and uwsgi is configured to run 10 worker processes (ie my uwsgi ini file has processes = 10 and threads = 1).
I'm seeing evidence that even though there are 10 uwsgi processes , ready() is still called exactly once on app startup and the value of POPULATE_THIS is the same across all workers (doing a str on it is giving the same memory address).
My question: How is that value shared across the uwsgi processes, as I thought separate processes are distinct and do not share any memory? And am I correct in assuming that ready() is going to be called once per app startup (ie when uwsgi itself spins up), and not once per uwsgi worker process startup?
This answer (Multiple server processes using nginx and uWSGI) on a different question seems to indicate that some data is shared across workers, but I can't seem to find any official docs that indicate what exactly is shared and how, specifically with respect to Django settings, so some explanation/details would be much appreciated.
django python-3.x uwsgi
1
Not quite sure about the answer, just some thoughts: first, the fact that the varPOPULATE_THISpoints to the same memory address in every uwsgi process does not mean that they share memory since this address is virtual (not physical) and every process has the full (with exceptions) virtual memory adresses space available. One possible explanation for both of your observations would be that uwsgi forks the processes AFTER every app has been initialized. I don't know if that is the case but it could be...
– ivissani
Nov 19 at 23:38
1
As far as I know, that value can not be shared across processses, but your tests suggest otherwise. Maybe as @ivissani suggested, uwsgi may be forking processes after initialization. You can write a view method that assigns a random value to POPULATE_THIS and another one to return it, and call these methods several times through requests to see how they behave.
– Ozgur Akcali
Nov 20 at 10:58
^^ Great tip, I did this and what I'm seeing is it looks like it does fork the process, so each worker gets an independent copy of the settings. If I update a settings value in one of the workers, that change isn't visible in the other(s).
– Adam Parkin
Nov 20 at 16:15
add a comment |
up vote
4
down vote
favorite
up vote
4
down vote
favorite
I have a Django app, with a setting (in my settings.py file) that's populated dynamically in my App Config's ready() function. Ie in settings.py I have:
POPULATE_THIS = None
and then in apps.py in ready I have:
def ready(self):
if POPULATE_THIS is None:
POPULATE_THIS = ... some code which instantiates an object I need that's effectively a singleton ...
This seems to work ok. But I'm now at the point where rather than just running the dev server locally (ie python manage.py runserver), I'm now running the Django app through uwsgi (proxied behind nginx), and uwsgi is configured to run 10 worker processes (ie my uwsgi ini file has processes = 10 and threads = 1).
I'm seeing evidence that even though there are 10 uwsgi processes , ready() is still called exactly once on app startup and the value of POPULATE_THIS is the same across all workers (doing a str on it is giving the same memory address).
My question: How is that value shared across the uwsgi processes, as I thought separate processes are distinct and do not share any memory? And am I correct in assuming that ready() is going to be called once per app startup (ie when uwsgi itself spins up), and not once per uwsgi worker process startup?
This answer (Multiple server processes using nginx and uWSGI) on a different question seems to indicate that some data is shared across workers, but I can't seem to find any official docs that indicate what exactly is shared and how, specifically with respect to Django settings, so some explanation/details would be much appreciated.
django python-3.x uwsgi
I have a Django app, with a setting (in my settings.py file) that's populated dynamically in my App Config's ready() function. Ie in settings.py I have:
POPULATE_THIS = None
and then in apps.py in ready I have:
def ready(self):
if POPULATE_THIS is None:
POPULATE_THIS = ... some code which instantiates an object I need that's effectively a singleton ...
This seems to work ok. But I'm now at the point where rather than just running the dev server locally (ie python manage.py runserver), I'm now running the Django app through uwsgi (proxied behind nginx), and uwsgi is configured to run 10 worker processes (ie my uwsgi ini file has processes = 10 and threads = 1).
I'm seeing evidence that even though there are 10 uwsgi processes , ready() is still called exactly once on app startup and the value of POPULATE_THIS is the same across all workers (doing a str on it is giving the same memory address).
My question: How is that value shared across the uwsgi processes, as I thought separate processes are distinct and do not share any memory? And am I correct in assuming that ready() is going to be called once per app startup (ie when uwsgi itself spins up), and not once per uwsgi worker process startup?
This answer (Multiple server processes using nginx and uWSGI) on a different question seems to indicate that some data is shared across workers, but I can't seem to find any official docs that indicate what exactly is shared and how, specifically with respect to Django settings, so some explanation/details would be much appreciated.
django python-3.x uwsgi
django python-3.x uwsgi
asked Nov 19 at 23:23
Adam Parkin
7,28794569
7,28794569
1
Not quite sure about the answer, just some thoughts: first, the fact that the varPOPULATE_THISpoints to the same memory address in every uwsgi process does not mean that they share memory since this address is virtual (not physical) and every process has the full (with exceptions) virtual memory adresses space available. One possible explanation for both of your observations would be that uwsgi forks the processes AFTER every app has been initialized. I don't know if that is the case but it could be...
– ivissani
Nov 19 at 23:38
1
As far as I know, that value can not be shared across processses, but your tests suggest otherwise. Maybe as @ivissani suggested, uwsgi may be forking processes after initialization. You can write a view method that assigns a random value to POPULATE_THIS and another one to return it, and call these methods several times through requests to see how they behave.
– Ozgur Akcali
Nov 20 at 10:58
^^ Great tip, I did this and what I'm seeing is it looks like it does fork the process, so each worker gets an independent copy of the settings. If I update a settings value in one of the workers, that change isn't visible in the other(s).
– Adam Parkin
Nov 20 at 16:15
add a comment |
1
Not quite sure about the answer, just some thoughts: first, the fact that the varPOPULATE_THISpoints to the same memory address in every uwsgi process does not mean that they share memory since this address is virtual (not physical) and every process has the full (with exceptions) virtual memory adresses space available. One possible explanation for both of your observations would be that uwsgi forks the processes AFTER every app has been initialized. I don't know if that is the case but it could be...
– ivissani
Nov 19 at 23:38
1
As far as I know, that value can not be shared across processses, but your tests suggest otherwise. Maybe as @ivissani suggested, uwsgi may be forking processes after initialization. You can write a view method that assigns a random value to POPULATE_THIS and another one to return it, and call these methods several times through requests to see how they behave.
– Ozgur Akcali
Nov 20 at 10:58
^^ Great tip, I did this and what I'm seeing is it looks like it does fork the process, so each worker gets an independent copy of the settings. If I update a settings value in one of the workers, that change isn't visible in the other(s).
– Adam Parkin
Nov 20 at 16:15
1
1
Not quite sure about the answer, just some thoughts: first, the fact that the var
POPULATE_THIS points to the same memory address in every uwsgi process does not mean that they share memory since this address is virtual (not physical) and every process has the full (with exceptions) virtual memory adresses space available. One possible explanation for both of your observations would be that uwsgi forks the processes AFTER every app has been initialized. I don't know if that is the case but it could be...– ivissani
Nov 19 at 23:38
Not quite sure about the answer, just some thoughts: first, the fact that the var
POPULATE_THIS points to the same memory address in every uwsgi process does not mean that they share memory since this address is virtual (not physical) and every process has the full (with exceptions) virtual memory adresses space available. One possible explanation for both of your observations would be that uwsgi forks the processes AFTER every app has been initialized. I don't know if that is the case but it could be...– ivissani
Nov 19 at 23:38
1
1
As far as I know, that value can not be shared across processses, but your tests suggest otherwise. Maybe as @ivissani suggested, uwsgi may be forking processes after initialization. You can write a view method that assigns a random value to POPULATE_THIS and another one to return it, and call these methods several times through requests to see how they behave.
– Ozgur Akcali
Nov 20 at 10:58
As far as I know, that value can not be shared across processses, but your tests suggest otherwise. Maybe as @ivissani suggested, uwsgi may be forking processes after initialization. You can write a view method that assigns a random value to POPULATE_THIS and another one to return it, and call these methods several times through requests to see how they behave.
– Ozgur Akcali
Nov 20 at 10:58
^^ Great tip, I did this and what I'm seeing is it looks like it does fork the process, so each worker gets an independent copy of the settings. If I update a settings value in one of the workers, that change isn't visible in the other(s).
– Adam Parkin
Nov 20 at 16:15
^^ Great tip, I did this and what I'm seeing is it looks like it does fork the process, so each worker gets an independent copy of the settings. If I update a settings value in one of the workers, that change isn't visible in the other(s).
– Adam Parkin
Nov 20 at 16:15
add a comment |
1 Answer
1
active
oldest
votes
up vote
0
down vote
Exactly.
It seems that uwsgi only spam processes of the django application itself, therefore all the functions ready will be called only once, during the first run.
add a comment |
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
up vote
0
down vote
Exactly.
It seems that uwsgi only spam processes of the django application itself, therefore all the functions ready will be called only once, during the first run.
add a comment |
up vote
0
down vote
Exactly.
It seems that uwsgi only spam processes of the django application itself, therefore all the functions ready will be called only once, during the first run.
add a comment |
up vote
0
down vote
up vote
0
down vote
Exactly.
It seems that uwsgi only spam processes of the django application itself, therefore all the functions ready will be called only once, during the first run.
Exactly.
It seems that uwsgi only spam processes of the django application itself, therefore all the functions ready will be called only once, during the first run.
answered Nov 19 at 23:34
Marcelo Duarte Fernandes
111
111
add a comment |
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Some of your past answers have not been well-received, and you're in danger of being blocked from answering.
Please pay close attention to the following guidance:
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53384081%2fare-django-settings-shared-across-uwsgi-workers%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
1
Not quite sure about the answer, just some thoughts: first, the fact that the var
POPULATE_THISpoints to the same memory address in every uwsgi process does not mean that they share memory since this address is virtual (not physical) and every process has the full (with exceptions) virtual memory adresses space available. One possible explanation for both of your observations would be that uwsgi forks the processes AFTER every app has been initialized. I don't know if that is the case but it could be...– ivissani
Nov 19 at 23:38
1
As far as I know, that value can not be shared across processses, but your tests suggest otherwise. Maybe as @ivissani suggested, uwsgi may be forking processes after initialization. You can write a view method that assigns a random value to POPULATE_THIS and another one to return it, and call these methods several times through requests to see how they behave.
– Ozgur Akcali
Nov 20 at 10:58
^^ Great tip, I did this and what I'm seeing is it looks like it does fork the process, so each worker gets an independent copy of the settings. If I update a settings value in one of the workers, that change isn't visible in the other(s).
– Adam Parkin
Nov 20 at 16:15