Laravel scoreboard of more than 1 million users
I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow
This is the architecture of my database:
Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});
Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});
The topics table have than 20 row
Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();
$table->unique(['user_id', 'topic_id']);
$table->float('timer');
$table->integer('score');
});
The query to make rank for users
User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});
Any optimization should I make the get the result quicker? Thanks.
php mysql laravel apache laravel-5
add a comment |
I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow
This is the architecture of my database:
Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});
Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});
The topics table have than 20 row
Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();
$table->unique(['user_id', 'topic_id']);
$table->float('timer');
$table->integer('score');
});
The query to make rank for users
User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});
Any optimization should I make the get the result quicker? Thanks.
php mysql laravel apache laravel-5
5
You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...
– Namoshek
Nov 22 '18 at 18:56
You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database
– Alex
Nov 22 '18 at 19:06
@Alex an example please
– Malek Ben el ouafi
Nov 22 '18 at 19:07
add a comment |
I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow
This is the architecture of my database:
Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});
Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});
The topics table have than 20 row
Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();
$table->unique(['user_id', 'topic_id']);
$table->float('timer');
$table->integer('score');
});
The query to make rank for users
User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});
Any optimization should I make the get the result quicker? Thanks.
php mysql laravel apache laravel-5
I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow
This is the architecture of my database:
Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});
Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});
The topics table have than 20 row
Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();
$table->unique(['user_id', 'topic_id']);
$table->float('timer');
$table->integer('score');
});
The query to make rank for users
User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});
Any optimization should I make the get the result quicker? Thanks.
php mysql laravel apache laravel-5
php mysql laravel apache laravel-5
edited Nov 22 '18 at 19:01
Malek Ben el ouafi
asked Nov 22 '18 at 18:53
Malek Ben el ouafiMalek Ben el ouafi
418322
418322
5
You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...
– Namoshek
Nov 22 '18 at 18:56
You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database
– Alex
Nov 22 '18 at 19:06
@Alex an example please
– Malek Ben el ouafi
Nov 22 '18 at 19:07
add a comment |
5
You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...
– Namoshek
Nov 22 '18 at 18:56
You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database
– Alex
Nov 22 '18 at 19:06
@Alex an example please
– Malek Ben el ouafi
Nov 22 '18 at 19:07
5
5
You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...
– Namoshek
Nov 22 '18 at 18:56
You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...
– Namoshek
Nov 22 '18 at 18:56
You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database
– Alex
Nov 22 '18 at 19:06
You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database
– Alex
Nov 22 '18 at 19:06
@Alex an example please
– Malek Ben el ouafi
Nov 22 '18 at 19:07
@Alex an example please
– Malek Ben el ouafi
Nov 22 '18 at 19:07
add a comment |
1 Answer
1
active
oldest
votes
As soon as you call get()
, all()
, find()
or first()
on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.
What you can do is to improve your query:
User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()
For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.
But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate()
instead of get()
.
The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)
– Malek Ben el ouafi
Nov 22 '18 at 19:40
withCount('scores')
will return the number of associated rows in thescore
table. You can print the query withtoSql()
instead ofget()
to have a look yourself.
– Namoshek
Nov 22 '18 at 19:45
I need to order by the value of sum, not by the number of rows
– Malek Ben el ouafi
Nov 22 '18 at 19:47
Then just switch theorderBy()
statements around.
– Namoshek
Nov 22 '18 at 19:48
add a comment |
Your Answer
StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");
StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);
StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});
function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});
}
});
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53436726%2flaravel-scoreboard-of-more-than-1-million-users%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
1 Answer
1
active
oldest
votes
1 Answer
1
active
oldest
votes
active
oldest
votes
active
oldest
votes
As soon as you call get()
, all()
, find()
or first()
on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.
What you can do is to improve your query:
User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()
For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.
But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate()
instead of get()
.
The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)
– Malek Ben el ouafi
Nov 22 '18 at 19:40
withCount('scores')
will return the number of associated rows in thescore
table. You can print the query withtoSql()
instead ofget()
to have a look yourself.
– Namoshek
Nov 22 '18 at 19:45
I need to order by the value of sum, not by the number of rows
– Malek Ben el ouafi
Nov 22 '18 at 19:47
Then just switch theorderBy()
statements around.
– Namoshek
Nov 22 '18 at 19:48
add a comment |
As soon as you call get()
, all()
, find()
or first()
on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.
What you can do is to improve your query:
User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()
For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.
But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate()
instead of get()
.
The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)
– Malek Ben el ouafi
Nov 22 '18 at 19:40
withCount('scores')
will return the number of associated rows in thescore
table. You can print the query withtoSql()
instead ofget()
to have a look yourself.
– Namoshek
Nov 22 '18 at 19:45
I need to order by the value of sum, not by the number of rows
– Malek Ben el ouafi
Nov 22 '18 at 19:47
Then just switch theorderBy()
statements around.
– Namoshek
Nov 22 '18 at 19:48
add a comment |
As soon as you call get()
, all()
, find()
or first()
on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.
What you can do is to improve your query:
User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()
For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.
But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate()
instead of get()
.
As soon as you call get()
, all()
, find()
or first()
on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.
What you can do is to improve your query:
User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()
For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.
But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate()
instead of get()
.
answered Nov 22 '18 at 19:24
NamoshekNamoshek
2,9362819
2,9362819
The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)
– Malek Ben el ouafi
Nov 22 '18 at 19:40
withCount('scores')
will return the number of associated rows in thescore
table. You can print the query withtoSql()
instead ofget()
to have a look yourself.
– Namoshek
Nov 22 '18 at 19:45
I need to order by the value of sum, not by the number of rows
– Malek Ben el ouafi
Nov 22 '18 at 19:47
Then just switch theorderBy()
statements around.
– Namoshek
Nov 22 '18 at 19:48
add a comment |
The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)
– Malek Ben el ouafi
Nov 22 '18 at 19:40
withCount('scores')
will return the number of associated rows in thescore
table. You can print the query withtoSql()
instead ofget()
to have a look yourself.
– Namoshek
Nov 22 '18 at 19:45
I need to order by the value of sum, not by the number of rows
– Malek Ben el ouafi
Nov 22 '18 at 19:47
Then just switch theorderBy()
statements around.
– Namoshek
Nov 22 '18 at 19:48
The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)
– Malek Ben el ouafi
Nov 22 '18 at 19:40
The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)
– Malek Ben el ouafi
Nov 22 '18 at 19:40
withCount('scores')
will return the number of associated rows in the score
table. You can print the query with toSql()
instead of get()
to have a look yourself.– Namoshek
Nov 22 '18 at 19:45
withCount('scores')
will return the number of associated rows in the score
table. You can print the query with toSql()
instead of get()
to have a look yourself.– Namoshek
Nov 22 '18 at 19:45
I need to order by the value of sum, not by the number of rows
– Malek Ben el ouafi
Nov 22 '18 at 19:47
I need to order by the value of sum, not by the number of rows
– Malek Ben el ouafi
Nov 22 '18 at 19:47
Then just switch the
orderBy()
statements around.– Namoshek
Nov 22 '18 at 19:48
Then just switch the
orderBy()
statements around.– Namoshek
Nov 22 '18 at 19:48
add a comment |
Thanks for contributing an answer to Stack Overflow!
- Please be sure to answer the question. Provide details and share your research!
But avoid …
- Asking for help, clarification, or responding to other answers.
- Making statements based on opinion; back them up with references or personal experience.
To learn more, see our tips on writing great answers.
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53436726%2flaravel-scoreboard-of-more-than-1-million-users%23new-answer', 'question_page');
}
);
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Sign up or log in
StackExchange.ready(function () {
StackExchange.helpers.onClickDraftSave('#login-link');
});
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Sign up using Google
Sign up using Facebook
Sign up using Email and Password
Post as a guest
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
Required, but never shown
5
You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...
– Namoshek
Nov 22 '18 at 18:56
You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database
– Alex
Nov 22 '18 at 19:06
@Alex an example please
– Malek Ben el ouafi
Nov 22 '18 at 19:07