Laravel scoreboard of more than 1 million users












-2















I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow



This is the architecture of my database:



Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});




Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});


The topics table have than 20 row





    Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();

$table->unique(['user_id', 'topic_id']);

$table->float('timer');
$table->integer('score');
});


The query to make rank for users



        User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});


Any optimization should I make the get the result quicker? Thanks.










share|improve this question




















  • 5





    You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...

    – Namoshek
    Nov 22 '18 at 18:56











  • You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database

    – Alex
    Nov 22 '18 at 19:06











  • @Alex an example please

    – Malek Ben el ouafi
    Nov 22 '18 at 19:07
















-2















I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow



This is the architecture of my database:



Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});




Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});


The topics table have than 20 row





    Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();

$table->unique(['user_id', 'topic_id']);

$table->float('timer');
$table->integer('score');
});


The query to make rank for users



        User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});


Any optimization should I make the get the result quicker? Thanks.










share|improve this question




















  • 5





    You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...

    – Namoshek
    Nov 22 '18 at 18:56











  • You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database

    – Alex
    Nov 22 '18 at 19:06











  • @Alex an example please

    – Malek Ben el ouafi
    Nov 22 '18 at 19:07














-2












-2








-2








I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow



This is the architecture of my database:



Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});




Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});


The topics table have than 20 row





    Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();

$table->unique(['user_id', 'topic_id']);

$table->float('timer');
$table->integer('score');
});


The query to make rank for users



        User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});


Any optimization should I make the get the result quicker? Thanks.










share|improve this question
















I'm working in a biggest application ( more than 1 million users ) and I try to get the ranking of each user in the scoreboard section but had this problem: the result is very very slow



This is the architecture of my database:



Schema::create('users', function (Blueprint $table) {
$table->increments('id');
...
});




Schema::create('topics', function (Blueprint $table) {
$table->increments('id');
...
});


The topics table have than 20 row





    Schema::create('user_scores', function (Blueprint $table) {
$table->increments('id');
$table->integer('user_id')->unsigned();
$table->integer('topic_id')->unsigned();

$table->unique(['user_id', 'topic_id']);

$table->float('timer');
$table->integer('score');
});


The query to make rank for users



        User::where('type',0)->get()->each(function ($user) {
$user->topics= $user->scores->sum('score');
$user->timing= $user->scores->sum('timer');
})->sort(function ($a, $b){
return ($b->topics - $a->topics) == 0
? ($a->timing - $b->timing)
: ($b->topics - $a->topics);
})->values()->each(function($user, $key){
$user->rank = $key +1;
});


Any optimization should I make the get the result quicker? Thanks.







php mysql laravel apache laravel-5






share|improve this question















share|improve this question













share|improve this question




share|improve this question








edited Nov 22 '18 at 19:01







Malek Ben el ouafi

















asked Nov 22 '18 at 18:53









Malek Ben el ouafiMalek Ben el ouafi

418322




418322








  • 5





    You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...

    – Namoshek
    Nov 22 '18 at 18:56











  • You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database

    – Alex
    Nov 22 '18 at 19:06











  • @Alex an example please

    – Malek Ben el ouafi
    Nov 22 '18 at 19:07














  • 5





    You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...

    – Namoshek
    Nov 22 '18 at 18:56











  • You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database

    – Alex
    Nov 22 '18 at 19:06











  • @Alex an example please

    – Malek Ben el ouafi
    Nov 22 '18 at 19:07








5




5





You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...

– Namoshek
Nov 22 '18 at 18:56





You can't just expect everyone to guess what queries you are executing. Please give some examples, otherwise any suggestion will just be a better guess...

– Namoshek
Nov 22 '18 at 18:56













You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database

– Alex
Nov 22 '18 at 19:06





You sort them in PHP arrays, that's why the speed... Try to sort the rows in the database

– Alex
Nov 22 '18 at 19:06













@Alex an example please

– Malek Ben el ouafi
Nov 22 '18 at 19:07





@Alex an example please

– Malek Ben el ouafi
Nov 22 '18 at 19:07












1 Answer
1






active

oldest

votes


















2














As soon as you call get(), all(), find() or first() on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.



What you can do is to improve your query:



User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()


For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.



But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate() instead of get().






share|improve this answer
























  • The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)

    – Malek Ben el ouafi
    Nov 22 '18 at 19:40











  • withCount('scores') will return the number of associated rows in the score table. You can print the query with toSql() instead of get() to have a look yourself.

    – Namoshek
    Nov 22 '18 at 19:45











  • I need to order by the value of sum, not by the number of rows

    – Malek Ben el ouafi
    Nov 22 '18 at 19:47











  • Then just switch the orderBy() statements around.

    – Namoshek
    Nov 22 '18 at 19:48











Your Answer






StackExchange.ifUsing("editor", function () {
StackExchange.using("externalEditor", function () {
StackExchange.using("snippets", function () {
StackExchange.snippets.init();
});
});
}, "code-snippets");

StackExchange.ready(function() {
var channelOptions = {
tags: "".split(" "),
id: "1"
};
initTagRenderer("".split(" "), "".split(" "), channelOptions);

StackExchange.using("externalEditor", function() {
// Have to fire editor after snippets, if snippets enabled
if (StackExchange.settings.snippets.snippetsEnabled) {
StackExchange.using("snippets", function() {
createEditor();
});
}
else {
createEditor();
}
});

function createEditor() {
StackExchange.prepareEditor({
heartbeatType: 'answer',
autoActivateHeartbeat: false,
convertImagesToLinks: true,
noModals: true,
showLowRepImageUploadWarning: true,
reputationToPostImages: 10,
bindNavPrevention: true,
postfix: "",
imageUploader: {
brandingHtml: "Powered by u003ca class="icon-imgur-white" href="https://imgur.com/"u003eu003c/au003e",
contentPolicyHtml: "User contributions licensed under u003ca href="https://creativecommons.org/licenses/by-sa/3.0/"u003ecc by-sa 3.0 with attribution requiredu003c/au003e u003ca href="https://stackoverflow.com/legal/content-policy"u003e(content policy)u003c/au003e",
allowUrls: true
},
onDemand: true,
discardSelector: ".discard-answer"
,immediatelyShowMarkdownHelp:true
});


}
});














draft saved

draft discarded


















StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53436726%2flaravel-scoreboard-of-more-than-1-million-users%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown

























1 Answer
1






active

oldest

votes








1 Answer
1






active

oldest

votes









active

oldest

votes






active

oldest

votes









2














As soon as you call get(), all(), find() or first() on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.



What you can do is to improve your query:



User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()


For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.



But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate() instead of get().






share|improve this answer
























  • The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)

    – Malek Ben el ouafi
    Nov 22 '18 at 19:40











  • withCount('scores') will return the number of associated rows in the score table. You can print the query with toSql() instead of get() to have a look yourself.

    – Namoshek
    Nov 22 '18 at 19:45











  • I need to order by the value of sum, not by the number of rows

    – Malek Ben el ouafi
    Nov 22 '18 at 19:47











  • Then just switch the orderBy() statements around.

    – Namoshek
    Nov 22 '18 at 19:48
















2














As soon as you call get(), all(), find() or first() on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.



What you can do is to improve your query:



User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()


For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.



But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate() instead of get().






share|improve this answer
























  • The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)

    – Malek Ben el ouafi
    Nov 22 '18 at 19:40











  • withCount('scores') will return the number of associated rows in the score table. You can print the query with toSql() instead of get() to have a look yourself.

    – Namoshek
    Nov 22 '18 at 19:45











  • I need to order by the value of sum, not by the number of rows

    – Malek Ben el ouafi
    Nov 22 '18 at 19:47











  • Then just switch the orderBy() statements around.

    – Namoshek
    Nov 22 '18 at 19:48














2












2








2







As soon as you call get(), all(), find() or first() on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.



What you can do is to improve your query:



User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()


For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.



But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate() instead of get().






share|improve this answer













As soon as you call get(), all(), find() or first() on a query builder, you'll ask the Eloquent engine to perform the query and return you the result. So in your case, all the sorting and grouping is performed in memory, which comes with incredibly bad performance.



What you can do is to improve your query:



User::query()
->where('type', 0)
->withCount('scores as topics')
->withCount(['scores as timing' => function ($query) {
$query->selectRaw('SUM(timer)'); // might look weird, but works...
}])
->orderBy('topics', 'desc')
->orderBy('timing', 'desc')
->get()


For the row number (or rank, or however you wanna call it), you might want to search through existing questions and answers. Answering that as well would be too much for this answer, to be honest. Clearly you should not use your approach though, as it will also calculate the row number in memory.



But obviously it is also important what you are doing with the query results. Are you displaying one million rows to the user? If so, the bottleneck will be the browser in the end for sure. You might want to consider using pagination with paginate() instead of get().







share|improve this answer












share|improve this answer



share|improve this answer










answered Nov 22 '18 at 19:24









NamoshekNamoshek

2,9362819




2,9362819













  • The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)

    – Malek Ben el ouafi
    Nov 22 '18 at 19:40











  • withCount('scores') will return the number of associated rows in the score table. You can print the query with toSql() instead of get() to have a look yourself.

    – Namoshek
    Nov 22 '18 at 19:45











  • I need to order by the value of sum, not by the number of rows

    – Malek Ben el ouafi
    Nov 22 '18 at 19:47











  • Then just switch the orderBy() statements around.

    – Namoshek
    Nov 22 '18 at 19:48



















  • The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)

    – Malek Ben el ouafi
    Nov 22 '18 at 19:40











  • withCount('scores') will return the number of associated rows in the score table. You can print the query with toSql() instead of get() to have a look yourself.

    – Namoshek
    Nov 22 '18 at 19:45











  • I need to order by the value of sum, not by the number of rows

    – Malek Ben el ouafi
    Nov 22 '18 at 19:47











  • Then just switch the orderBy() statements around.

    – Namoshek
    Nov 22 '18 at 19:48

















The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)

– Malek Ben el ouafi
Nov 22 '18 at 19:40





The 'withCount' return only the count of row (always 1). I don't display all result I get only top 3, and before and after the connected user ( user-1, user, user+1)

– Malek Ben el ouafi
Nov 22 '18 at 19:40













withCount('scores') will return the number of associated rows in the score table. You can print the query with toSql() instead of get() to have a look yourself.

– Namoshek
Nov 22 '18 at 19:45





withCount('scores') will return the number of associated rows in the score table. You can print the query with toSql() instead of get() to have a look yourself.

– Namoshek
Nov 22 '18 at 19:45













I need to order by the value of sum, not by the number of rows

– Malek Ben el ouafi
Nov 22 '18 at 19:47





I need to order by the value of sum, not by the number of rows

– Malek Ben el ouafi
Nov 22 '18 at 19:47













Then just switch the orderBy() statements around.

– Namoshek
Nov 22 '18 at 19:48





Then just switch the orderBy() statements around.

– Namoshek
Nov 22 '18 at 19:48


















draft saved

draft discarded




















































Thanks for contributing an answer to Stack Overflow!


  • Please be sure to answer the question. Provide details and share your research!

But avoid



  • Asking for help, clarification, or responding to other answers.

  • Making statements based on opinion; back them up with references or personal experience.


To learn more, see our tips on writing great answers.




draft saved


draft discarded














StackExchange.ready(
function () {
StackExchange.openid.initPostLogin('.new-post-login', 'https%3a%2f%2fstackoverflow.com%2fquestions%2f53436726%2flaravel-scoreboard-of-more-than-1-million-users%23new-answer', 'question_page');
}
);

Post as a guest















Required, but never shown





















































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown

































Required, but never shown














Required, but never shown












Required, but never shown







Required, but never shown







Popular posts from this blog

404 Error Contact Form 7 ajax form submitting

How to know if a Active Directory user can login interactively

TypeError: fit_transform() missing 1 required positional argument: 'X'