Tukukkk

I've a query that takes about 18 seconds to finish:

THE QUERY:

SELECT YEAR(c.date), MONTH(c.date), p.district_id, COUNT(p.owner_id)

FROM commission c

  INNER JOIN partner p ON c.customer_id = p.id

WHERE (c.date BETWEEN '2018-01-01' AND '2018-12-31')

  AND (c.company_id = 90)

  AND (c.source = 'ACTUAL')

  AND (p.id IN (3062, 3063, 3064, 3065, 3066, 3067, 3068, 3069, 3070, 3071,

    3072, 3073, 3074, 3075, 3076, 3077, 3078, 3079, 3081, 3082, 3083, 3084,

    3085, 3086, 3087, 3088, 3089, 3090, 3091, 3092, 3093, 3094, 3095, 3096,

    3097, 3098, 3099, 3448, 3449, 3450, 3451, 3452, 3453, 3454, 3455, 3456,

    3457, 3458, 3459, 3460, 3461, 3471, 3490, 3491, 6307, 6368, 6421))

  GROUP BY YEAR(c.date), MONTH(c.date), p.district_id

The commission table has around 2,8 millions of records, of which 860 000+ belong to the current year 2018. The partner table has at this moment 8600+ records.

RESULT

| `YEAR(c.date)` | `MONTH(c.date)` | district_id | `COUNT(c.id)` | 

|----------------|-----------------|-------------|---------------| 

| 2018           | 1               | 1           | 19154         | 

| 2018           | 1               | 5           | 9184          | 

| 2018           | 1               | 6           | 2706          | 

| 2018           | 1               | 12          | 36296         | 

| 2018           | 1               | 15          | 13085         | 

| 2018           | 2               | 1           | 21231         | 

| 2018           | 2               | 5           | 10242         | 

| ...            | ...             | ...         | ...           | 



55 rows retrieved starting from 1 in 18 s 374 ms 

(execution: 18 s 368 ms, fetching: 6 ms)

EXPLAIN:

| id | select_type | table | partitions | type  | possible_keys                                                                                        | key                  | key_len | ref             | rows | filtered | extra                                        | 

|----|-------------|-------|------------|-------|------------------------------------------------------------------------------------------------------|----------------------|---------|-----------------|------|----------|----------------------------------------------| 

| 1  | SIMPLE      | p     | null       | range | PRIMARY                                                                                              | PRIMARY              | 4       |                 | 57   | 100      | Using where; Using temporary; Using filesort | 

| 1  | SIMPLE      | c     | null       | ref   | UNIQ_6F7146F0979B1AD62FC0CB0F5F8A7F73,IDX_6F7146F09395C3F3,IDX_6F7146F0979B1AD6,IDX_6F7146F0AA9E377A | IDX_6F7146F09395C3F3 | 5       | p.id            | 6716 | 8.33     | Using where                                  | 

DDL:

create table if not exists commission (

    id int auto_increment

        primary key,

    date date not null,

    source enum('ACTUAL', 'EXPECTED') not null,

    customer_id int null,

    transaction_id varchar(255) not null,

    company_id int null,

    constraint UNIQ_6F7146F0979B1AD62FC0CB0F5F8A7F73 unique (company_id, transaction_id, source),

    constraint FK_6F7146F09395C3F3 foreign key (customer_id) references partner (id),

    constraint FK_6F7146F0979B1AD6 foreign key (company_id) references companies (id)

) collate=utf8_unicode_ci;

create index IDX_6F7146F09395C3F3 on commission (customer_id);

create index IDX_6F7146F0979B1AD6 on commission (company_id);

create index IDX_6F7146F0AA9E377A on commission (date);

I noted that by removing the partner IN condition MySQL takes only 3s. I tried to replace it doing something crazy like this:

AND (',3062,3063,3064,3065,3066,3067,3068,3069,3070,3071,3072,3073,3074,3075,3076,3077,3078,3079,3081,3082,3083,3084,3085,3086,3087,3088,3089,3090,3091,3092,3093,3094,3095,3096,3097,3098,3099,3448,3449,3450,3451,3452,3453,3454,3455,3456,3457,3458,3459,3460,3461,3471,3490,3491,6307,6368,6421,'

     LIKE CONCAT('%,', p.id, ',%')) 

and the result was about 5s... great! but it's a hack.

WHY this query is taking a very long execution time when I uses IN statement? workaround, tips, links, etc. Thanks!

MySQL can use one index at a time. For this query you need a compound index covering the aspects of the search. Constant aspects of the WHERE clause should be used before range aspects like:

ALTER TABLE commission

DROP INDEX IDX_6F7146F0979B1AD6,

ADD INDEX IDX_6F7146F0979B1AD6 (company_id, source, date)

Here's what the Optimizer sees in your query.

Checking whether to use an index for the GROUP BY:

• Functions (YEAR()) in the GROUP BY, so no.


• Multiple tables (c and p) mentioned, so no.

For a JOIN, Optimizer will (almost always) start with one, then reach into the other. So, let's look at the two options:

If starting with p:

Assuming you have PRIMARY KEY(id), there is not much to think about. It will simply use that index.

For each row selected from p, it will then look into c, and any variation of this INDEX would be optimal.

c: INDEX(company_id, source, customer_id,  -- in any order (all are tested "=")

         date)       -- last, since it is tested as a range

If starting with c:

c: INDEX(company_id, source,  -- in any order (all are tested "=")

         date)       -- last, since it is tested as a range

-- slightly better:

c: INDEX(company_id, source,  -- in any order (all are tested "=")

         date,       -- last, since it is tested as a range

         customer_id)  -- really last -- added only to make it "covering".

The Optimizer will look at "statistics" to crudely decide which table to start with. So, add all the indexes I suggested.

A "covering" index is one that contains all the columns needed anywhere in the query. It is sometimes wise to extend a 'good' index with more columns to make it "covering".

But there is a monkey wrench in here. c.customer_id = p.id means that customer_id IN (...) effectively exists. But now there are two "range-like" constraints -- one is an IN, the other is a 'range'. In some newer versions, the Optimizer will happily jump around due to the IN and still be able to do "range" scans. So, I recommend this ordering:

1. Test(s) of column = constant
   


2. Test(s) with IN
   


3. 
   One 'range' test (BETWEEN, >=, LIKE with trailing wildcard, etc)


4. Perhaps add more columns to make it "covering" -- but don't do this step if you end up with more than, say, 5 columns in the index.

Hence, for c, the following is optimal for the WHERE, and happens to be "covering".

INDEX(company_id, source,  -- first, but in any order (all "=")

      customer_id,  -- "IN"

      date)       -- last, since it is tested as a range



p: (same as above)

Since there was an IN or "range", there is no use seeing if the index can also handle the GROUP BY.

A note on COUNT(x) -- it checks that x is NOT NULL. It is usually just as correct to say COUNT(*), which counts the number of rows without any extra checking.

This is a non-starter since it hides the indexed column (id) in a function:

AND (',3062,3063,3064,3065,3066,...6368,6421,'

     LIKE CONCAT('%,', p.id, ',%'))

With your LIKE-hack you are tricking optimizer so it uses different plan (most probably using IDX_6F7146F0AA9E377A index on the first place).
You should be able to see this in explain.

I think the real issue in your case is the second line of explain: server executing multiple functions (MONTH, YEAR) for 6716 rows and then trying to group all these rows. During this time all these 6716 rows should be stored (in memory or on disk that is based on your server configuration).

SELECT COUNT(*) FROM commission WHERE (date BETWEEN '2018-01-01' AND '2018-12-31') AND company_id = 90 AND source = 'ACTUAL';

=> How many rows are we talking about?

If the number in above query is much lower then 6716 I'd try to add covering index on columns customer_id, company_id, source and date. Not sure about the best order as it depends on data you have (check cardinality for these columns). I'd started with index (date, company_id, source, customer_id). Also, I'd add unique index (id, district_id, owner_id) on partner.

It is also possible to add additional generated stored columns _year and _month (if your server is a bit old you can add normal columns and fill them in with trigger) to rid off the multiple function executions.