Problem/Motivation

Aggregation is not properly working in views. It is not possible currently to have a single field return it's distinct values using aggregates.

Steps to reproduce

  1. Use Devel Generate and create content for, at least, 2 node-types ('page' and 'article' comming with Standard profile should be fine).
  2. Add the "Content Type" field to the View. It should be the only field.
  3. Enable aggregation.
  4. Set the "Content Type" fileds aggregation settings "Group results together" (is should be the default setting).

We are expecting to see only a distinct list of Content Type names and only for the types that have nodes.

Proposed resolution

Add an additional condition that lets GROUP BY to be added to the query even no aggregation function has been set.

Remaining tasks

  • Review
  • RTBC
  • Commit

User interface changes

None.

API changes

None.

Original report by @david_garcia

Beta phase evaluation

Reference: https://www.drupal.org/core/beta-changes
Issue category Bug
Unfrozen changes Unfrozen because it only fixes a bug in Views aggregation

Reporting 2 issues here:

[1] Aggregation is not properly working in views. It is not possible currently to have a single field return it's distinct values using aggregates.

[2] Incompatiblity with SQLServer because having constant values in an aggregate clausule makes no sense (oh yeah MySQL will swallow).

Explained in detail:

- Taking a very simple test case, I want to get a distinct list of all different content types in my system:

0. Create a view for the Node entity.
1. Add the "Content Type" field to the View.
2. Add contextual filter for user ID.
3. Enable agreggation on the query properties.

Result: No GROUP BY is added to the query, results are repeated (as long as you have some content in the system).

After looking through code and debugging, it looks like views is asumming that for a view to have aggregates one of these 2 things must happen:

(a) count($this->having) should be > 0
(b) in compile_fields, at least one of the fields needs to have !empty($field['function'])

\views\modules\field\views_handler_field_field.inc line 1316, method query()

   
    $this->has_aggregate = FALSE;
    $non_aggregates = array();

    list($non_aggregates) = $this->compile_fields($fields_array, $query);

    if (count($this->having)) {
      $this->has_aggregate = TRUE;
    }

The solution is to add an aditional condition:

(c) $this->view->display_handler->options['group_by'] is set to 1

thus, the above code will turn into:

    $this->has_aggregate = FALSE;
    $non_aggregates = array();

    list($non_aggregates) = $this->compile_fields($fields_array, $query);

    if (count($this->having)) {
      $this->has_aggregate = TRUE;
    }elseif($this->has_aggregate == FALSE){
      $this->has_aggregate = $this->view->display_handler->options['group_by'];
    }

A non-related issue I came accross, but is just a few lines below, has to do with the fact that on SQL SERVER you CANNOT USE CONSTANT VALUES IN A GROUP BY CLAUSULE:

http://www.sql-server-helper.com/error-messages/msg-164.aspx

The thrown exception, for google:

Each GROUP BY expression must contain at least one column that is not an outer reference.

Postgre and MySQL will swallow this.

Actually, having a constant value in a GROUP BY makes no sense as it will not alter the results, so I think we can safely prevent any constant column being added to the group by:

Turning this:

\views\modules\field\views_handler_field_field.inc line 1328, method query()

      foreach ($groupby as $field) {
        $query->groupBy($field);
      }

into this:

      foreach ($groupby as $field) {
        if($fields_array[$field]['table'] == NULL){
            continue;
        }
        $query->groupBy($field);
      }

About this change, I am not sure how $fields_array[$field]['table'] behaves on other scenarios such as computed columns.

Files: 
CommentFileSizeAuthor
#75 aggregation_not_working-2159347-63-d7.patch5.89 KBdavid_garcia
Test request sent.
Previous result: FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion.
[ View ]
#72 aggregation_not_working-2159347-60-d7.patch5.9 KBdavid_garcia
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch aggregation_not_working-2159347-60-d7.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
#63 aggregation_not_working-2159347-63-d7-do-not-test.patch5.89 KBclaudiu.cristea
#63 interdiff.txt1.09 KBclaudiu.cristea
#63 aggregation_not_working-2159347-63.patch3.08 KBclaudiu.cristea
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,786 pass(es).
[ View ]

Comments

david_garcia’s picture

Issue summary:View changes
infojunkie’s picture

Status:Active» Needs work

Please follow these guidelines to make a patch.

david_garcia’s picture

StatusFileSize
new919 bytes
FAILED: [[SimpleTest]]: [MySQL] 1,646 pass(es), 1 fail(s), and 0 exception(s).
[ View ]
david_garcia’s picture

Status:Needs work» Needs review

Status:Needs review» Needs work

The last submitted patch, 3: views-aggregation_problem-2159347-0.patch, failed testing.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new984 bytes
PASSED: [[SimpleTest]]: [MySQL] 1,658 pass(es).
[ View ]
xl_cheese’s picture

I'm currently using the patch in #6 with no issues at velorangutan.com

sachbearbeiter’s picture

using the patch with no issues ...

axel.rutz’s picture

Status:Needs review» Reviewed & tested by the community

So we have #6 and #7 as positive indication so i think it's time for maintainers to take a look on this.

vidorado’s picture

#6 works for me

cmseasy’s picture

Testing the functionality in combination with views selective filter (in sandbox) https://drupal.org/sandbox/david_garcia_garcia/2162097. No issues found.

david_garcia’s picture

REVIEW POSSIBLE ISSUE WHEN HAVING AGGREGATE AND DISTINCT OR PURE DISCTINCT AT THE SAME TIME IN A VIEW. (DOES THAT MAKE SENSE?¿)

antpre’s picture

Works for me also.

rooby’s picture

Status:Reviewed & tested by the community» Needs work

Sorry to do this but any maintainer that see this will say the same thing so better to fix it before they review it.

You won't get a patch committed until it adheres to the drupal coding standards.

In this case, no using tabs for indenting, uniform 2 space indenting, elseif on new line, adding missing spaces in if statements.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new992 bytes
Test request sent.
Previous result: PASSED: [[SimpleTest]]: [MySQL] 1,664 pass(es).
[ View ]

@rooby, very true...

This was one of the first patches I ever made for Drupal, it's been a long way since then.

Reviewd and re-rolled against latest dev.

rooby’s picture

Status:Needs review» Reviewed & tested by the community

Thanks.

Setting back to previous status seeing as the reason for needs work has been addressed.

roball’s picture

Just wanted to mention here that committing this patch is required to run the Views Selective Filters module. Any changes to get it committed?

mandreato’s picture

Migrated from Views Hacks to Views Selective Exposed Filters and needed this patch. +1 RTBC

bigMuzzy’s picture

I need it so much too!

P.S: Hands off the Andromeda Galaxy!

P.P.S: I can't figure out what else should I write here to get this patch into Views module.

aufumy’s picture

Maybe that you will buy the maintainers a drink?

roball’s picture

Priority:Normal» Major

Increasing the priority since a lot of users are requesting this feature.

Nicolas Bouteille’s picture

I just installed Views Selective Filters module and it is awesome. The module needs this patch which I have applied manually. No errors at all. Since there is very few code in the patch I really hope it'll get committed before next release version! Or my client site will break :(

rooby’s picture

~ Off topic ~

@Nicolas Bouteille:
It is not uncommon to need to use patched versions of modules.
Part of a good module updating procedure is handing patched modules.
If you keep the patch files you use with your site, when it comes to updating you check anything you are going to update for patch files, if there is a patch file you review the issue on drupal.org (all patches should have the node id of the issue in the file name - if not I recommend you add it to the file name when you add it to your site for this reason) and see if the patch has been committed in the new version (or if there is a newer version of the patch).
Then you update then module and reapply the patch if it still hasn't been committed.

For more info on this see http://drupal.stackexchange.com/a/59104/10729

Nicolas Bouteille’s picture

I like to auto update modules. I think it works very well. So instead of manually checking for patch files before updating, I prefer blocking the possibility to auto-update and get a warning that tells me that I cannot auto-update but have to do it manually instead and check if patch has been committed before:

<?php
/**
 * Implements hook_form_alter();
 */
function patched_modules_update_form_alter(&$form, $form_state, $form_id) {

 
 

//PREVENT PATCHED MODULES FROM BEING AUTOMATICALLY UPDATED
 
if($form_id == 'update_manager_update_form') {
   
//dpm($form);
   
if( isset($form['projects']["#options"]['views']['title']) ){
     
$form['projects']["#options"]['views']['title'] .= ' ' . t("THIS MODULE HAS BEEN PATCHED - YOU CAN'T UPDATE AUTOMATICALLY — SEE CUSTOM-MODIFS-DETAILS.TXT FILE");
     
$form['projects']['views']['#disabled'] = true;
    }
  }
 
?>

So yes indeed my site won't break if the patch is not committed in next release but I was just saying that so that I can be committed faster ;)

spazfox’s picture

After applying the patch in #15, many of my reports generated by Commerce Reporting are now broken. Specifically, sales reports now show only overall totals and can no longer be broken down to show daily, weekly, monthly, or yearly sales.

dawehner’s picture

Status:Reviewed & tested by the community» Needs work
Issue tags:+Needs tests

Such fundamental patches seriously need a test. Ensure that we also have a test for the other cases where it used to work, sorry.

Status:Needs work» Needs review
massiws’s picture

I migrated from Views Hacks to Views Selective Exposed Filters.
No problem after applying this patch.

david_garcia’s picture

StatusFileSize
new2.42 KB
Test request sent.
[ View ]

Added a test to the patch. Hope it helps to push forward the patch. This is critical, I cannot believe that with views we cannot simply get a distinct list of values over a single column.

david_garcia’s picture

@spazfox: An underlying bug like this, that has been wondering around for such long has, for sure, made other modules to incorrectly implement views.

david_garcia’s picture

Issue summary:View changes
asb’s picture

Coming here to get 'Views Selective Filters' working. The patch from #29 applies cleanly:

$ patch -p1 < views-aggregation_problem-2159347-0_2.patch
patching file plugins/views_plugin_query_default.inc
patching file tests/views_basic.test

So far, no side effects encountered. However, I don't see any 'Views Selective Filters' in Views UI.

david_garcia’s picture

@asb: Make sure to clear off all your caches!

asb’s picture

@david_garcia: I did (drush cc all). So I can not tell if it's 'Views Selective Exposed Filters', or if it's the patch that does not work as advertised.

david_garcia’s picture

@asb: The patch has nothing to do with the Views Selective filters showing in the filter list. You should open the issue in that project and add some screen captures to describe what you are seeing and what you are expecting to see.

timwood’s picture

Status:Needs review» Reviewed & tested by the community

Patch in #29 works for me.

Placinta’s picture

Status:Reviewed & tested by the community» Needs work

The patch extracts the group by option directly from the display handler, which is not correct, if the setting was set in the default display, and was not overridden.
To fix that the line

<?php
       $this
->has_aggregate = $this->view->display_handler->options['group_by'];
 
?>

should be replaced with

<?php
       $this
->has_aggregate = $this->view->display_handler->get_option('group_by');
 
?>

so it correctly tries to fallback to the option set on the default display as well.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new2.42 KB
Test request sent.
[ View ]

Totally true, fixed.

david_garcia’s picture

Project:Views» Drupal core
Version:7.x-3.x-dev» 8.0.x-dev
Component:Code» views.module
StatusFileSize
new1.12 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 77,416 pass(es), 16 fail(s), and 1,460 exception(s).
[ View ]

Patched for D8.

Status:Needs review» Needs work

The last submitted patch, 39: d8-2159347-no-aggregates.patch, failed testing.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new1.12 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] 77,470 pass(es), 0 fail(s), and 16 exception(s).
[ View ]

Status:Needs review» Needs work

The last submitted patch, 41: d8-2159347-no-aggregates.patch, failed testing.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new2.92 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 77,624 pass(es).
[ View ]
david_garcia’s picture

Status:Needs review» Reviewed & tested by the community

As this is a straight port to D8 and the D7 version is tested on over 600 sites, I think I can safely mark as RTBC.

Hope it can get some attention in D8, then backport. Looks like any Core or Views D7 interventions are freezed but for backports.

catch’s picture

Status:Reviewed & tested by the community» Needs work
Issue tags:+VDC
  1. +++ b/core/modules/views/src/Plugin/views/query/Sql.php
    @@ -1266,6 +1269,15 @@ public function query($get_count = FALSE) {
    +        // See: https://www.drupal.org/node/2159347

    I'd rather see the comments expanded than the link back to the issue, or both.

  2. +++ b/core/modules/views/src/Tests/QueryGroupByTest.php
    @@ -84,7 +84,16 @@ public function groupByTestHelper($aggregation_function, $values) {
    +    // in orther to have aggregation.

    typo: orther.

  3. +++ b/core/modules/views/src/Tests/QueryGroupByTest.php
    @@ -84,7 +84,16 @@ public function groupByTestHelper($aggregation_function, $values) {
    +      // If ID is included in the aggregate, bad bussines...

    typo: bussines. Could use a more descriptive comment explaining why this needs to be removed.

  4. +++ b/core/modules/views/src/Tests/QueryGroupByTest.php
    @@ -84,7 +84,16 @@ public function groupByTestHelper($aggregation_function, $values) {
    +    } ¶

    Trailing whitespace here and elsewhere.

+++ b/core/modules/views/src/Plugin/views/query/Sql.php
@@ -1266,6 +1269,15 @@ public function query($get_count = FALSE) {
+        $fieldField = isset($this->fields[$field]['field']) ? $this->fields[$field]['field'] : NULL;
...
+        if ($fieldTable == NULL && $fieldCount != $fieldField) {

What's a $fieldField?

Also these aren't object properties so should be $field_table etc.

Since we check if $fieldTable == NULL, we could do that first, and only then figure out the field count, no need to do it all up front.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new3.25 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 77,626 pass(es).
[ View ]

Let's try again, I tried to fix all issues.

cmseasy’s picture

For D7 #38 worked for me without issues.
Using it with Views Selectieve Filters https://www.drupal.org/project/views_selective_filters.
Thanks.

jhedstrom’s picture

Status:Needs review» Needs work
+++ b/core/modules/views/src/Tests/QueryGroupByTest.php
@@ -153,6 +166,13 @@ public function testGroupByMin() {
+  ¶

Still trailing whitespace.

I think the remainder of the issues from #45 have been addressed though.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new3.25 KB
FAILED: [[SimpleTest]]: [PHP 5.4 MySQL] Unable to apply patch d8-2159347-no-aggregates_3.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Status:Needs review» Needs work

The last submitted patch, 49: d8-2159347-no-aggregates.patch, failed testing.

claudiu.cristea’s picture

Issue summary:View changes
Status:Needs work» Needs review
Issue tags:-Needs tests
StatusFileSize
new2.68 KB
new3.06 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,189 pass(es).
[ View ]

The patch doesn't apply.

  • Removed SQL Server issue. Please open a separate issue for that.
  • Rearranged/compacted comments. Those were not followed coding standards.
claudiu.cristea’s picture

StatusFileSize
new3.26 KB

Correct interdiff.txt here.

claudiu.cristea’s picture

Issue summary:View changes
claudiu.cristea’s picture

Here's also the D7 patch but tests are backported from #51.

Again, I removed the SQL Server logic. That should be handled in its own issue node.

david_garcia’s picture

@claudiu.cristea Thank you for the effort. Makes me sad though to see the SQL Server part removed. Because of the nature of that bug and the small SQL Server community around Drupal it is prompt to become one of those one-line-no-brain-melter-never-commited issues.

claudiu.cristea’s picture

@david_garcia, I can see the problem but with that inside this patch will never pass the core committers. And it's correct that every problem/issue should have its own ticket.

david_garcia’s picture

Done: #2375407: Views should not use constant values or expressions in aggregates

Will have to repatch, probaly wait to see if this first issue ever gets commited.

jhedstrom’s picture

+++ b/core/modules/views/src/Plugin/views/query/Sql.php
@@ -1232,6 +1232,9 @@ public function query($get_count = FALSE) {
+      $this->hasAggregate = $this->view->display_handler->getOption('group_by');

Could we add a code comment here?

claudiu.cristea’s picture

StatusFileSize
new3.13 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,739 pass(es).
[ View ]
new594 bytes
new5.9 KB

Voilà!

jhedstrom’s picture

Issue summary:View changes
Status:Needs review» Reviewed & tested by the community

#60 looks good. This fixes a major bug in Views, and is therefore allowed under the beta phase (I've added this to the issue summary too).

tstoeckler’s picture

Status:Reviewed & tested by the community» Needs review
+++ b/core/modules/views/src/Tests/QueryGroupByTest.php
@@ -75,16 +75,26 @@ public function testAggregateCount() {
-  public function groupByTestHelper($aggregation_function, $values) {
+  public function groupByTestHelper($aggregation_function = NULL, $values) {

To have an optional argument being followed by a non optional argument is very strange. Let's remove the default here. It's fine to support passing NULL explicitly, but we do not have to provide a default.

claudiu.cristea’s picture

Status:Needs review» Reviewed & tested by the community
StatusFileSize
new3.08 KB
PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,786 pass(es).
[ View ]
new1.09 KB
new5.89 KB

OK. @tstoeckler can you RTBC back?

claudiu.cristea’s picture

Status:Reviewed & tested by the community» Needs review
jhedstrom’s picture

Status:Needs review» Reviewed & tested by the community

#63 addresses the concerns in #62. Back to RTBC.

tstoeckler’s picture

Thanks for that, looks much better IMO.

Back to RTBC.

Edit: X-post

alexpott’s picture

Status:Reviewed & tested by the community» Fixed

This issue addresses a major bug and is allowed per https://www.drupal.org/core/beta-changes. Committed 23b70df and pushed to 8.0.x. Thanks!

  • alexpott committed 23b70df on 8.0.x
    Issue #2159347 by david_garcia, claudiu.cristea: Aggregation not working...
david_garcia’s picture

Version:8.0.x-dev» 7.x-dev
Status:Fixed» Needs work

Then back to 7.x...

roball’s picture

Great this finally found the way into D8 - congrats david_garcia !

Hopefully we will see this in D7, too.

david_garcia’s picture

Status:Needs work» Needs review
StatusFileSize
new1.21 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch d8-2159347-no-constants-in-aggregate_0.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]
david_garcia’s picture

StatusFileSize
new5.9 KB
FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch aggregation_not_working-2159347-60-d7.patch. Unable to apply patch. See the log in the details link for more information.
[ View ]

Wrong patch in previous post....

The last submitted patch, 71: d8-2159347-no-constants-in-aggregate.patch, failed testing.

Status:Needs review» Needs work

The last submitted patch, 72: aggregation_not_working-2159347-60-d7.patch, failed testing.

david_garcia’s picture

Project:Drupal core» Views
Version:7.x-dev» 7.x-3.x-dev
Component:views.module» Code
Status:Needs work» Needs review
StatusFileSize
new5.89 KB
Test request sent.
Previous result: FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion.
[ View ]

Wrong project, moved back to views.

Status:Needs review» Needs work

The last submitted patch, 75: aggregation_not_working-2159347-63-d7.patch, failed testing.

Status:Needs work» Needs review