Problem/Motivation
Aggregation is not properly working in views. It is not possible currently to have a single field return it's distinct values using aggregates.
Steps to reproduce
- Use Devel Generate and create content for, at least, 2 node-types ('page' and 'article' comming with Standard profile should be fine).
- Add the "Content Type" field to the View. It should be the only field.
- Enable aggregation.
- Set the "Content Type" fileds aggregation settings "Group results together" (is should be the default setting).
We are expecting to see only a distinct list of Content Type names and only for the types that have nodes.
Proposed resolution
Add an additional condition that lets GROUP BY to be added to the query even no aggregation function has been set.
Remaining tasks
- Review
- RTBC
- Commit
User interface changes
None.
API changes
None.
Original report by @david_garcia
Beta phase evaluation
| Issue category | Bug |
|---|---|
| Unfrozen changes | Unfrozen because it only fixes a bug in Views aggregation |
Reporting 2 issues here:
[1] Aggregation is not properly working in views. It is not possible currently to have a single field return it's distinct values using aggregates.
[2] Incompatiblity with SQLServer because having constant values in an aggregate clausule makes no sense (oh yeah MySQL will swallow).
Explained in detail:
- Taking a very simple test case, I want to get a distinct list of all different content types in my system:
0. Create a view for the Node entity.
1. Add the "Content Type" field to the View.
2. Add contextual filter for user ID.
3. Enable agreggation on the query properties.
Result: No GROUP BY is added to the query, results are repeated (as long as you have some content in the system).
After looking through code and debugging, it looks like views is asumming that for a view to have aggregates one of these 2 things must happen:
(a) count($this->having) should be > 0
(b) in compile_fields, at least one of the fields needs to have !empty($field['function'])
\views\modules\field\views_handler_field_field.inc line 1316, method query()
$this->has_aggregate = FALSE;
$non_aggregates = array();
list($non_aggregates) = $this->compile_fields($fields_array, $query);
if (count($this->having)) {
$this->has_aggregate = TRUE;
}
The solution is to add an aditional condition:
(c) $this->view->display_handler->options['group_by'] is set to 1
thus, the above code will turn into:
$this->has_aggregate = FALSE;
$non_aggregates = array();
list($non_aggregates) = $this->compile_fields($fields_array, $query);
if (count($this->having)) {
$this->has_aggregate = TRUE;
}elseif($this->has_aggregate == FALSE){
$this->has_aggregate = $this->view->display_handler->options['group_by'];
}
A non-related issue I came accross, but is just a few lines below, has to do with the fact that on SQL SERVER you CANNOT USE CONSTANT VALUES IN A GROUP BY CLAUSULE:
http://www.sql-server-helper.com/error-messages/msg-164.aspx
The thrown exception, for google:
Each GROUP BY expression must contain at least one column that is not an outer reference.
Postgre and MySQL will swallow this.
Actually, having a constant value in a GROUP BY makes no sense as it will not alter the results, so I think we can safely prevent any constant column being added to the group by:
Turning this:
\views\modules\field\views_handler_field_field.inc line 1328, method query()
foreach ($groupby as $field) {
$query->groupBy($field);
}into this:
foreach ($groupby as $field) {
if($fields_array[$field]['table'] == NULL){
continue;
}
$query->groupBy($field);
}About this change, I am not sure how $fields_array[$field]['table'] behaves on other scenarios such as computed columns.
| Comment | File | Size | Author |
|---|---|---|---|
| #75 | 5.89 KB | david_garcia | |
| Test request sent. Previous result: FAILED: [[SimpleTest]]: [MySQL] Setup environment: Test cancelled by admin prior to completion. [ View ] | |||
| #72 | 5.9 KB | david_garcia | |
| FAILED: [[SimpleTest]]: [MySQL] Unable to apply patch aggregation_not_working-2159347-60-d7.patch. Unable to apply patch. See the log in the details link for more information. [ View ] | |||
| #63 | 5.89 KB | claudiu.cristea | |
| #63 | 1.09 KB | claudiu.cristea | |
| #63 | 3.08 KB | claudiu.cristea | |
| PASSED: [[SimpleTest]]: [PHP 5.4 MySQL] 81,786 pass(es). [ View ] | |||
Comments
Comment #1
david_garcia commentedComment #2
infojunkie commentedPlease follow these guidelines to make a patch.
Comment #3
david_garcia commentedComment #4
david_garcia commentedComment #6
david_garcia commentedComment #7
xl_cheese commentedI'm currently using the patch in #6 with no issues at velorangutan.com
Comment #8
sachbearbeiter commentedusing the patch with no issues ...
Comment #9
axel.rutz commentedSo we have #6 and #7 as positive indication so i think it's time for maintainers to take a look on this.
Comment #10
vidorado commented#6 works for me
Comment #11
cmseasy commentedTesting the functionality in combination with views selective filter (in sandbox) https://drupal.org/sandbox/david_garcia_garcia/2162097. No issues found.
Comment #12
david_garcia commentedREVIEW POSSIBLE ISSUE WHEN HAVING AGGREGATE AND DISTINCT OR PURE DISCTINCT AT THE SAME TIME IN A VIEW. (DOES THAT MAKE SENSE?¿)
Comment #13
antpre commentedWorks for me also.
Comment #14
rooby commentedSorry to do this but any maintainer that see this will say the same thing so better to fix it before they review it.
You won't get a patch committed until it adheres to the drupal coding standards.
In this case, no using tabs for indenting, uniform 2 space indenting, elseif on new line, adding missing spaces in if statements.
Comment #15
david_garcia commented@rooby, very true...
This was one of the first patches I ever made for Drupal, it's been a long way since then.
Reviewd and re-rolled against latest dev.
Comment #16
rooby commentedThanks.
Setting back to previous status seeing as the reason for needs work has been addressed.
Comment #17
roball commentedJust wanted to mention here that committing this patch is required to run the Views Selective Filters module. Any changes to get it committed?
Comment #18
mandreato commentedMigrated from Views Hacks to Views Selective Exposed Filters and needed this patch. +1 RTBC
Comment #19
bigMuzzy commentedI need it so much too!
P.S: Hands off the Andromeda Galaxy!
P.P.S: I can't figure out what else should I write here to get this patch into Views module.
Comment #20
aufumy commentedMaybe that you will buy the maintainers a drink?
Comment #21
roball commentedIncreasing the priority since a lot of users are requesting this feature.
Comment #22
Nicolas Bouteille commentedI just installed Views Selective Filters module and it is awesome. The module needs this patch which I have applied manually. No errors at all. Since there is very few code in the patch I really hope it'll get committed before next release version! Or my client site will break :(
Comment #23
rooby commented~ Off topic ~
@Nicolas Bouteille:
It is not uncommon to need to use patched versions of modules.
Part of a good module updating procedure is handing patched modules.
If you keep the patch files you use with your site, when it comes to updating you check anything you are going to update for patch files, if there is a patch file you review the issue on drupal.org (all patches should have the node id of the issue in the file name - if not I recommend you add it to the file name when you add it to your site for this reason) and see if the patch has been committed in the new version (or if there is a newer version of the patch).
Then you update then module and reapply the patch if it still hasn't been committed.
For more info on this see http://drupal.stackexchange.com/a/59104/10729
Comment #24
Nicolas Bouteille commentedI like to auto update modules. I think it works very well. So instead of manually checking for patch files before updating, I prefer blocking the possibility to auto-update and get a warning that tells me that I cannot auto-update but have to do it manually instead and check if patch has been committed before:
<?php
/**
* Implements hook_form_alter();
*/
function patched_modules_update_form_alter(&$form, $form_state, $form_id) {
//PREVENT PATCHED MODULES FROM BEING AUTOMATICALLY UPDATEDif($form_id == 'update_manager_update_form') {
//dpm($form);
if( isset($form['projects']["#options"]['views']['title']) ){
$form['projects']["#options"]['views']['title'] .= ' ' . t("THIS MODULE HAS BEEN PATCHED - YOU CAN'T UPDATE AUTOMATICALLY — SEE CUSTOM-MODIFS-DETAILS.TXT FILE");
$form['projects']['views']['#disabled'] = true;
}
}
?>
So yes indeed my site won't break if the patch is not committed in next release but I was just saying that so that I can be committed faster ;)
Comment #25
spazfox commentedAfter applying the patch in #15, many of my reports generated by Commerce Reporting are now broken. Specifically, sales reports now show only overall totals and can no longer be broken down to show daily, weekly, monthly, or yearly sales.
Comment #26
dawehner commentedSuch fundamental patches seriously need a test. Ensure that we also have a test for the other cases where it used to work, sorry.
Comment #28
massiws commentedI migrated from Views Hacks to Views Selective Exposed Filters.
No problem after applying this patch.
Comment #29
david_garcia commentedAdded a test to the patch. Hope it helps to push forward the patch. This is critical, I cannot believe that with views we cannot simply get a distinct list of values over a single column.
Comment #30
david_garcia commented@spazfox: An underlying bug like this, that has been wondering around for such long has, for sure, made other modules to incorrectly implement views.
Comment #31
david_garcia commentedComment #32
asb commentedComing here to get 'Views Selective Filters' working. The patch from #29 applies cleanly:
$ patch -p1 < views-aggregation_problem-2159347-0_2.patchpatching file plugins/views_plugin_query_default.inc
patching file tests/views_basic.test
So far, no side effects encountered. However, I don't see any 'Views Selective Filters' in Views UI.
Comment #33
david_garcia commented@asb: Make sure to clear off all your caches!
Comment #34
asb commented@david_garcia: I did (
drush cc all). So I can not tell if it's 'Views Selective Exposed Filters', or if it's the patch that does not work as advertised.Comment #35
david_garcia commented@asb: The patch has nothing to do with the Views Selective filters showing in the filter list. You should open the issue in that project and add some screen captures to describe what you are seeing and what you are expecting to see.
Comment #36
timwood commentedPatch in #29 works for me.
Comment #37
Placinta commentedThe patch extracts the group by option directly from the display handler, which is not correct, if the setting was set in the default display, and was not overridden.
To fix that the line
should be replaced with
so it correctly tries to fallback to the option set on the default display as well.
Comment #38
david_garcia commentedTotally true, fixed.
Comment #39
david_garcia commentedPatched for D8.
Comment #41
david_garcia commentedComment #43
david_garcia commentedComment #44
david_garcia commentedAs this is a straight port to D8 and the D7 version is tested on over 600 sites, I think I can safely mark as RTBC.
Hope it can get some attention in D8, then backport. Looks like any Core or Views D7 interventions are freezed but for backports.
Comment #45
catch commented+++ b/core/modules/views/src/Plugin/views/query/Sql.php@@ -1266,6 +1269,15 @@ public function query($get_count = FALSE) {
+ // See: https://www.drupal.org/node/2159347
I'd rather see the comments expanded than the link back to the issue, or both.
+++ b/core/modules/views/src/Tests/QueryGroupByTest.php@@ -84,7 +84,16 @@ public function groupByTestHelper($aggregation_function, $values) {
+ // in orther to have aggregation.
typo: orther.
+++ b/core/modules/views/src/Tests/QueryGroupByTest.php@@ -84,7 +84,16 @@ public function groupByTestHelper($aggregation_function, $values) {
+ // If ID is included in the aggregate, bad bussines...
typo: bussines. Could use a more descriptive comment explaining why this needs to be removed.
+++ b/core/modules/views/src/Tests/QueryGroupByTest.php@@ -84,7 +84,16 @@ public function groupByTestHelper($aggregation_function, $values) {
+ } ¶
Trailing whitespace here and elsewhere.
+++ b/core/modules/views/src/Plugin/views/query/Sql.php@@ -1266,6 +1269,15 @@ public function query($get_count = FALSE) {
+ $fieldField = isset($this->fields[$field]['field']) ? $this->fields[$field]['field'] : NULL;
...
+ if ($fieldTable == NULL && $fieldCount != $fieldField) {
What's a $fieldField?
Also these aren't object properties so should be $field_table etc.
Since we check if $fieldTable == NULL, we could do that first, and only then figure out the field count, no need to do it all up front.
Comment #46
david_garcia commentedLet's try again, I tried to fix all issues.
Comment #47
cmseasy commentedFor D7 #38 worked for me without issues.
Using it with Views Selectieve Filters https://www.drupal.org/project/views_selective_filters.
Thanks.
Comment #48
jhedstrom commented+++ b/core/modules/views/src/Tests/QueryGroupByTest.php@@ -153,6 +166,13 @@ public function testGroupByMin() {
+ ¶
Still trailing whitespace.
I think the remainder of the issues from #45 have been addressed though.
Comment #49
david_garcia commentedComment #51
claudiu.cristea commentedThe patch doesn't apply.
Comment #52
claudiu.cristea commentedCorrect interdiff.txt here.
Comment #53
claudiu.cristea commentedComment #54
claudiu.cristea commentedHere's also the D7 patch but tests are backported from #51.
Again, I removed the SQL Server logic. That should be handled in its own issue node.
Comment #55
david_garcia commented@claudiu.cristea Thank you for the effort. Makes me sad though to see the SQL Server part removed. Because of the nature of that bug and the small SQL Server community around Drupal it is prompt to become one of those one-line-no-brain-melter-never-commited issues.
Comment #56
claudiu.cristea commented@david_garcia, I can see the problem but with that inside this patch will never pass the core committers. And it's correct that every problem/issue should have its own ticket.
Comment #57
david_garcia commentedDone: #2375407: Views should not use constant values or expressions in aggregates
Will have to repatch, probaly wait to see if this first issue ever gets commited.
Comment #58
claudiu.cristea commentedBlocks #2375407: Views should not use constant values or expressions in aggregates.
Comment #59
jhedstrom commented+++ b/core/modules/views/src/Plugin/views/query/Sql.php@@ -1232,6 +1232,9 @@ public function query($get_count = FALSE) {
+ $this->hasAggregate = $this->view->display_handler->getOption('group_by');
Could we add a code comment here?
Comment #60
claudiu.cristea commentedVoilà!
Comment #61
jhedstrom commented#60 looks good. This fixes a major bug in Views, and is therefore allowed under the beta phase (I've added this to the issue summary too).
Comment #62
tstoeckler commented+++ b/core/modules/views/src/Tests/QueryGroupByTest.php@@ -75,16 +75,26 @@ public function testAggregateCount() {
- public function groupByTestHelper($aggregation_function, $values) {
+ public function groupByTestHelper($aggregation_function = NULL, $values) {
To have an optional argument being followed by a non optional argument is very strange. Let's remove the default here. It's fine to support passing NULL explicitly, but we do not have to provide a default.
Comment #63
claudiu.cristea commentedOK. @tstoeckler can you RTBC back?
Comment #64
claudiu.cristea commentedComment #65
jhedstrom commented#63 addresses the concerns in #62. Back to RTBC.
Comment #66
tstoeckler commentedThanks for that, looks much better IMO.
Back to RTBC.
Edit: X-post
Comment #67
alexpott commentedThis issue addresses a major bug and is allowed per https://www.drupal.org/core/beta-changes. Committed 23b70df and pushed to 8.0.x. Thanks!
Comment #69
david_garcia commentedThen back to 7.x...
Comment #70
roball commentedGreat this finally found the way into D8 - congrats david_garcia !
Hopefully we will see this in D7, too.
Comment #71
david_garcia commentedComment #72
david_garcia commentedWrong patch in previous post....
Comment #75
david_garcia commentedWrong project, moved back to views.