Fix ordering detection and placement - #109
Merged
Merged
Conversation
QueryAnalyzer was an ExpressionVisitor that descended into every node,
including the lambdas passed to operators. Any OrderBy inside a subquery
lambda, for example
context.Employees.Where(_ => _.Tasks.OrderBy(task => task.Title).Count() >= 0)
set HasOrdering, so the interceptor treated the query as explicitly
ordered and applied no default ordering at all. The generated SQL had no
ORDER BY. Include and Select lambdas were already guarded against this,
but every other operator was not.
The same false positive hit filtered Include, where an ordering nested in
the filter suppressed the default ordering of the included collection.
Replace the visitor with a walk of the chain itself, which is the only
place top level ordering and Include can appear. Ordering nested in a
lambda now orders its own subquery and leaves the outer query alone.
Ordering was only pushed beneath Select and Include, and appended to the
end of the chain otherwise. So Take produced
select * from (select top(3) * from Employees) order by HireDate desc
which takes three arbitrary rows and only then sorts them. Paging could
skip and repeat rows across pages while looking correctly sorted, and EF
Core still raised RowLimitingOperationWithoutOrderByWarning.
First, FirstOrDefault, Last, LastOrDefault and ElementAt were missed
entirely. Their result type is the entity rather than IQueryable<T>, so
element type detection returned null and no ordering was applied at all,
leaving EF Core to raise FirstWithoutOrderByAndFilterWarning.
Replace the two placement walks with one predicate naming the operators
the ordering has to precede, used both to find the source entity type and
to insert the ordering. Aggregates and Single are deliberately excluded,
since ordering them is pointless work.
FindSource followed the first argument of any static method whose source looked enumerable, which includes Concat, Union, Join and the other operators that combine two sequences. Ordering applied to one side was then reported as redundant, even though the default ordering is never applied to the combined result, so the advice to remove it was wrong. Skip those operators when walking to the source.
CreateForDerivedType looked the property up with the default binding
flags, which only match public members. An entity can map a non public
property, and ordering configured on one failed to inherit with
Property 'SortOrder' not found on derived type 'Derived'.
even though the derived type does have it. Include non public instance
properties in the lookup.
IncludeOrderingApplicator is an ExpressionVisitor, so it rebuilt the entire query tree on every compilation even though most queries have no Include for it to touch. The chain analysis already reports whether one is present, so run the analysis first and only visit when it is. The rewrite only replaces the inside of Include lambdas, so neither the ordering nor the Include flag can change as a result of it. Also resolve the options extension once and pass it to both callers, instead of RequiredOrder and RedundantOrder each resolving IDbContextOptions from the context on every compilation.
OrderByClause held the full method tables of Queryable and Enumerable in static fields, used only to resolve eight methods during static init. The MethodInfo instances are cached by the runtime either way, so what the fields actually kept alive was the two arrays. Resolve per method and let them go. The element type lookups were cached in static dictionaries keyed by Type. A query root is already IQueryable<T> and a navigation is declared as one of a handful of collection types, so both lookups hit a fast path of a few type comparisons, which is cheaper than the dictionary probe guarding it. The dictionaries also pinned every type they were handed for the life of the process, which stops a collectible load context unloading. Removed. The Include method cache stays, since MakeGenericMethod is genuinely expensive. Also append and reverse instead of repeatedly inserting at the head of the clause list, and mark the remaining mutable statics readonly.
The placement of the default ordering relative to Skip, Take and First is now user visible, so say where it goes and which operators are left alone.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Four correctness fixes and two performance changes, one commit each.
Ordering nested in a lambda suppressed the default ordering
QueryAnalyzerwas anExpressionVisitorthat descended into every node, including the lambdas passed to operators. AnyOrderByinside a subquery lambda setHasOrdering, so the query was treated as explicitly ordered and got no default ordering at all:IncludeandSelectlambdas already had hand rolled guards against this. Every other operator did not, soWhere,Any,SelectManyandGroupByall triggered it, as did an ordering nested inside a filteredInclude.Replaced the visitor with a walk of the chain itself, which is the only place top level ordering and
Includecan appear.Ordering was applied after the operators that choose rows
Ordering was only pushed beneath
SelectandInclude, and appended to the end of the chain otherwise. SoTakeproduced:Three arbitrary rows, sorted after the fact. Paging could skip and repeat rows across pages while looking correctly sorted, and EF Core still raised
RowLimitingOperationWithoutOrderByWarning.First,FirstOrDefault,Last,LastOrDefaultandElementAtwere missed entirely: their result type is the entity rather thanIQueryable<T>, so element type detection returned null and nothing was applied, leaving EF Core to raiseFirstWithoutOrderByAndFilterWarning.The two placement walks are now one predicate naming the operators the ordering has to precede, used both to find the source entity type and to insert the ordering. Aggregates and
Singleare excluded, since ordering them is pointless work, and snapshots assert they stay unordered.Redundant order detection walked into combined sequences
FindSourcefollowed the first argument of any static method whose source looked enumerable, which includesConcat,Union,Joinand friends. Ordering applied to one side was reported as redundant even though the default ordering is never applied to the combined result, so the advice to remove it was wrong.Inherited ordering could not see non public properties
CreateForDerivedTypelooked the property up with the default binding flags, which only match public members. Ordering configured on a mapped non public property failed to inherit withProperty 'SortOrder' not found on derived type 'Derived', even though the derived type does have it.Performance
Both are compile time only, since
QueryCompilationStartingsits behind EF Core's compiled query cache.IncludeOrderingApplicatorrebuilt the entire query tree on every compilation even when there was noIncludeto touch. The chain analysis already reports whether one is present, so it now runs first and gates the visit. The options extension is also resolved once instead of once each byRequiredOrderandRedundantOrder.Includemethod cache stays, sinceMakeGenericMethodis genuinely expensive.OrderByClausealso no longer holds theQueryableandEnumerablemethod tables in static fields after resolving its eight methods.Notes
The second fix changes behaviour:
Takenow returns different rows than before, deterministic ones, and queries that previously tripped EF Core's row limiting and first without ordering warnings no longer do. That may warrant a major version bump rather than a patch.Two things left alone deliberately.
Configuration.cachestill pins entity types for the process lifetime, since it is the config lookup on every compilation and has to be a real cache.RequiredOrder.validatedstill keys byDbContexttype, so the same context type used with differentUseDefaultOrderByoptions would share a validation result; the tests already declare a separate context type per option combination, so the design was left as is.Verification
122 tests pass. Each fix has tests, and for the last two I confirmed they are regression tests by reverting the source fix and watching them fail.