matching stats as join table #3835

awildturtok · 2026-01-22T16:50:07Z

Eine Beispielhafte join-table könnte folgende Einträge enthalten:

                     resolved_id                      | col_val
------------------------------------------------------+----------
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T612
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T611
 database.icd.s00-t98.s00-s09.s00.s00_0.s00_00 | T610
 database.icd.j00-j99.j30-j39.j38.j38_0.j38_00 | J951

  select
    diag.id as "pid",
    least(
      'infinity'::date,
      lower("datum"),
      upper("datum")
    ) as "lower_bound",
    greatest(
      '-infinity'::date,
      lower("datum"),
      upper("datum")
    ) as "upper_bound",
    "resolved_id"
  from "diag"
    left outer join "icd_ids"
      on "diag"."icd_code" = "col_val"

im Falle von ColumnEquals wären dann noch weitere Spalten dabei. Bei PRESENT/EMPTY wären stattdessen true/false drin, die dann im join abgebildet werden.

# Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/local/LocalStorageListener.java

… SqlMatchingStats easier

Also fixes a case, where missing entries were not registered to the root.

…ostgres

…tats-as-sql-function # Conflicts: # backend/src/main/java/com/bakdata/conquery/models/messages/namespaces/specific/UpdateMatchingStatsMessage.java

(experimental)

(this is a bit absurd tbh)

#2

#3

#4

thoniTUB · 2026-01-23T07:21:32Z

backend/src/main/java/com/bakdata/conquery/mode/StorageListener.java

+@Data
+public abstract class StorageListener<T extends Namespace>{


Ich hätte das Interface beibehalten und davon diese abstract Class abgeleitet.

Interfaces können auch praktisch sein, da man für diese Proxies implementieren kann

thoniTUB · 2026-01-23T07:25:52Z

backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/conditions/CTCondition.java

+	 */
+	//TODO better name
+	record Expression(ConceptElement<?> conceptElement, Map<Field<?>, Set<Param<?>>> conditions) {
+		public Expression and(Expression other) {


Braust du hier nicht eine parallel Struktur zu dem eigentlich Klassen Konstruct auf (AndCondition)?

nein, weil das nicht die Condition Logik abbildet. OR kann ich soweit ich das sehe nicht mit den join-tables abbilden und wird auch einfach nirgends verwendet.

ebenso hab ich NOT nicht implmenetiert. Also es wird bei and bleiben, weil es die Logik vom Baum abbildet

thoniTUB · 2026-01-23T07:28:58Z

...end/src/main/java/com/bakdata/conquery/models/datasets/concepts/conditions/NotCondition.java

+	public Expression buildExpression(CTConditionContext context, ConceptElement<?> id) {
+		throw new IllegalStateException("Not implemented");
+	}


buildExpression aufzurufen wirkt wie ein Minenfeld 👀
Ich bin aber auch verwirrt, weil ich nur einen Aufruf dafür finde, der wiederum in einem anderen buildExpression ist

na ich hab einfach noch nicht alles implementiert und wir werden auch nicht alles implementieren können. NOT wurde eh nur mit IS_EMPTY/_PRESENT in Verbindung verwendet. deswegen hab ich das unrolled. NotCondition kommt dann weg

thoniTUB · 2026-01-23T07:34:23Z

backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/MatchingStats.java

+	@JsonIgnore
+	private long numberOfEntities = -1L;
+
+	public synchronized long countEvents() {


müssen die synchronized sein?

thoniTUB · 2026-01-23T07:36:02Z

backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/MatchingStats.java

-            if (foundEntities.add(entityForEvent)) {
-                numberOfEntities++;
-            }
+		public void addEventFromBucket(String entityForEvent, Bucket bucket, int event, Iterable<Column> dateColumns) {


Bucket ist bei SQL aber kein Ding mehr oder? Kann die Logik an einen passenderen Ort gepackt werden?

thoniTUB · 2026-01-23T07:42:32Z

backend/src/main/java/com/bakdata/conquery/mode/local/UpdateMatchingStatsSqlJob.java

thoniTUB · 2026-01-23T07:47:31Z

backend/src/main/java/com/bakdata/conquery/sql/conquery/SqlMatchingStats.java

+							  .buildExpression(context, current)
+							  .and(parentExpression);


Bei Equal wirst du hier aber viele Doppelungen haben, da die Parentnodes ein Superset der ChildNodes sind

Die werden schon geschnitten in der Expression and Implementierung

thoniTUB · 2026-01-23T07:55:02Z

backend/src/main/java/com/bakdata/conquery/sql/conquery/SqlMatchingStats.java

+		out.add(forCurrent);
+
+		for (ConceptTreeChild child : current.getChildren()) {
+			out.addAll(collectAllExpressions(child, forCurrent, context));


Du kannst dir hier viele kurzlebige Allokationen sparen in dem du die Liste zum hinzufügen als Argument mitgibst und nicht in jeder Recursion eine weitere Liste erzeugst.

thoniTUB · 2026-01-23T07:57:52Z

backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/HanaSqlFunctionProvider.java

+		return """
+							     CREATE OR REPLACE FUNCTION %s(%s) RETURNS output NVARCHAR(500) AS
+							     BEGIN
+							     	output = %s;


Muss %s nicht sanitized werden?

die methode wird eh nicht mehr gebraucht aber für das was ich gemacht hatte nein, weil ich da jooq SQL reingesteckt hatte und das auch ausgeführt haben wollte

thoniTUB · 2026-01-23T07:58:55Z

backend/src/main/java/com/bakdata/conquery/sql/conversion/dialect/PostgreSqlDialect.java

 			case REAL -> field.getDataType().isNumeric();
 			case DECIMAL -> field.getDataType().isDecimal();
-			case MONEY -> field.getDataType().isDecimal();
+			case MONEY -> true; // TODO Need to find proper name


Ist das ein Workaround?

Jooq hat kein money type und es gibt keine introspection, die mir money herausgibt

jnsrnhld · 2026-01-30T10:15:50Z

backend/src/main/java/com/bakdata/conquery/models/datasets/concepts/MatchingStats.java

+	private CDateRange span;
+
+	@JsonIgnore
+	private long numberOfEvents = -1L;


Vielleicht könnte man hier generell auf LongAdder umsteigen und auf die Map verzichten, aber ist nur so ein Gedanke. Würde glaube ich die Synchronisierungs- und Summen-Logik vereinfachen.

jnsrnhld · 2026-01-30T10:17:46Z

...main/java/com/bakdata/conquery/models/datasets/concepts/conditions/ColumnEqualCondition.java

 	@Override
 	public WhereCondition convertToSqlCondition(CTConditionContext context) {
-		Field<String> field = DSL.field(DSL.name(context.getConnectorTable().getName(), column), String.class);
+		Field<String> field = (Field<String>) (Field<?>) field(DSL.name(column));


Suggested change

Field<String> field = (Field<String>) (Field<?>) field(DSL.name(column));

Field<String> field = field(DSL.name(column), String.class);

jnsrnhld · 2026-01-30T10:18:14Z

...main/java/com/bakdata/conquery/models/datasets/concepts/conditions/ColumnEqualCondition.java

+
+	@Override
+	public Expression buildExpression(CTConditionContext context, ConceptElement<?> id) {
+		return new Expression(id, Map.of(field(DSL.name(getColumn()), VARCHAR).as("%s_equal".formatted(column)), values.stream().map(DSL::val).collect(Collectors.toSet())));


Suggested change

return new Expression(id, Map.of(field(DSL.name(getColumn()), VARCHAR).as("%s_equal".formatted(column)), values.stream().map(DSL::val).collect(Collectors.toSet())));

return new Expression(id, Map.of(field(name(getColumn()), VARCHAR).as("%s_equal".formatted(column)), values.stream().map(DSL::val).collect(Collectors.toSet())));

jnsrnhld · 2026-01-30T10:23:26Z

...src/main/java/com/bakdata/conquery/models/datasets/concepts/conditions/IsEmptyCondition.java

+
+	@Override
+	public WhereCondition convertToSqlCondition(CTConditionContext context) {
+		Condition condition = DSL.field(DSL.name(column)).isNull();


Hier hast du überall noch nicht den static import für DSL. gemacht, an anderer Stelle schon.

awildturtok and others added 30 commits November 12, 2025 15:24

First draft of function to resolve concepts

74d9da4

# Conflicts: # backend/src/main/java/com/bakdata/conquery/mode/local/LocalStorageListener.java

remove qualification from CTConditionContext to make integration with…

e028841

… SqlMatchingStats easier

Collect Auxiliary columns to then generate proper signature

4cccd65

properly insert function (only postgres atm)

209d881

first draft towards applying matching stats

8620c65

Reworks registration of MatchingStats.

51f1aea

Also fixes a case, where missing entries were not registered to the root.

fixes usage in MatchingStatsTests.java

98b69c4

wip

f09cdaa

implements conversion and extraction of matching stats for Hana and P…

107407d

…ostgres

Merge remote-tracking branch 'origin/develop' into feature/matching-s…

1b4949f

…tats-as-sql-function # Conflicts: # backend/src/main/java/com/bakdata/conquery/models/messages/namespaces/specific/UpdateMatchingStatsMessage.java

disable daterange and money type compat check

3f0b1bb

adds missing coalesce for primaryColumn

ff18d93

don't remove connector column

b86af08

(experimental)

removes very verbose logging

7735740

use cursor to iterate results

ed98bd2

adds timing to SQL matching stats fetching

368b4f5

use transaction to disable autocommit

5ffdade

log select statement for debugging

0cb22e8

add PARALLEL SAFE marker to created functions

8e64d47

reworks SqlMatchingStats function as flattened table

8d5cd0a

(this is a bit absurd tbh)

adds some logging

44899ee

adds missing concept Id

e2df161

adds missing error handling in outer loop

750a03b

adds grouping by params to map into most specific child

152fc72

fixes wrong usage of immutable datasctructure

37009b0

try to insert the join-tables

67a0547

adds typing to fields

34b8bb5

remove primary key (nullability issue) and add index

4ff4776

fix index creation

11bd235

fix index creation #2

8e56eda

awildturtok added 16 commits January 15, 2026 17:36

fix index creation #3

0c355b3

outcomment index creation

811afeb

delete prior table

ced3e38

first draft of using join tables

bf1f37e

fix dupe join

f04d739

cleanup of SqlMatchingStats

6aef402

fix naming

f5fe46e

adds exception handling

8ef440f

hopefully fixes reference on ColumnValue

556f5ee

hopefully fixes reference on ColumnValue

4faabfb

#2

hopefully fixes reference on ColumnValue

46f46a1

#3

hopefully fixes reference on ColumnValue

c53ce5f

#4

minor fixes for extraction of matching stats

3093a9a

more cleanup

53ba076

more cleanup

11c304f

fix union to intersection

3d02375

awildturtok requested a review from jnsrnhld January 22, 2026 16:50

awildturtok requested a review from thoniTUB as a code owner January 22, 2026 16:50

thoniTUB reviewed Jan 23, 2026

View reviewed changes

jnsrnhld reviewed Jan 30, 2026

View reviewed changes

		@Data
		public abstract class StorageListener<T extends Namespace>{

	Field<String> field = (Field<String>) (Field<?>) field(DSL.name(column));
	Field<String> field = field(DSL.name(column), String.class);

	return new Expression(id, Map.of(field(DSL.name(getColumn()), VARCHAR).as("%s_equal".formatted(column)), values.stream().map(DSL::val).collect(Collectors.toSet())));
	return new Expression(id, Map.of(field(name(getColumn()), VARCHAR).as("%s_equal".formatted(column)), values.stream().map(DSL::val).collect(Collectors.toSet())));

matching stats as join table #3835

Are you sure you want to change the base?

matching stats as join table #3835

Uh oh!

Conversation

awildturtok commented Jan 22, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

awildturtok commented Jan 22, 2026 •

edited

Loading