[BUG] Memory leak in array of maps

[nverwer]

This bug is basically the same as eXist-db/exist#5375, which should be solved by #183 and #177.

I am submitting this bug again, because I can still reproduce it in Elemental 6.10.0.

My configuration (on stdout after starting Elemental) is:

Operating System: Linux 6.17.0-23-generic amd64] 
Using Java: 11.0.30 [Ubuntu (OpenJDK 64-Bit Server VM) in /usr/lib/jvm/java-11-openjdk-amd64] 
Approximate maximum amount of memory for JVM: 8 GB 
Number of processors available to JVM: 16 
[Elemental Version: 6.10.0] 
[Elemental Build: 2026-04-28T18:50:24Z] 
[Elemental Git commit: 1da1cf54200a2ead7a4b1ce42c6f282cf1bf2218] 
[Elemental Git commit timestamp: 20260428210445] 

To reproduce the bug, I use a slightly modified XQuery, compared to eXist issue #5375.

let $key-prefix := 'k0-'
let $doc as item() := array
  { for $i in 1 to 500000
    return map
      { $key-prefix||'id' : 'id'||$i
      , $key-prefix||'status' : if ($i mod 100 = 0) then 'inactive' else if ($i mod 80 = 1) then 'withdrawn' else 'active'
      , $key-prefix||'relationships' : array{ map
        { 'type' : 'Related'
        , 'id' : 'id'||($i+1)
        }}
      }
  }
let $doc-size as xs:integer := array:size($doc)

let $ids :=
  for $doc-index in 1 to $doc-size
    let $item as map(*) := $doc($doc-index)
    where $item?($key-prefix||'status') = ('withdrawn','inactive') and exists($item?($key-prefix||'relationships'))
  return $item?($key-prefix||'id')

return count($ids)

Every time I run this script, I change the $key-prefix to 'k1-', 'k2-', ...
After a garbage collection, the 'Java Memory' in Monex goes back to a low value. But in jconsole, the heap memory usage keeps going up, also after garbage collections.

This suggests that unused map keys are not freed. It also seems that the problem is still in the lookup operator (which should have been fixed by PR #183); When I replace the let $ids by

let $ids = ()

garbage collection makes the heap space return to a low value.

When I use array:get and map:get instead of the lookup operators, there is also no buildup of used heap space. So the problem is in the lookup operators, not their equivalent functions. For completeness, this is the code with functions instead of lookup operators:

let $key-prefix := 'k0-'
let $doc as item() := array
  { for $i in 1 to 500000
    return map
      { $key-prefix||'id' : 'id'||$i
      , $key-prefix||'status' : if ($i mod 100 = 0) then 'inactive' else if ($i mod 80 = 1) then 'withdrawn' else 'active'
      , $key-prefix||'relationships' : array{ map
        { 'type' : 'Related'
        , 'id' : 'id'||($i+1)
        }}
      }
  }
let $doc-size as xs:integer := array:size($doc)

let $ids :=
  for $doc-index in 1 to $doc-size
    let $item as map(*) := array:get($doc, $doc-index)
    where map:get($item, $key-prefix||'status') = ('withdrawn','inactive') and exists(map:get($item, $key-prefix||'relationships'))
  return map:get($item, $key-prefix||'id')

return count($ids)

[nverwer]

I experimented with variations on the XQuery that demonstrates the memory leak.
This variation is interesting:

let $key-prefix := 'k0-'
let $doc as item() := array
  { for $i in 1 to 500000
    return map
      { $key-prefix||'id0' : 'id'||$i
      , $key-prefix||'id1' : 'id'||$i
      , $key-prefix||'id2' : 'id'||$i
      , $key-prefix||'id3' : 'id'||$i
      , $key-prefix||'id4' : 'id'||$i
      , $key-prefix||'id5' : 'id'||$i
      }
  }
let $doc-size as xs:integer := array:size($doc)

let $ids :=
  for $doc-index in 1 to $doc-size
    let $item as map(*) := $doc($doc-index)
  return ()
return count($ids)

This also causes used heap space to grow when the $key-prefix is changed on every run.
The amount of irrecoverable heap space seems to be proportional to the number of keys in the maps, which is why I define 6 of them.

Note that the code does not access the maps. Also, the lookup operator is not used. Instead, the array is used as a function. Maybe the code that implements array-as-function (org.exist.xquery.functions.array.ArrayType.AccessorFunc) has a similar leak as Lookup.java?