Load relevant packages

[PatrickHaecker]

With the now merged #112 (thanks!) together with the current PR , I hope to get back to a reasonable memory usage. Claude Opus 4.8 did most of the work and wrote the text following this paragraph. I reviewed everything and changed some things, but I haven't followed all details when they looked reasonable. I ran the indexer, but have not used it in VS Code, as I don't know how to do that.

Summary

get_store indexed every top-level dependency of a project's environment, even packages the workspace never using/imports. This PR extracts the packages each project actually imports and loads only those.

Impact

Indexer on a real 145-dep environment (worst case — 24 deved pkgs) with Julia 1.13.0-rc1: peak RSS 4.4 GB → 1.3 GB, wall time 22.5 → 4 min. Also avoids running unused packages' __init__ side effects (conda installs, GPU init, …).

Correctness

• deved packages are always indexed.
• Transitive deps of imported packages are still cached, so re-exports keep resolving.
• The imported set is part of WatchEnvironmentKey, so adding an import re-triggers indexing.
• Method/piracy capture is a strict subset; each .jstore stays self-contained.

Commits

1. feat: index only the packages the workspace imports (+ tests)
2. refactor: reactor work messages carry their key (no behavior change)

[codecov]

Codecov Report

❌ Patch coverage is 0% with 99 lines in your changes missing coverage. Please review.
✅ Project coverage is 0.00%. Comparing base (b7a7141) to head (8931f1b).
⚠️ Report is 12 commits behind head on main.

Files with missing lines	Patch %	Lines
src/dynamic_feature/dynamic_feature.jl	0.00%	52 Missing ⚠️
src/layer_environment.jl	0.00%	44 Missing ⚠️
src/types.jl	0.00%	3 Missing ⚠️

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #117   +/-   ##
=====================================
  Coverage   0.00%   0.00%           
=====================================
  Files         56      56           
  Lines       7382    7436   +54     
=====================================
- Misses      7382    7436   +54     

Flag	Coverage Δ
unittests	0.00% <0.00%> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

[pfitzseb]

I'm a bit hesitant to go with this. The additional complexity doesn't seem like it would be worth it in the long term to me, given that cloud caching should cover all non-dev'ed packages eventually.

Can you give https://pub-13b30ea54b6946de895ab5362f62b4cf.r2.dev a go via the julia.symbolserverUpstream setting? That should have fairly decent coverage at this point.

[PatrickHaecker]

Can you give https://pub-13b30ea54b6946de895ab5362f62b4cf.r2.dev/ a go via the julia.symbolserverUpstream setting? That should have fairly decent coverage at this point.

This seems to work and seems to have a similar RAM usage (for the whole process) as with the default settings (2.5 GB vs. 2.4 GB – not considering the current PR).

I'm a bit hesitant to go with this. The additional complexity doesn't seem like it would be worth it in the long term to me, given that cloud caching should cover all non-dev'ed packages eventually.

You can probably judge this best. As long as CPU and memory usage scales with actually used packages and not with packages in the environment, both seems to work for me. You might think that you need the "per package" logic anyway and whether it's coming from the cloud or from the local cache would not add too much complexity either way, but it's probably more complex than that, I guess. Occasionally, I quickly change an added package (instead of a developed). It's nice if this use case is supported, but it's not the end of the world if it isn't.

So feel free to close it, if you don't think it's worth it. I deliberately split the PR into multiple commits. The second commit seems to be worth it even if you don't want to have the rest. But again: Your call. I am happy either way if my VS Code RAM usage is reasonable again.

[pfitzseb]

Ok, so what you're saying is that loading the (not usinged, but in-Manifest) cache files into the LS has an overhead of 1GB for your env/workspace? That seems excessive and like something to fix at the root. The OP was talking about peak memory usage, so I was assuming the indexing (which does actual package loading) to be the culprit.

You might think that you need the "per package" logic anyway and whether it's coming from the cloud or from the local cache would not add too much complexity either way, but it's probably more complex than that, I guess. Occasionally, I quickly change an added package (instead of a developed). It's nice if this use case is supported, but it's not the end of the world if it isn't.

Not entirely sure what you mean here. Indexing/cache deserialization is currently based on the selected environment, which seems correct to me under the assumption that it's cheap (which should hold with cloud indexing and a bunch of safeguards around cache serialization, e.g. the max depth stuff you implemented). Given that we're registering watchers for the *.tomls that define the environment, we should be handling changes fine (not sure how well tested that is though).

[PatrickHaecker]

Actually I wasn't so worried about the indexing CPU load. Sure, that is annoying, but kind of a one-time effect. What was really OOM-ing my machine was the RAM usage. This was disastrous without the max depth logic (~ 20 GB of RAM usage), but is now down to 2.5 GB for the corresponding Julia LanguageServer process. However, I haven't analyzed what that process entails, so only a part of these 2.5 GB is the deserialized cache. It's really the size of the deserialized cache that I am now most concerned about, because that's really RAM effectively permanently gone from my machine as I have VS Code with the Julia plugin permanently open.

However, it does not look too bad now:

~> du -hs ~/.config/Code/User/globalStorage/julialang.language-julia/symbolstorev5
1005M   /home/hp74lr/.config/Code/User/globalStorage/julialang.language-julia/symbolstorev5

If I can trust Claude, loading all cache files uses 1.6 GB of RAM, which seems to be an upper bound, because obviously not all packages are in the same environment. If I can further trust Claude, loading the caches files for my largest environment is using 29 MB of RAM, which I would consider perfectly reasonable.

Another script from Claude:

So a freshly reproduced LS with the full package stack and the env's symbol caches sits at ~0.75 GB — and the caches themselves are only 29 MiB of that. Your real process is at 2.4 GB. That extra ~1.6 GB is not symbol caches; it's StaticLint's analysis of your workspace + dependency sources plus accumulated type‑inference/method‑table — exactly the runtime state this script doesn't reproduce.

I guess that's the long version for "I don't really know". :-)

Indexing/cache deserialization is currently based on the selected environment, which seems correct to me under the assumption that it's cheap (which should hold with cloud indexing and a bunch of safeguards around cache serialization, e.g. the max depth stuff you implemented).

In the default environment I am always using only a small fraction of the available packages at the same time. While using the current environment is not wrong, I'd argue that the environment is a superset of which packages can be used, while what matters are the packages that are actually used (okay, maybe opened might also count). Therefore, I think it is a good idea to only have the currently used (imported) packages in RAM. I was under the impression that holding all the deserialized caches in RAM was more expensive than it seems to be, so while my point is probably still true, it is not as important as I thought.

I am not sure exactly to what that brings us. But it's probably again: Please judge whether the PR brings enough benefits to be worth it. Otherwise please close it.

[pfitzseb]

Ok, I've folded that commit int #119. Closing this.