2024-08-12 - KDB Project Call Meeting Minutes

[alvin-c-shih]

Date

Monday, August 12, 2024 - 9:30am ET / 2:30pm UK

// Second Monday of every month

Attendees

Fullname	Affiliation	GitHub Username
Alvin Shih	Morgan Stanley	@alvin-c-shih
Aaron Davies	Morgan Stanley	@adavies42
Peter Gyorok	KX Systems	@gyorokpeter
Karl Moll	FINOS	@karlmoll
Ross Duffy	Morgan Stanley	@rdffy
Rian Ó Cuinneagáin	KX Systems	@rianoc
Ryan Hamilton	TimeStored	@ryanhamilton
Gary Davies	Data Intellect	@drgdavies

Meeting notices

• FINOS Project leads are responsible for observing the FINOS guidelines for running project meetings. Project maintainers can find additional resources in the FINOS Maintainers Cheatsheet.

• All participants in FINOS project meetings are subject to the LF Antitrust Policy, the FINOS Community Code of Conduct and all other FINOS policies.

• FINOS meetings involve participation by industry competitors, and it is the intention of FINOS and the Linux Foundation to conduct all of its activities in accordance with applicable antitrust and competition laws. It is therefore extremely important that attendees adhere to meeting agendas, and be aware of, and not participate in, any activities that are prohibited under applicable US state, federal or foreign antitrust and competition laws. Please contact legal@finos.org with any questions.

• FINOS project meetings may be recorded for use solely by the FINOS team for administration purposes. In very limited instances, and with explicit approval, recordings may be made more widely available.

Agenda

• (5 min) Convene, roll call, welcome new people.

• Display FINOS Antitrust Policy summary slide

• Review Meeting Notices (see above)

• (5 min) Approve previous meeting minutes.

  • Approved by acclamation.

• (10 min) Ryan Hamilton's "The Future of kdb+?" blog post.

  • https://www.timestored.com/b/the-future-of-kdb/
    • "Likely Outcome: – Kdb+ can hold their existing clients but haven’t and won’t get the 2nd tier firms as they either want cloud native or something else. The previous major customers for this had to invest heavily to build their own platform."
      • No one on the call had any news of inroads into non-finance verticals.
    • "The best open source companies have stolen the best kdb+ ideas:"
      • How much of a moat does libq have (eg. as a plugin for Snowflake?).
    • "3. Reduce the steep learning curve. "
      • verbose.q was one attempt: Initial commit for verbose.q #87
      • Making packages easier to install was another:
        • https://github.com/finos/kdb/tree/main/q/dep
        • https://github.com/jshinonome/kuki
      • PyKX also looking to be easier to learn than q.
  • What does the developer pipeline look like?
    • Consulting companies able to hire senior developers.
    • Not as much demand for junior.
  • Should the scope of this project be widened to include Arrow / DuckDB?
    • Or would that be a separate project?

• (5 min) Conor Twomey moving on to an AI startup?

  • https://www.linkedin.com/posts/conortwomey_leadership-kx-fd-activity-7226880674257891329-1L0s
  • Shane Mulholland and James Corcoran have also moved on after long tenures at KX.

• (10 min) PyKX Discussion Continued

  • Some find it hard to learn q without "immersion".
    • Is there a "learning plan" that can teach useful abstractions through PyKX in stages?
    • Course on kdb+ Architecture available now:
      • https://learninghub.kx.com/courses/kdb-architecture/
    • Traditional q training comes at it from a programming rather than querying perspective.
    • PyKX interface is now my Pythonic and lessons have been reworked:
      • https://learninghub.kx.com/courses/introduction-to-pykx/
      • Iterators have been added to the query interface.
      • Named versions of adverbs check if operator doesn't make sense.
        • Error messages and docs make it easier to move to the correct one.
      • Context awareness and fusing operator with adverbs should also help.

• (5 min) Why GitHub Copilot will struggle with q.

  • https://thakkarparth007.github.io/copilot-explorer/posts/copilot-internals.html#a-close-look-at-snippet-extraction
    • Depending upon the options, this either uses a “Fixed window Jaccard matcher” or an “Indentation based Jaccard Matcher”.
  • https://github.blog/2024-04-04-what-is-retrieval-augmented-generation-and-what-does-it-do-for-generative-ai/
    • "RAG doesn’t require embeddings or vector databases.".
    • Remains to be seen whether q developers are motivated to document more to help the coding assistants.
    • ChatGPT purportedly able to explain Collatz Conjecture implemented in q.
    • Overloaded operators can be a problem (both on types and on values!).

• (10 min) Demo of GitHub Copilot Enterprise Chat using https://github.com/KxSystems/docs as a "knowledge base" (@github #kb):

  • https://docs.github.com/en/enterprise-cloud@latest/copilot/managing-copilot/managing-github-copilot-in-your-organization/managing-github-copilot-features-in-your-organization/managing-copilot-knowledge-bases
  • Easy to forget the step of picking the knowledge base in GitHub Copilot Enterprise.
  • Tried an example of space-delimited file parsing.
    • Generated a spec of "S S" when the correct spec was SS.
    • Generated code used of show which has not been required since kdb+ 2.3.
  • Also failed to generate code that used fby correctly.
  • FINOS has collaborations related to AI for finance.
    • Could get some GPU cycles for LLM fine tuning.

• (5 min) AOB, Q&A, Adjourn.

Join Zoom Meeting

• https://zoom.us/j/94523415326?pwd=NkJDaUtUUlZTcHVwcUJHdG0yRFJUdz09
• Meeting ID: 945 2341 5326
• Passcode: 361990
• Find your local number: https://zoom.us/u/abTtYRZ0B6

[adavies42]

Aaron Davies/Morgan Stanley

[rdffy]

Ross Duffy/ Morgan Stanley

[gyorokpeter]

Péter Györök / KX

[karlmoll]

Karl Moll / FINOS

[rianoc-kx]

Rian Ó Cuinneagáin / KX

[drgdavies]

Gary Davies / Data Intellect

[ryanhamilton]

Ryan Hamilton / TimeStored
Link I mentioned:
https://www.timestored.com/kdb-guides/kola-python-api

[rianoc-kx]

• Latest LinkedIn event: Mastering kdb+ Architecture: A Panel Discussion
• Newest training courses:
  • kdb+ Architecture course
  • Updated PyKX course

[alvin-c-shih]

Alvin Shih / Morgan Stanley