Plans for a future rewrite

[langfield]

Considering a rewrite, possibly in another language.

In order to handle future distribution in the form of an addon, I will simply build binaries for all relevant architectures, and include all of them in the addon package. Barring this, people will have to build themselves with stack, but I will endeavor to ensure that is never necessary. There will be a microwrapper python program that only exists to detect the platform architecture and call the relevant Haskell executable.

It is very possible to build portable binaries for macOS, Linux, and Windows via stack in CI, and indeed there have been lengthy discussions on the topic.

The work on sql-parser will not have gone to waste, because we have proven that we don't need the anki python package for nearly as much as I thought. In particular, the clone and pull commands can absolutely be written using only sqlite-simple and a custom parser analogous to the one we wrote in Lark to parse the output of sqldiff. I think it may even be possible to do the entirety of the push operation without relying on calling functions from anki. We'll be reimplementing a tiny portion of the rust backend, but I believe it will be fairly simple.

To be continued...

[langfield]

I have been using gitlib, but it has occurred to me that doing this is making the problem more complex with no benefit. While gitlib is high-level, it is not high-level in the same way that GitPython or the git CLI is, and this is what we need. One would not need to think about blobs at all if doing these operations manually, and so there is no need for me to have to think about them while writing this code, which should just be an automation of the manual procedure.

As such, I'm switching to libgit (aren't these all named rather too similarly?) which is basically just a wrapper around the CLI.

[langfield]

Linking C libraries in Haskell

The following resources were useful:

• https://stackoverflow.com/a/53146721/12400477
• http://www.the-efficient-programmer.com/programming/link-c-lib-in-haskell.html

[DonCiervo]

Hi there, just stumbled across this project and will give it a try over the weekend - seems awesome!

What does the Haskell rewrite mean for the project, the current issues raised and ways to contribute? I'd love to contribute to the project in a way, but I'm not (yet) comfortable in Haskell.

[langfield]

@DonCiervo I'm looking forward to hearing how it works for you! Let me know if there's issues you run into that I might be able to fix. (Feel free to open more issues!)

The Haskell rewrite means that eventually, the python version will be deprecated. At that point, the command-line version of the tool will be distributed either as signed binaries, right here on Github, or alternatively, I may consider distributing the binaries via pip if it seems that's easier for people.

As for the current issues raised, they will likely not be fixed in the python codebase in the meantime. A lot of the open issues are simply features I thought might be necessary or improvements to the development workflow. There is only one real bug that could impact users, which is #109, and it only matters if you have notes whose cards are split between decks. This is at least somewhat uncommon, and only rears its head if you make edits to those notes on the markdown side.

I've found solutions to nearly all of these things on the Haskell side. The biggest problem I was running into was that it was simply too onerous and too slow to perform effective property-based testing on the python codebase. Hypothesis is the best thing folks have in the python ecosystem, but it is quite unpleasant to use, in my opinion.

Ways to contribute

I'm extremely happy to hear that you're interested in contributing, and I will bend-over backwards to make it as easy and enjoyable as possible for you if you are committed to helping out. There are a number of ways you can do this without even writing any code. Here's a list for you to consider:

• Just use the tool - Getting an idea for how users will interact with this thing is very important to me. Feel free to send as much feedback as you like in issues. Complaints are more useful than compliments. The documentation is thorough but sorely outdated. Complain about parts you don't like or stuff that needs to be updated, and I will fix it.
• Hang out in the issues - If you'd like to participate in high-level discussions of the design decisions happening during the rewrite, this would also be very helpful. There's lots of guesswork involved in deciding how things should function and what the end-users will need. A second opinion is always welcome.
• Bring in more users - Convincing more people in the Anki community to try this out would be helpful for obvious reasons. More feedback will make it better.
• Support deck maintainers - There are already a few maintainers on Github who I've convinced to use Ki. They may be on older versions, and they may run into trouble as I put new versions out, as we're still in alpha. Simply checking in with these guys to see how they're doing would be useful.
• Contribute to the codebase - This would be the most useful sort of help, obviously. But I would understand if you were unwilling to pick up a whole new language. However, I can tell you that coming from the python ecosystem, programming in Haskell feels like someone took a list of everything that was wrong or bad with python and fixed all of them, then added a bunch of extra improvements I'd have never even thought of on top of that. See Co-Star's explanation of why they use it.

On contributing code

There are a couple ways you could do this.

As an individual. If you're comfortable, I could simply pick some very simple tasks that would get you warmed-up to the language and the codebase (which is still miniscule), and you could just run with it.

As a side-along dev. If you like pair programming and have some free time for it, I could also spend some time each week bringing you up to speed on some sort of call using a collaborative environment or screenshare. It would be easier to get a handle on the repository and the implicit knowledge that goes along with the code in this way, for obvious reasons.

Whatever you choose to do, I encourage you not to take on too much too fast. I've been working on this for about a year now, and it would be much more useful to have a very small amount of someone's time over a long timescale rather than a couple weeks of intense attention.

[DonCiervo]

@langfield Cheers, appreciate the extensive answer. Maybe you should consider copying this information to a separate markdown file entirely so newcomers can get a quick overview. Personally I am also very partial to pinned issues for any "Wanna help? Here's how" type stuff.

Regarding the move away from Python, what is the long-term goal for ki ? Granted I haven't installed it yet, but it seems the basic functionality is there, judging from the documentation here and the fact it's being actively used. Surely the best next step to gain more users would be to package this into an add-on that lets people connect the decks of their choice to an upstream repo from within Anki.

Now, I am aware this is by no means as easy as I'm making it out to be - but neither is a rewrite in Haskell. I get that Haskell is one of the more beautiful languages to use, but if maintainability and stability is the goal, then the Haskell environment doesn't have the best reputation either, if I'm being honest.

[langfield]

Long-term goals

An addon is planned, but it's rather low priority for me. This tool is first-and-foremost for my own use, and I have no desire for a GUI. It is also quite a tricky problem to figure out how to allow users unfamiliar with git or the command-line to resolve merge conflicts intelligently.

I am more interested in rolling things out to deck maintainers first, and the next big user-facing feature I have planned is rendering collections via Jekyll in a Github actions workflow. See #41. Deck maintainers are a bit more tolerant of bugs and sharp edges, and make excellent alpha testers. They also better appreciate the usefulness of git.

If you trawl through the Anki forums, you will notice that one of the first complaints I got about Ki was that it was simply too slow. I did a lot of work on optimizing the Python version, and got quite far, but it would all be unnecessary in Haskell. This problem is really not computationally intensive enough to require heavy optimization in a compiled language in order for the user experience to be quick and snappy.

[DonCiervo]

Having an interactive github pages front-end would be very close to appeal to even a more basic user, so that would make a great feature, methinks. I should probably get a bit more familiar with the program as it is and look into the issues one by one before I ask any more questions that have already been answered somewhere.

The problem of a slowdown is obviously the big letdown in Python, and also the reason why the Anki backend has moved away from it. I was curious why you were deadset on using Haskell instead of Rust, for instance. But with the direction you're taking the project in, it only makes sense.

[langfield]

look into the issues one by one before I ask any more questions that have already been answered somewhere.

Don't worry about this too much, I'm happy to answer any questions you might have. Discussing existing issues and problems will probably serve to clarify them anyway.

The problem of a slowdown is obviously the big letdown in Python, and also the reason why the Anki backend has moved away from it. I was curious why you were deadset on using Haskell instead of Rust, for instance. But with the direction you're taking the project in, it only makes sense.

Yes after writing collection ops in pure python, I have a greater appreciation for just how fast the rust backend is.

However, I'm not sure I'd say the slowdown is the biggest letdown. Correctness guarantees, IMO, are more important. If haskell were slow and python were fast, I'd still strongly consider switching.

[langfield]

The first pseudo-correct output from clone!

(base) user@computer:~/proving-grounds$ cd ~/pkgs/ki && stack install && cd - && rm -rf dd && Ki-exe multifield/collection.anki2 dd && tree -a -I .git --filelimit 100 dd
Copying from /home/user/pkgs/ki/.stack-work/install/x86_64-linux/e79dab73246224887016704ede3d4c53affc0117456ea68c975b50a5c094bcee/8.10.7/bin/Ki-exe to /home/user/.local/bin/Ki-exe

Copied executables to /home/user/.local/bin:
- Ki-exe
/home/user/proving-grounds
Cloning media from Anki media directory '/home/user/proving-grounds/multifield/collection.media/'...
parts: ["Default"]
Committing contents to repository...
Done!
dd
├── Default
│   └── abcd
├── .gitignore
├── .gitmodules
├── .ki
│   ├── config
│   └── hashes
├── _media
└── _models
    ├── Mid 1673577708734.yaml
    ├── Mid 1673577708735.yaml
    ├── Mid 1673577708736.yaml
    ├── Mid 1673577708737.yaml
    ├── Mid 1673577708738.yaml
    ├── Mid 1673577710620.yaml
    ├── Mid 1673577743038.yaml
    └── Mid 1673577758568.yaml

4 directories, 13 files

The things that are not yet quite right:

• Missing .md suffix on note file
• Only sort field should be included in the note filename, assuming it is nonempty
• The model filenames should be the model names
• The model file contents are slightly wrong and incomplete

[langfield]

Even closer-to-correct clone output!

(base) user@computer:~/proving-grounds$ tree -I .git -a --filelimit 25 bb
bb
├── [1] Main Course
│   ├── [a] Option 1: Parisian French Audio
│   │   ├── I) French to English (Start here)
│   │   │   └── 5000 Most Common French Words  [500 entries exceeds filelimit, not opening dir]
│   │   └── II) English to French
│   │       └── 5000 Most Common French Words  [500 entries exceeds filelimit, not opening dir]
│   └── [b] Option 2: Canadian French Audio
│       ├── I) French to English
│       │   └── 5000 Most Common French Words  [500 entries exceeds filelimit, not opening dir]
│       └── II) English to French
│           └── 5000 Most Common French Words  [500 entries exceeds filelimit, not opening dir]
├── [A. 1] Irregular Verbs Training
│   ├── [a] Option 1: Most Frequent Conjs. Come First
│   │   └── 5000 Most Common French Words  [47 entries exceeds filelimit, not opening dir]
│   ├── [a] Option 2: One Verb at a Time
│   │   └── 5000 Most Common French Words  [47 entries exceeds filelimit, not opening dir]
│   ├── [b] Past Participle
│   │   └── 5000 Most Common French Words  [46 entries exceeds filelimit, not opening dir]
│   ├── [c] Present Participle
│   │   └──  Gerund
│   │       └── 5000 Most Common French Words  [47 entries exceeds filelimit, not opening dir]
│   ├── [d] Imperative
│   │   └── 5000 Most Common French Words  [47 entries exceeds filelimit, not opening dir]
│   └── [e] Literary
│       └──  Poetic Verb Tenses
│           ├── Past Historic
│           │   └── 5000 Most Common French Words  [47 entries exceeds filelimit, not opening dir]
│           └── Subjunctive Imperfect
│               └── 5000 Most Common French Words  [47 entries exceeds filelimit, not opening dir]
├── [A. 2] The Study of Sounds (Phonology)
│   ├── I) Basic IPA & Phonetics
│   │   ├── [1] IPA Letters
│   │   │   └── 5000 Most Common French Words
│   │   │       ├── baie.md
│   │   │       ├── boue.md
│   │   │       ├── chou.md
│   │   │       ├── clé.md
│   │   │       ├── cou.md
│   │   │       ├── doux.md
│   │   │       ├── fou.md
│   │   │       ├── gnouf.md
│   │   │       ├── goût.md
│   │   │       ├── jeune.md
│   │   │       ├── jeûne.md
│   │   │       ├── joue.md
│   │   │       ├── là.md
│   │   │       ├── loup.md
│   │   │       ├── mou.md
│   │   │       ├── nous.md
│   │   │       ├── peau.md
│   │   │       ├── pou.md
│   │   │       ├── roue.md
│   │   │       ├── si.md
│   │   │       ├── sort.md
│   │   │       ├── sous.md
│   │   │       ├── tout.md
│   │   │       ├── vous.md
│   │   │       └── zou.md
│   │   ├── [2a] Comparing French & English phonemes
│   │   │   └── 5000 Most Common French Words
│   │   │       ├── an-unreleased-stop-is-a-hard-consonant-that-does-not-relea.md
│   │   │       ├── compared-to-the-english-a-the-french-a-is-spoke.md
│   │   │       ├── compared-to-the-english-e-the-french-e-is-spoke.md
│   │   │       ├── compared-to-the-english-ɛ-the-french-ɛ-is-spoke.md
│   │   │       ├── compared-to-the-english-i-the-french-i-is-spoke.md
│   │   │       ├── compared-to-the-english-o-the-french-o-is-spoke.md
│   │   │       ├── compared-to-the-english-ɔ-the-french-ɔ-is-spoke.md
│   │   │       ├── compared-to-the-english-u-the-french-u-is-spoke.md
│   │   │       ├── how-is-the-french-b-d-g-different-from-english-b-d-g.md
│   │   │       ├── how-is-the-french-p-t-k-different-from-english-p-t-k.md
│   │   │       ├── place-of-articulation-is-where-a-consonant-is-formed-in-the.md
│   │   │       └── to-illustrate-the-difference-between-aspirated-and-unaspirat.md
│   │   ├── [2b] An illustration of French & English vowels
│   │   │   └── 5000 Most Common French Words
│   │   │       └── vowels-uncompressed02png.md
│   │   ├── [2c] Audio comparison of English & French Vowels
│   │   │   └── 5000 Most Common French Words
│   │   │       ├── comparing-french-n-and-english-nfrench-vowels-in-sequenc.md
│   │   │       ├── comparing-french-s-and-english-sin-french-s-the-tip-o.md
│   │   │       ├── comparing-french-z-and-english-zin-french-z-the-tip-o.md
│   │   │       ├── french-b-p-and-english-b-p-notice-how-french-release.md
│   │   │       ├── french-t-d-and-english-t-dnotice-how-french-releases.md
│   │   │       └── using-real-and-made-up-words-this-card-will-compare-french.md
│   │   └── [3] Notes on the IPA used in this deck
│   │       └── 5000 Most Common French Words
│   │           └── about-the-ipa-in-this-deck-let-a-be-any-vowel-let-l-be.md
│   ├── III) Aspirated h
│   │   ├── [1] Intro
│   │   │   └── 5000 Most Common French Words
│   │   │       └── this-deck-contains-words-that-are-aspirated-h-and-words-t.md
│   │   └── [2] Practice
│   │       └── 5000 Most Common French Words  [59 entries exceeds filelimit, not opening dir]
│   ├── II) Words With Irregular Pronunciation
│   │   ├── [1] Intro
│   │   │   └── 5000 Most Common French Words
│   │   │       └── in-a-way-or-another-the-words-of-this-deck-have-irregular-p.md
│   │   ├── [2a] Sorted by frequency
│   │   │   └── 5000 Most Common French Words  [96 entries exceeds filelimit, not opening dir]
│   │   └── [2b] Sorted by group
│   │       └── 5000 Most Common French Words  [96 entries exceeds filelimit, not opening dir]
│   ├── IV) Museum of Sounds
│   │   ├── [1] Intro
│   │   │   └── 5000 Most Common French Words
│   │   │       └── the-purpose-of-this-deck-is-to-showcase-the-sounds-of-the-fr.md
│   │   └── [2] Showcase (Formal Accent)
│   │       ├── a. Parisian French
│   │       │   └── 5000 Most Common French Words  [39 entries exceeds filelimit, not opening dir]
│   │       └── b. Canadian French
│   │           └── 5000 Most Common French Words  [39 entries exceeds filelimit, not opening dir]
│   └── V) The Skill of Listening
│       ├── [1] Intro
│       │   └── 5000 Most Common French Words
│       │       └── this-deck-has-the-purpose-of-training-your-listeningthere-a.md
│       ├── [2] Formal Accent
│       │   ├── a. Parisian
│       │   │   └── 5000 Most Common French Words  [1000 entries exceeds filelimit, not opening dir]
│       │   └── b. Canadian
│       │       └── 5000 Most Common French Words  [1000 entries exceeds filelimit, not opening dir]
│       └── [3] Informal Accent
│           └── a. Mixed Accents (98% Parisian)
│               ├── ii. Longer Sentences
│               │   └── 5000 Most Common French Words  [1323 entries exceeds filelimit, not opening dir]
│               └── i. Short Sentences
│                   └── 5000 Most Common French Words  [1492 entries exceeds filelimit, not opening dir]
├── [A. 3] Read & Speak Training (Where the Real Learning Happens)
│   ├── I) Basic
│   │   ├── [0] Note about this deck
│   │   │   └── 5000 Most Common French Words
│   │   │       └── 1-all-sentences-in-this-deck-are-present-in-the-main-deck.md
│   │   ├── [1] Read
│   │   │   └── 5000 Most Common French Words  [2305 entries exceeds filelimit, not opening dir]
│   │   └── [2] Speak
│   │       ├── a. Short sentences
│   │       │   └── 5000 Most Common French Words  [1000 entries exceeds filelimit, not opening dir]
│   │       ├── b. Medium-sized sentences
│   │       │   └── 5000 Most Common French Words  [676 entries exceeds filelimit, not opening dir]
│   │       └── c. Long sentences
│   │           └── 5000 Most Common French Words  [629 entries exceeds filelimit, not opening dir]
│   └── II) Intermediate
│       ├── [0] Note about this deck
│       │   └── 5000 Most Common French Words
│       │       └── 1-i-took-the-wiktionary-articles-for-each-word-in-the-5000.md
│       ├── [1] Read
│       │   └── 5000 Most Common French Words  [1154 entries exceeds filelimit, not opening dir]
│       └── [2] Speak
│           └── 5000 Most Common French Words  [1154 entries exceeds filelimit, not opening dir]
├── [B] Download this deck in e-book form
│   └── 5000 Most Common French Words
│       └── the-main-course-of-this-deck-5000-most-common-french-words.md
├── .gitignore
├── .gitmodules
├── .ki
│   ├── config
│   └── hashes
├── _media  [6755 entries exceeds filelimit, not opening dir]
└── _models
    ├── 01 BASIC.yaml
    ├── 5000 French Words 2.0 (E to F) C.yaml
    ├── 5000 French Words 2.0 (E to F).yaml
    ├── 5000 French Words 2.0 (F to E) C.yaml
    ├── 5000 French Words 2.0 (F to E).yaml
    ├── Basic (and reversed card).yaml
    ├── Basic (optional reversed card).yaml
    ├── Basic (type in the answer).yaml
    ├── Basic.yaml
    ├── Cloze (overlapping).yaml
    ├── Cloze.yaml
    ├── French aspirated h.yaml
    ├── French Ear Training.yaml
    ├── French IPA deck.yaml
    ├── French irregular pronunciation.yaml
    ├── French phonology.yaml
    ├── French sentences Read Training.yaml
    ├── French sentences Speak Training.yaml
    ├── French Verbs.yaml
    ├── French vowels comparison.yaml
    └── Intro card.yaml

100 directories, 77 files

[langfield]

See #39 for schemas.

[langfield]

Probably it was all too complicated even directly parsing the SQLite dumps in Haskell. If anyone is trying to build something similar and is using this as a reference, I'd highly recommend doing as much as you can as a shell script with no control flow. Git is meant to be used from the shell and you end up with a ton of unnecessary code trying to use it from inside another programming language.