Merging keyfreq data from multiple accounts

[grettke]

Good evening,

My approach for using my computer is to use different accounts depending on the
focus at hand. For example, all of my personal things like email and journaling
are done with the account 'gcr' and all of my day job stuff is done with the
account 'gcr-work'. My goals is just to keep parts of my life separate and there
are obvious benefits like focus and privacy. My Emacs config file is one thing
though that is mostly reused across all of my accounts and that means that I
am collecting keyfreq data that is kind of sitting on an island.

My personal account is my "main" account and that is where I store keyfreq data
in version control. That is the "master" account of sorts. My goal is to
periodically accrue all of my keyfreq data from all of my accounts simply by
copying it over every so often. The next task is how to utilize all if with
keyfreq.

My questions are:

• Anyone else doing something like this?
• How are you handling it?
• Have you gotten keyfreq to utilize data from multiple accounts for a
  situation like this?
• If not, what are your thoughts on where someone may begin?

Kind regards,

gcr

[glyph]

I am using keyfreq as a back-end for https://bitbucket.org/gvol/emacs-achievements and I have this same issue. Right now, I just try to push my keyfreq data after each editing session, and if I have a conflict, I just resolve it in favor of whichever session I recall having more significant changes to the data. Keyfreq produces gigantic diffs though, and a system which could periodically snapshot and aggregate would be very welcome indeed.

[glyph]

Here are my thoughts as to how such a system might work:

• change keyfreq to use a directory, rather than a single file
• write a new keyfreq file for each editing session
• define a roll-up window and interval: for example, once every 72 hours, roll up any files older than 1 week into the current snapshot.

this way, as long as you sync your working directory once every 24 hours or so, you will never lose any keyfreq history. any code which wishes to query keyfreq data will be looking at an in-memory snapshot including the current saved snapshot and any present editing sessions applied to it; keyfreq will have to go through at most 72 hours worth of editing-session files, which shouldn't be too huge.

[zimoun]

@grettke what do you mean by account ?
@glyph not sure that keyfreq should provide sort of logging system, because in short, it is what you are proposing if I understand well.

What could be useful is: be able to merge two emacs-keyfreq files, right ?

With this hypothetic new feature, you should be able to cover your needs ?

[glyph]

I wrote this Python script to merge two files from version control history which is letting me limp along for now:

from __future__ import print_function

from sexpdata import load, car, cdr
from os import popen, rename, system

KFFILE = "Emacs/data/keyfreq.el"


def triples(revision):
    cmd = "git cat-file blob {}:{}".format(revision, KFFILE)
    print("$ " + cmd)
    with popen(cmd) as f:
        sexp = load(f)
    for elt in sexp:
        mode = car(car(elt)).value()
        command = cdr(car(elt)).value()
        count = cdr(elt)
        yield ((mode, command), count)
import sys
toMerge = sys.argv[1]

here = dict(triples("HEAD"))
there = dict(triples(toMerge))
with popen("git merge-base HEAD {}".format(toMerge)) as f:
    commonParentRev = f.read().strip()
parent = dict(triples(commonParentRev))

def getDeltas(d1, d2):
    deltas = {}
    for key in set(d1.keys()) | set(d2.keys()):
        value1 = d1.get(key, 0)
        value2 = d2.get(key, 0)
        # backwards merges can cause keys to go negative.
        deltas[key] = max(0, value2 - value1)
    return deltas

# get the deltas from parent to 'here'
hereDeltas = getDeltas(parent, here)

# get the deltas from parent to 'there'
thereDeltas = getDeltas(parent, there)

# apply the deltas to parent

def applyDeltas(base, deltas):
    applied = {}
    for key in set(base.keys()) | set(deltas.keys()):
        baseValue = base.get(key, 0)
        delta = deltas.get(key, 0)
        applied[key] = baseValue + delta
    return applied

merged = applyDeltas(applyDeltas(parent, hereDeltas), thereDeltas)

# serialize the deltas into the path

list = []

with open("kf-merged.el", "wb") as f:
    f.write("(")
    for ((mode, command), count) in sorted(merged.items()):
        f.write("(({} . {}) . {})\n".format(mode, command, count))
    f.write(")")
rename("kf-merged.el", KFFILE)
system("git add {}".format(KFFILE))

[grettke]

@zimoun I meant user account in the operating system.

[zimoun]

Really nice this python script !!

What I have in mind is more to adapt current functions, such as keyfreq-table-load, keyfreq-list and keyfreq-table-save, and add an interactive new one, e.g., keyfreq-merge.

Let keyfreq-file-A and keyfreq-file-B, which both contains stats.
Something in the flavor, call this new function will do: using keyfreq-table-load, you load the table-A from keyfreq-file-A and the table-B from keyfreq-file-B, then you append the two tables to produce a new table (which is basically a list), and you apply keyfreq-list to this new table (this should merge the stats from the both files, adding the two freq values of common functions), and finish by save it in keyfreq-file-C.

Are we on the same page ?

@grettke If you are using Unix-like and you do not want keyfreq stats for each account, i.e., unique common shared stats, the easiest seems to have one folder at the root. Basically, it is what /etc does. Maybe I missed your point.

[grettke]

@zimoun Gotcha, thanks!

[zimoun]

@grettke @glyph I added the interactive keyfreq-merge function. Is it useful ?

I have not fully tested with huge several files. It is possible that the lock-file will block a bit, even it is not required on the paper in some case. In other words, the function keyfreq-save-table should be re-written or options should be given to the locks function.

Moreover, the batch mode leads to unexpected freezes.... but it does not seems to coming from the new feature. Anyway. The idea would be to use: emacs --batch --eval="(keyfreq-merge--do file-A file-B store-in-C)"

[grettke]

@zimoun As soon as I've got another file I'm going to test it out. Thanks you!