sort symbols in order of frequency rather than lexicographically

[cstyan]

for #249

as we keep track of symbols in head we now also keep track of how many times we've seen that symbol, and when we write the symbols in IndexWriter we sort the symbols in order of frequency seen before writing them

[fabxc]

Thanks for working on this! Currently we appear to have some severe bugs in Prometheus 2.1 tied to the storage. Thus, I'd suggest we freeze any new features to prometheus/tsdb until things go back to being stable.

So this PR will probably be on hold for a bit.

[cstyan]

@fabxc no problem! I can pick this (#249 ) back up when we're in a more stable state. Is there anything I can help with regarding the bugs in 2.1?

[cstyan]

@fabxc will the 2.2.0 release unblock this?

[cstyan]

rebased off master, fixed a merge conflict that I'd missed when I pushed last

[gouthamve]

LGTM. I'll run it locally for a while and report back.

[gouthamve]

Why is this 0? It should be:

for s, num := range h.head.symbols {
    res[s] = num
}

[cstyan]

If I had a reason, I don't remember at this point :)

[cstyan]

changed the value assigned to the symbols in Reader in head.go and squashed it into the original commit

let me know if there's anything I can do to help test this :)

[cstyan]

@gouthamve @fabxc are these changes still relevant?

[krasi-georgiev]

Shouldn't you sort the symbols in the index writer , before writing it to disk?
IIUC this is where we want to save some space.

https://github.com/prometheus/tsdb/blob/c848349f07c83bd38d5d19faa5ea71c7fd8923ea/index/index.go#L343
instead of this , sort by popularity.

[cstyan]

@krasi-georgiev I'll have to double check, haven't looked at this in a while

[cstyan]

@krasi-georgiev is that not what is happening here: https://github.com/prometheus/tsdb/pull/280/files#diff-71ebe2bcf31a915b1fa3b3b289d5d31dR354 ?

rebased off master to fix the conflict in head.go

[krasi-georgiev]

failing tests

[krasi-georgiev]

Can you test how much are the savings by this change?

Also I don't see any test to ensure the behaviour that Symbols are saved ordered by frequency.
something like

• Add some symbols
• Save index
• Read index
• Check Symbols order.

[krasi-georgiev]

should also update
https://github.com/prometheus/tsdb/blob/master/docs/format/index.md#symbol-table

[cstyan]

Added a test for the sorting of symbols

I'm not sure if we want to get back the frequency #'s when we read the symbols back out of the table?

[brian-brazil]

I don't see a reason to expose that in the API.

[krasi-georgiev]

failing tests

also

Can you test how much are the savings by this change?

[cstyan]

@krasi-georgiev

Can you test how much are the savings by this change?

Yes I'll have a look at, I guess including a benchmark test? But if you read #249 the goal is to reduce the size of the index file.

@brian-brazil

I don't see a reason to expose that in the API.

I'll have to double check. When I was reading the use of the index and block reader to get the symbols, when compaction happens we read the current symbols, determine which are still in use, and then write those to a new index. In that case we would want the frequencies to persist across that write. But I've probably just misread what's happening.

[krasi-georgiev]

Yes I'll have a look at, I guess including a benchmark test? But if you read #249 the goal is to reduce the size of the index file.

yes this is what I meant , how much is the index file size reduced by this change.

[krasi-georgiev]

@cstyan would you have time to continue with this?
Otherwise I can try to find some time to continue from here.

[cstyan]

@krasi-georgiev yeah I should have some time next week, if you wanted to try something before then feel free.

[krasi-georgiev]

updated to the latest master and resolved the conflicts.

Now will run some tests locally to compare the index file savings with this change.

[krasi-georgiev]

using the following test I don't see any difference in the index file size.

The index file size with or without the changes in this PR is 180Mb.

func TestIndexSize(t *testing.T) {
	tmpdir, err := ioutil.TempDir("", "testIndex")
	testutil.Ok(t, err)
	var dirs []string

	for i := int64(0); i < 5; i++ {
		dirs = append(dirs, createBlock(t, tmpdir, genSeries(1000, 10, i*1000, i*1000+1000)))

	}
	compactor, err := NewLeveledCompactor(nil, log.NewNopLogger(), []int64{1000000}, nil)
	testutil.Ok(t, err)

	uid, err := compactor.Compact(tmpdir, dirs, nil)
	testutil.Ok(t, err)
	fmt.Println(tmpdir + "/" + uid.String())
}

This generates random series so it should generate enough churn.

[krasi-georgiev]

ping @cstyan