Fix Akka's and Pekko's doomsday-wildcard config

[mkurz]

Caution

I am pretty pretty sure the current code is flawed

doomsday-wildcard is off by default (see config).

All I do now is, I set includes to "**". Now I expect that includes just doesn't have any effect at all - basically like having an empty includes (which is the default).
So I end up with this (partially) config:

{
  "kamon": {
    "instrumentation": {
      "pekko": {
        "filters": {
          "actors": {
            "doomsday-wildcard": "off",
            "track": {
              "excludes": [
                "*/system/**",
                "*/user/IO-**"
              ],
              "includes": [
                "**"
              ]
            }
          }
        }
    }
  }
}

However - I do see actors getting tracked now which should not - because the doomsday-wildcard just does not work...

Let's look at the code:

Kamon/instrumentation/kamon-pekko/src/main/scala/kamon/instrumentation/pekko/PekkoInstrumentation.scala

Lines 115 to 125 in db2c3ed

	private def safeFilter(config: Config, allowDoomsday: Boolean): Filter = {
	val includes = config.getStringList("includes").asScala
	if (!allowDoomsday && includes.contains("**")) {
	val newIncludes = "includes = " + includes.filter(_ == "**").map(s => s""""$s"""").mkString("[ ", ", ", " ]")
	val safeFilterConfig = ConfigFactory.parseString(newIncludes).withFallback(config)
	Filter.from(safeFilterConfig)
	} else Filter.from(config)
	}
	}

val newIncludes always (!) ends up as string "includes = [ "**" ]":

$ scala
Welcome to Scala 2.13.16 (OpenJDK 64-Bit Server VM, Java 17.0.15).
Type in expressions for evaluation. Or try :help.

scala> val includes = List("one", "**", "two") // simulates: config.getStringList("includes").asScala 
val includes: List[String] = List(one, **, two)

scala> val newIncludes = "includes = " + includes.filter(_ == "**").map(s => s""""$s"""").mkString("[ ", ", ", " ]")
val newIncludes: String = includes = [ "**" ]

So you always filter out the ** wildcard and therefore always build the string "includes = [ "**" ]"!
Afterwards you always override the provided config with

val safeFilterConfig = ConfigFactory.parseString(newIncludes).withFallback(config)
// always (!) ends up as
val safeFilterConfig = ConfigFactory.parseString("includes = [ \"**\" ]").withFallback(config)

Correct would be to either use filterNot or filter(_ != "**"):

scala> val newIncludes = "includes = " + includes.filterNot(_ == "**").map(s => s""""$s"""").mkString("[ ", ", ", " ]")
val newIncludes: String = includes = [ "one", "two" ]

IMHO it would be nice to push out a 2.7.8 release with this fix 😉
Maybe even including #1402?
Thanks!

[ihostage]

Maybe even including #1402?

And #1395 🤞 😄

[ivantopo]

🤦

You are absolutely right @mkurz, what a catch!

Regarding releasing 2.7.8 with this fix: is that something that maybe you or someone you know needs? we already have a bunch of changes for the new Kanela in master and that should be released as 2.8.0. I could cherry pick a few commits on top of 2.7.7 and run the release, but I'll only do it if someone actually shows interest and needs these fixes. If nobody shows interest we can probably just move to 2.8.0.

[mkurz]

What's the ETA of 2.8?

[ivantopo]

I'm planning to deploy it to our production services in APM today and, if all is fine, release by the end of the week. The functionality with the new Kanela is roughly the same (it does less things, but many of those extra things were not used before) so I'm expecting a smooth transition.

[ivantopo]

FYI: I published Kamon 2.8.0-beta.1 and so far it is working fine for us. I am testing in a few different apps besides what we have in APM and couldn't find any issues so far.