Update to support latest versions of ktor, exposed, kotlin, etc

[kmayoral]

Hi there,

I'm a new user of ktor and attempted to use the Ktor project generator (https://start.ktor.io/) to start a new project and included a few of your plugins into my project template.

When I went to run the sample, I got some build errors due to the fact that the generated project runs with Exposed 0.60.0 while the latest supported here currently is 0.44.0.

I went down the rabbit hole of checking out this repo and attempted to update to support the latest ktor (3.1.1), exposed (0.60.0) and kotlin (2.1.10) to hopefully be compatible with the project that is generated from the start.ktor.io site.

Through the process, I had to update several other dependencies and make several modifications to the task scheduling related implementations for postgres, mongo and redis implementations.

Some notes:

• I learned that the kreds library didn't expose some transitive dependencies as needed by our usages, so also added a few of those to resolve some build errors
• I wasn't able to get any of the server scheduling lock task tests to run successfully without significant modifications to the core logic. I'm not sure if those tests were working previously or being silently ignored, but I'd appreciate a code review on those sections to confirm.
• Several of these tests and others appear to be a little flaky on my builds but I've gotten them all to pass in one go several times.
• To resolve the last issues with the redis lock manager, I had to wrap the acquire and release lock key calls with a mutex owned by the generated key. This feels like a smell to me but I'm happy to let anyone take a stab at cleaning things up as I mentioned this is my first time navigating into using many of these libraries.

Thank you!

[Flaxoos]

Hi @kmayoral thank you for your adoption and contribution!

To help me understand the issue and review your proposed changes, would you be able to produce a minimal reproducible example?
My goal is for the libraries to be able to be used without dependency mismatch issues, so I want to see what the issues are, at least when it comes to the versioning issues

It would also help if you could attach any errors you got, if they are not related to your specific use case, but rather to any version misalignment

Thanks!

[kmayoral]

Hi @Flaxoos , of course! I apologize for not submitting more details earlier, I definitely should have.

Repro Steps

Here are some steps I went through just now to reproduce:

• I went to https://start.ktor.io/ to create a ktor project
• I selected the following plugins: Content Negotiation, Exposed, kotlinx.serialization, Postgres, Routing, and from this repository Rate Limiting and Task Scheduling. My suspicion is that all we need is Exposed and Task Scheduling to truly reproduce the issue based on the details shared below, but I figured this example was small enough to test with.
  
• This generated a template project included here: mre.zip
• Upon opening the project in IntelliJ and gradle syncing, I ran the only task that was generated EngineMain to attempt to build and run the project.
  

Upon attempting to run the task, I received this error:

Exception in thread "main" java.lang.NoSuchMethodError: 'org.jetbrains.exposed.sql.Column io.github.flaxoos.ktor.server.plugins.taskscheduling.managers.lock.database.DefaultTaskLockTable.integer(java.lang.String)'
	at io.github.flaxoos.ktor.server.plugins.taskscheduling.managers.lock.database.DefaultTaskLockTable.<clinit>(JdbcLockManager.kt:136)
	at com.flaxoos.AdministrationKt.configureAdministration$lambda$14$lambda$7$lambda$6$lambda$5(Administration.kt:53)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTransaction$run(ThreadLocalTransactionManager.kt:355)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTransaction$lambda$11(ThreadLocalTransactionManager.kt:402)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.keepAndRestoreTransactionRefAfterRun(ThreadLocalTransactionManager.kt:410)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.inTopLevelTransaction(ThreadLocalTransactionManager.kt:401)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.transaction$lambda$5(ThreadLocalTransactionManager.kt:310)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.keepAndRestoreTransactionRefAfterRun(ThreadLocalTransactionManager.kt:410)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.transaction(ThreadLocalTransactionManager.kt:259)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.transaction(ThreadLocalTransactionManager.kt:237)
	at org.jetbrains.exposed.sql.transactions.ThreadLocalTransactionManagerKt.transaction$default(ThreadLocalTransactionManager.kt:236)
	at com.flaxoos.AdministrationKt.configureAdministration$lambda$14$lambda$7(Administration.kt:53)
	at io.github.flaxoos.ktor.server.plugins.taskscheduling.managers.lock.database.JdbcLockManagerKt.jdbc(JdbcLockManager.kt:168)
	at com.flaxoos.AdministrationKt.configureAdministration$lambda$14(Administration.kt:46)
	at io.ktor.server.application.CreatePluginUtilsKt.createPluginInstance(CreatePluginUtils.kt:254)
	at io.ktor.server.application.CreatePluginUtilsKt.access$createPluginInstance(CreatePluginUtils.kt:1)
	at io.ktor.server.application.ApplicationPluginImpl.install(CreatePluginUtils.kt:89)
	at io.ktor.server.application.ApplicationPluginImpl.install(CreatePluginUtils.kt:77)
	at io.ktor.server.application.ApplicationPluginKt.install(ApplicationPlugin.kt:121)
	at com.flaxoos.AdministrationKt.configureAdministration(Administration.kt:35)
	at com.flaxoos.ApplicationKt.module(Application.kt:10)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:577)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method.callMethod(CallerImpl.kt:97)
	at kotlin.reflect.jvm.internal.calls.CallerImpl$Method$Static.call(CallerImpl.kt:106)
	at kotlin.reflect.jvm.internal.KCallableImpl.callDefaultMethod$kotlin_reflection(KCallableImpl.kt:207)
	at kotlin.reflect.jvm.internal.KCallableImpl.callBy(KCallableImpl.kt:112)
	at io.ktor.server.engine.internal.CallableUtilsKt.callFunctionWithInjection(CallableUtils.kt:120)
	at io.ktor.server.engine.internal.CallableUtilsKt.executeModuleFunction(CallableUtils.kt:36)
	at io.ktor.server.engine.EmbeddedServer.launchModuleByName$lambda$26(EmbeddedServerJvm.kt:380)
	at io.ktor.server.engine.EmbeddedServer.avoidingDoubleStartupFor(EmbeddedServerJvm.kt:404)
	at io.ktor.server.engine.EmbeddedServer.launchModuleByName(EmbeddedServerJvm.kt:379)
	at io.ktor.server.engine.EmbeddedServer.instantiateAndConfigureApplication$lambda$25(EmbeddedServerJvm.kt:360)
	at io.ktor.server.engine.EmbeddedServer.avoidingDoubleStartup(EmbeddedServerJvm.kt:386)
	at io.ktor.server.engine.EmbeddedServer.instantiateAndConfigureApplication(EmbeddedServerJvm.kt:358)
	at io.ktor.server.engine.EmbeddedServer.createApplication(EmbeddedServerJvm.kt:159)
	at io.ktor.server.engine.EmbeddedServer.start(EmbeddedServerJvm.kt:289)
	at io.ktor.server.netty.EngineMain.main(EngineMain.kt:25)

which is what first triggered my investigation into this repository.

Initial Investigation

From my initial investigation, I noticed that the error we were seeing was complaining about

extra-ktor-plugins/ktor-server-task-scheduling/ktor-server-task-scheduling-jdbc/src/jvmMain/kotlin/io/github/flaxoos/ktor/server/plugins/taskscheduling/managers/lock/database/JdbcLockManager.kt

Line 62 in 7afcb7d

repetitionAttempts = 0

where we set repetitionAttempts = 0 on a Transaction object from the Exposed library. The stack trace doesn't immediately indicate that because this library was referencing the 0.44.0 version of Exposed where repetitionAttempts was still defined but at runtime the 0.60.0 version of Exposed was being used where that field had been replaced with maxAttempts. You can see this change outlined in the Exposed v0.50.0 breaking changes page here.

So that was the recap of what started my attempts at updating this repository.

My Changes

I went ahead with the intention of updating the Exposed dependency version in this repo to 0.60.0 and updating the line of code mentioned above but when I went to run the Whole Project tasks which also runs all the associated integration tests, I noticed that many were not passing.

I had also noticed this before when I saw that dependabot triggered version update PRs were not passing, but that made sense since there was an associated code change that was also required.

Based on this, I decided to try and update many of the other dependencies to their latest versions and then go through to see what issues remained to fix the tests. That's what led to the remaining changes that resolved the integration test issues seen specifically in the taskscheduling plugin. And that is where this PR stands currently.

Current Main Branch Test State

For reference (I should have done this much earlier in my process), I ran the same Whole Project task on the current HEAD of the main branch and noticed that some of the same errors were still present such as in the mongodb lock manager tests which were a result of the reactor-io dependency not being included in the test classpath and errors in the redis lock manager tests which were a result of the kreds dependency not transitively including some other required dependencies as well (such as netty).

Here are some screenshots of the test results and stacktraces for reference.

2025-03-19 09:02:14.373 [pool-1-thread-1 @coroutine#40] INFO  i.g.f.k.s.p.t.TaskSchedulingPluginTest - Adding task: Test Kron Task: 1
2025-03-19 09:02:14.373 [pool-1-thread-1 @coroutine#40] INFO  i.g.f.k.s.p.t.TaskSchedulingPluginTest - Adding task: Test Kron Task: 2
2025-03-19 09:02:14.373 [pool-1-thread-1 @coroutine#40] INFO  io.ktor.test - Application started in 0.0 seconds.
Exception in thread "DefaultDispatcher-worker-3 @coroutine#45" java.lang.NoClassDefFoundError: reactor/core/publisher/Mono
	at com.mongodb.reactivestreams.client.internal.OperationExecutorImpl.execute(OperationExecutorImpl.java:117)
	at com.mongodb.reactivestreams.client.internal.MongoOperationPublisher.createWriteOperationMono(MongoOperationPublisher.java:509)
	at com.mongodb.reactivestreams.client.internal.MongoOperationPublisher.createIndexes(MongoOperationPublisher.java:423)
	at com.mongodb.reactivestreams.client.internal.MongoOperationPublisher.createIndex(MongoOperationPublisher.java:416)
	at com.mongodb.reactivestreams.client.internal.MongoCollectionImpl.createIndex(MongoCollectionImpl.java:722)
	at com.mongodb.kotlin.client.coroutine.MongoCollection.createIndex(MongoCollection.kt:1369)
	at io.github.flaxoos.ktor.server.plugins.taskscheduling.managers.lock.database.MongoDBLockManager.initTaskLockTable(MongoDBLockManager.kt:85)
	at io.github.flaxoos.ktor.server.plugins.taskscheduling.managers.lock.database.DatabaseTaskLockManager.init(DatabaseTaskLockManager.kt:19)
	at io.github.flaxoos.ktor.server.plugins.taskscheduling.TaskSchedulerPluginKt$initializeTaskManagers$1$1$1.invokeSuspend(TaskSchedulerPlugin.kt:57)
	at _COROUTINE._BOUNDARY._(CoroutineDebugging.kt:42)
	at io.github.flaxoos.ktor.server.plugins.taskscheduling.TaskSchedulerPluginKt$TaskScheduling$2$1$1$1$1.invokeSuspend(TaskSchedulerPlugin.kt:42)
	Suppressed: kotlinx.coroutines.internal.DiagnosticCoroutineContextException: [CoroutineId(45), "coroutine#45":StandaloneCoroutine{Cancelling}@255f2939, Dispatchers.Default]

2025-03-19 09:02:47.641 [pool-1-thread-1 @coroutine#4] INFO  io.ktor.test - No ktor.deployment.watch patterns specified, automatic reload is not active.
2025-03-19 09:02:47.648 [pool-1-thread-1 @coroutine#4] INFO  i.g.f.k.s.p.t.TaskSchedulingPluginTest - Adding task: Test Kron Task: 1
2025-03-19 09:02:47.652 [pool-1-thread-1 @coroutine#4] INFO  i.g.f.k.s.p.t.TaskSchedulingPluginTest - Adding task: Test Kron Task: 2
2025-03-19 09:02:47.842 [nioEventLoopGroup-2-1] WARN  io.netty.channel.ChannelInitializer - Failed to initialize a channel. Closing: [id: 0xaa28512a]
java.lang.NoClassDefFoundError: io/netty/handler/timeout/ReadTimeoutHandler
	at io.github.crackthecodeabhi.kreds.connection.KonnectionImpl$createChannelInitializer$1.initChannel(KonnectionImpl.kt:82)
	at io.github.crackthecodeabhi.kreds.connection.KonnectionImpl$createChannelInitializer$1.initChannel(KonnectionImpl.kt:70)
	at io.netty.channel.ChannelInitializer.initChannel(ChannelInitializer.java:129)
	at io.netty.channel.ChannelInitializer.handlerAdded(ChannelInitializer.java:112)
	at io.netty.channel.AbstractChannelHandlerContext.callHandlerAdded(AbstractChannelHandlerContext.java:1114)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAdded0(DefaultChannelPipeline.java:609)
	at io.netty.channel.DefaultChannelPipeline.access$100(DefaultChannelPipeline.java:46)
	at io.netty.channel.DefaultChannelPipeline$PendingHandlerAddedTask.execute(DefaultChannelPipeline.java:1463)
	at io.netty.channel.DefaultChannelPipeline.callHandlerAddedForAllHandlers(DefaultChannelPipeline.java:1115)
	at io.netty.channel.DefaultChannelPipeline.invokeHandlerAddedIfNeeded(DefaultChannelPipeline.java:650)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.register0(AbstractChannel.java:514)
	at io.netty.channel.AbstractChannel$AbstractUnsafe.access$200(AbstractChannel.java:429)
	at io.netty.channel.AbstractChannel$AbstractUnsafe$1.run(AbstractChannel.java:486)
	at io.netty.util.concurrent.AbstractEventExecutor.runTask(AbstractEventExecutor.java:173)
	at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:166)
	at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:470)
	at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:569)
	at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:997)
	at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
	at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
	at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: java.lang.ClassNotFoundException: io.netty.handler.timeout.ReadTimeoutHandler
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:641)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
	... 21 common frames omitted

So it seems that my changes didn't introduce those issues but they were already present (but should be fixed now by this PR). I'm fairly comfortable with my changes to address the testing issues that were already present with the exception of the addition of a Mutex to control local access to the redis lock manager. I'd still like to revisit that and see if we can figure out what the true root cause is and a cleaner fix.

Thanks for your time in reading this and I hope it helps in some way. Perhaps it might make more sense to fix these things little by little, but I wasn't sure how much bandwidth you had to devote to smaller PRs which is why I started with this one. Thanks for a great set of plugins, these are awesome!!

[kmayoral]

If we want to handle these changes in chunks, here is the first PR to handle resolving the existing integration test errors on main: #86

Thank you again!

[kmayoral]

Another update for you @Flaxoos , I was able to isolate the version of Exposed that first causes the jdbc based integration tests to fail. I commented on the other PR which you can see at #86 (comment)

[Flaxoos]

@kmayoral Amazing, thank you for the detailed report. Let me look into this in the coming days. Your approach of updating the dependencies is a good one. Another approach would be to shadow the exposed dependencies the implementation relies on, so the implementation wouldn't rely on dependencies provided by the consuming project, since the db used by the consuming project is separate from the one the plug-in uses (I say this with limited certainty, without looking at the code at the moment of writing this)

[Flaxoos]

@kmayoral I finally got around to paying attention to this, sorry for the delay. I merged your other smaller PR already.
For this, do you mind putting this on a branch in the original repository? this way it will run the workflows for the PR. currently they aren't run

[kmayoral]

Hi @Flaxoos , thanks for the reviews! Happy to help where I can. Not sure I follow though, do you mean you'd like me to push this branch to to the Flaxoos/extra-ktor-plugins repository? Wouldn't I need to have push access to your repo for that? If you'd like to add me, I'd be happy to push directly, thanks!

[Flaxoos]

Are you not able to crate a branch? There aren't any protection rules to any branch besides main. In any case I invited you to collaborate but please try before accepting it, I just want to see if it's possible

[kmayoral]

Hi @Flaxoos , I tried pushing directly to your repo prior to accepting the invitation and got the following error:

git push --set-upstream origin km/update-to-latest-deps
ERROR: Permission to Flaxoos/extra-ktor-plugins.git denied to kmayoral.
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.

But tried again immediately after accepting the invite and was able to push successfully. I created a new PR #89 which you can see should contain the same commit and merge commit update you had applied to this branch. Thanks!