Dapr actor incompability when using ResponseBody observability

[papabytes]

Describe the bug
After enabling ResponseContent Observability we started receiving error codes from Dapr actors. The only difference we made in our code was adding the following line:

app.UseRequestTracking(trackingOptions =>
{
  trackingOptions.IncludeResponseBody = true;
});

To Reproduce
While configuring tracking middleware, enable the following option:

trackingOptions.IncludeResponseBody = true;

We started seeing the following error stack trace:

Dapr.DaprApiException: error invoke actor method: error finding address for actor type <actor_type> with id <actor_id>
Dapr.DaprApiException:
   at Dapr.Actors.DaprHttpInteractor+<SendAsyncHandleUnsuccessfulResponse>d__19.MoveNext (Dapr.Actors, Version=1.0.0.0, Culture=neutral, PublicKeyToken=a686d5d635c0f0e6)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Dapr.Actors.DaprHttpInteractor+<SendAsync>d__16.MoveNext (Dapr.Actors, Version=1.0.0.0, Culture=neutral, PublicKeyToken=a686d5d635c0f0e6)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Dapr.Actors.DaprHttpInteractor+<InvokeActorMethodWithRemotingAsync>d__10.MoveNext (Dapr.Actors, Version=1.0.0.0, Culture=neutral, PublicKeyToken=a686d5d635c0f0e6)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Dapr.Actors.Communication.Client.ActorRemotingClient+<InvokeAsync>d__5.MoveNext (Dapr.Actors, Version=1.0.0.0, Culture=neutral, PublicKeyToken=a686d5d635c0f0e6)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Dapr.Actors.Client.ActorProxy+<InvokeMethodAsync>d__32.MoveNext (Dapr.Actors, Version=1.0.0.0, Culture=neutral, PublicKeyToken=a686d5d635c0f0e6)
   at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)~
 at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Private.CoreLib, Version=4.0.0.0, Culture=neutral, PublicKeyToken=7cec85d7bea7798e)
   at Arcus.WebApi.Logging.ExceptionHandlingMiddleware+<Invoke>d__5.MoveNext (Arcus.WebApi.Logging, Version=1.4.0.0, Culture=neutral, PublicKeyToken=null)

Expected behavior
Response bodies should be tracked.

Additional context

• Version: 2.3.0 and 2.2.2

[tomkerkhove]

I can confirm this is related to Arcus and not Dapr.

Well, Dapr is having issues because of our response logging somehow. (stream not handled correctly?)

[stijnmoreels]

Some integration test we can set up to reproduce this, then?

[stijnmoreels]

Something for the web API, I'll move this.

[tomkerkhove]

We can, but it will not be easy as we'll need to integrate Dapr in our tests - https://docs.dapr.io/developing-applications/sdks/dotnet/dotnet-actors/dotnet-actors-usage/

[stijnmoreels]

We can, but it will not be easy as we'll need to integrate Dapr in our tests - https://docs.dapr.io/developing-applications/sdks/dotnet/dotnet-actors/dotnet-actors-usage/

Ok, we can try. @papabytes , could you give more info on the used middleware and the order in which they are registered? Also, is this something you want to investigate yourself?