Module: Scope3 Real-Time Data

[bokelley]

Summary

This PR adds a new Scope3 Real-Time Data (RTD) module that integrates Scope3's audience segmentation API with Prebid Server to provide enhanced targeting capabilities.

Features

• Fetches real-time audience segments from Scope3 API during auction process
• Integrates with LiveRamp ATS for enhanced user identification (both sidecar and envelope support)
• Adds targeting data to auction response for GAM integration
• Thread-safe segment storage using module context
• Configurable timeout and endpoint settings
• Graceful error handling that doesn't fail auctions

Implementation Details

The module implements three hook stages:

• Entrypoint: Initialize module context with segment storage
• Raw Auction Request: Fetch segments from Scope3 API using bid request data
• Auction Response: Add segments to response for publisher control

LiveRamp Integration

Supports both LiveRamp ATS deployment scenarios:

• With Sidecar: Uses decrypted RampID when available
• Without Sidecar: Forwards encrypted ATS envelope for server-side decryption

Configuration

hooks:
  enabled: true
  modules:
    scope3:
      rtd:
        enabled: true
        auth_key: ${SCOPE3_API_KEY}
        timeout_ms: 1000
        cache_ttl_seconds: 60
        add_to_targeting: false
  host_execution_plan:
    endpoints:
      /openrtb2/auction:
        stages:
          entrypoint:
            groups:
              - timeout: 5
                hook_sequence:
                  - module_code: "scope3.rtd"
                    hook_impl_code: "HandleEntrypointHook"
          raw_auction_request:
            groups:
              - timeout: 2000
                hook_sequence:
                  - module_code: "scope3.rtd"
                    hook_impl_code: "HandleRawAuctionHook"
          auction_response:
            groups:
              - timeout: 5
                hook_sequence:
                  - module_code: "scope3.rtd"
                    hook_impl_code: "HandleAuctionResponseHook"

Testing

• Module has been tested with live Scope3 API integration
• Successfully retrieves and processes audience segments
• Confirmed targeting data is properly added to auction response
• Verified compatibility with existing bidder integrations
• Test Coverage: Module includes comprehensive unit tests with >80% coverage

Documentation

Documentation has been submitted to prebid.github.io: prebid/prebid.github.io#6217

This includes comprehensive documentation for both Prebid.js and Prebid Server Scope3 RTD modules.

Files Changed

• modules/scope3/rtd/module.go - Core RTD module implementation
• modules/scope3/rtd/module_test.go - Unit tests with >80% coverage
• modules/scope3/rtd/README.md - Documentation and configuration examples

🤖 Generated with Claude Code

[patmmccann]

as a potential module user, i would strongly prefer these also came back in bid.meta so i could decie if i want to send them togam or somewhere else and also how i format them if i do send them to gam. This could always happen in addition or be some sort of toggle

[patmmccann]

eg

prebid-server/adapters/teads/teads.go

Line 161 in 6acbc9f

RendererName: bidExtTeads.Prebid.Meta.RendererName,

[bokelley]

right but since we're not bidding that probably isn't the right spot? I changed it so by default it goes to response.ext.scope3.segments but if you add add_to_targeting: true it will add to the targeting list for GAM. does that make sense?

[patmmccann]

I think that location works, and you raise a good point, we need your result back even on a non-bid event. @bsardo any thoughts on the best place?

[bokelley]

@gravelg so this should be /prebid/rtii?

[gravelg]

Yes I've made that endpoint

[VeronikaSolovei9]

Local testing looks good, added a couple of minor comments. Mostly related to the code clean up

[VeronikaSolovei9]

Please do not ignore error here.

[gravelg]

Fixed this

[VeronikaSolovei9]

Nitpick: use = instead of :=. err variable already exists in this scope. Same for ok in lines 344-348, 186

[gravelg]

Fixed all the ones that were being reused

[VeronikaSolovei9]

Would it be reasonable to unmarshal bidRequest.User.Ext to a custom structure, like

 type UserExt struct {
    eids *[]openrtb2.EID
    rampId string
    live ramp_idl string
}

[gravelg]

Unmarshalling in a struct now

[VeronikaSolovei9]

Same comment about structure for user.ext

[gravelg]

Unmarshalling in a struct here also

[gravelg]

@VeronikaSolovei9 I've taken over this PR from Brian. Addressed all your comments, let me know if there's anything else!

[VeronikaSolovei9]

In case there are no Eids in user.ext this line of code throws Null Pointer Exception, because in userExt Eids declared as a pointer:

type userExt struct {
	Eids           *[]openrtb2.EID `json:"eids"`
	...
}

Is there a reason why Eids is a pointer?
I modified it locally to "Eids []openrtb2.EID json:"eids"" and it now can gracefully handle empty list and skip the for loop without throwing the exception. .

Nitpick: var userExt userExt - please give a name to the variable different from the structure name, like var userExtension userExt.

[gravelg]

Totally an oversight! I've corrected the pointer and variable names. Let me know if there's anything else!

[scr-oath]

suggestion: Rather than making shallow copy of the eid while iterating as well as when adding to the filtered slice, use SlicePointerValues from iterutil https://github.com/prebid/prebid-server/blob/master/util/iterutil/slices.go#L17

Suggested change

	for _, eid := range eids {
	if allowed[eid.Source] {
	filtered = append(filtered, eid)
	}
	}
	for eid := range iterutil.SlicePointerValue(eids) {
	if allowed[eid.Source] {
	filtered = append(filtered, *eid)
	}
	}

[gravelg]

Done!

[scr-oath]

nitpick (not-blocking): use errors.New for errors with no formatting - this is truly a stylistic nitpick feel free to say no but just mentioning so my preference is known.

[gravelg]

Done!

[scr-oath]

question: Can you use one of the transports from deps?

In the Yahoo fork, we add open telemetry on calls, so… just wondering how one might configure that piece. I don't see much benefit in these configurations as they stand; could you live with deps.HTTPClient ?

[gravelg]

Most of these settings would be fine is we used the deps client, except the timeout which we set pretty low (1s).

[scr-oath]

I see, the deps doesn't have a timeout…

	generalHttpClient := &http.Client{
		Transport: &http.Transport{
			Proxy:               http.ProxyFromEnvironment,
			MaxConnsPerHost:     cfg.Client.MaxConnsPerHost,
			MaxIdleConns:        cfg.Client.MaxIdleConns,
			MaxIdleConnsPerHost: cfg.Client.MaxIdleConnsPerHost,
			IdleConnTimeout:     time.Duration(cfg.Client.IdleConnTimeout) * time.Second,
			TLSClientConfig:     &tls.Config{RootCAs: certPool},
		},
	}

and from client.go

	// Timeout specifies a time limit for requests made by this
	// Client. The timeout includes connection time, any
	// redirects, and reading the response body. The timer remains
	// running after Get, Head, Post, or Do return and will
	// interrupt reading of the Response.Body.
	//
	// A Timeout of zero means no timeout.
	//
	// The Client cancels requests to the underlying Transport
	// as if the Request's Context ended.
	//
	// For compatibility, the Client will also use the deprecated
	// CancelRequest method on Transport if found. New
	// RoundTripper implementations should use the Request's Context
	// for cancellation instead of implementing CancelRequest.
	Timeout time.Duration

[scr-oath]

Perhaps you could add a Timeout configuration to the deps.HttpClient as it doesn't appear to have one currently? You could leave the default empty/unset/0 and allow us to set it - if we have tmax of 750ms then 1s is plenty long enough to cover both calls to adapaters, modules, storedRequests, and maybe even vendorListFetcher (not sure on this one but hopefully it can get the answer <1s).

[scr-oath]

question: I do see that prebid has https://github.com/coocood/freecache/blob/master/cache.go as a dependency - would this work just as well as implementing your own or do you have custom requirements?

[gravelg]

No reason, I replaced to use freecache!

[scr-oath]

nitpick (not-blocking): For better or worse, I usually create a var ret hookstage.HookResult[hookstage.AuctionResponsePayload] and then, above, can merely use its changeset to AddMutation rather than needing to construct it and then copy into this object for return.

[gravelg]

Done

[scr-oath]

Could you use jsoniter here?

Suggested change

	if err = json.NewDecoder(resp.Body).Decode(&scope3Resp); err != nil {
	if err = jsoniter.ConfigCompatibleWithStandardLibrary.NewDecoder(resp.Body).Decode(&scope3Resp); err != nil {

[gravelg]

Done!

[scr-oath]

here again, try not to shallow-copy the imps; as a struct, they're fairly large; use the iterutil.SlicePointerValues

[gravelg]

Done!

[scr-oath]

suggestion: Set returns an err, consider logging or analytics for this error.

[scr-oath]

Just an idea - go added sequences a while back and maps + slices utilities. Would something like this be slightly more succinct as a pattern?

Suggested change

	// Convert to slice
	segments := make([]string, 0, len(segmentMap))
	for segment := range segmentMap {
	segments = append(segments, segment)
	}
	// Convert to slice
	segments := slices.AppendSeq(make([]string, 0, len(segmentMap)), maps.Keys(segmentMap))

[scr-oath]

Why is the value type struct{} - could it be bool? setting to true would just look nicer than struct{}{}. I realize this is just for a "set" and all you want is the keys, and I'm sure this struct thing is on the web somewhere; just thinking bool works well too.

[scr-oath]

nitpick: While go doesn't require this, I do prefer declare first then use; can you move these to the top? Also consider having one type (...) block to avoid that slight redundancy of needing the type keyword over and over.

[scr-oath]

This seems like a lot of duplicate code - can you merely populate the missing things and use the same code on the resulting map whether new or existing?

Suggested change

	if prebidMap, ok := extMap["prebid"].(map[string]interface{}); ok {
	if targetingMap, ok := prebidMap["targeting"].(map[string]interface{}); ok {
	// Add each segment as individual targeting key
	for _, segment := range segments {
	targetingMap[segment] = "true"
	}
	} else {
	// Create targeting map with individual segment keys
	newTargeting := make(map[string]interface{})
	for _, segment := range segments {
	newTargeting[segment] = "true"
	}
	prebidMap["targeting"] = newTargeting
	}
	} else {
	// Create prebid map with targeting
	newTargeting := make(map[string]interface{})
	for _, segment := range segments {
	newTargeting[segment] = "true"
	}
	extMap["prebid"] = map[string]interface{}{
	"targeting": newTargeting,
	}
	}
	prebidMap, ok := extMap["prebid"].(map[string]interface{})
	if !ok {
	prebidMap = make(map[string]interface{})
	extMap["prebid"] = prebidMap
	}
	targetingMap, ok := prebidMap["targeting"].(map[string]interface{})
	if !ok {
	targetingMap = make(map[string]interface{})
	prebidMap["targeting"] = targetingMap
	}
	// Add each segment as individual targeting key
	for _, segment := range segments {
	targetingMap[segment] = "true"
	}

[gravelg]

@scr-oath all of the requested changes have been made! Any chance you can take another look?

[scr-oath]

Senior Golang Architect Review - Scope3 RTD Module

I've completed a comprehensive review focusing on your specific concerns and general Go best practices. Overall, this is well-architected code with good attention to safety and privacy, but there are a few items that should be addressed.

✅ EXCELLENT ASPECTS

Iterator Usage: Perfect use of iterutil.SlicePointerValues() throughout the codebase to avoid unnecessary struct copying during iteration.

HTTP Client Transport: Correctly preserves the transport from deps.HTTPClient.Transport, maintaining OpenTelemetry and other wrapping.

Context Propagation: Excellent implementation - context from the original HTTP request properly spans all hook stages via context.WithCancel(req.Context()).

Asynchronous Design: Good goroutine usage with proper channel signaling and resource cleanup in defer asyncRequest.Cancel().

Privacy/Security: Comprehensive masking implementation with appropriate defaults and fail-safe behavior.

⚠️ ITEMS TO ADDRESS

1. Goroutine Panic Recovery: Consider adding panic recovery to prevent crashes
2. Blocking Channel Read: The <-asyncRequest.Done could benefit from a timeout/context check
3. Minor Typo: Line 259 has "reqeuest" instead of "request"
4. Clarifying Comment: The struct copying in masking.go is intentional but could use a comment

🎯 RECOMMENDATIONS

• Priority: Add panic recovery to the async goroutine
• Consider: Timeout protection for the channel read operation
• Enhancement: Add clarifying comments where intentional design choices might be questioned

📊 ARCHITECTURE ASSESSMENT

• Safety: Excellent error handling that doesn't fail auctions
• Performance: Good use of caching and efficient iteration patterns
• Maintainability: Clear separation of concerns and comprehensive testing
• Documentation: All public functions properly documented

This module demonstrates solid Go engineering practices and thoughtful design for production ad-tech systems. The suggested changes are refinements rather than critical flaws.

🤖 Reviewed by Claude Code

[scr-oath]

Consider adding panic recovery to prevent goroutine panics from causing issues:

go func() {
    defer func() {
        if r := recover(); r != nil {
            ar.Err = fmt.Errorf("panic in async request: %v", r)
        }
        close(ar.Done)
    }()
    ar.Segments, ar.Err = ar.Module.fetchScope3Segments(ar.Context, request)
}()

🤖 Reviewed by Claude Code

[scr-oath]

Typo: "reqeuest" should be "request"

🤖 Reviewed by Claude Code

[scr-oath]

Consider adding a timeout to prevent indefinite blocking if the goroutine encounters issues:

select {
case <-asyncRequest.Done:
    // Continue with processing
case <-ctx.Done():
    return ret, nil // Context cancelled, exit gracefully
}

This ensures the auction doesn't hang if the async request gets stuck.

🤖 Reviewed by Claude Code

[scr-oath]

Consider adding a clarifying comment since this intentionally copies the struct to create a new filtered slice:

// Intentionally copy the EID struct to create a new filtered slice
// that's independent of the original data
filtered = append(filtered, *eid)

This makes it clear the copying is by design for data ownership in the masking context.

🤖 Reviewed by Claude Code

[scr-oath]

✅ Excellent! The HTTP client correctly uses the transport from deps, preserving any wrapping (OpenTelemetry, etc.). This addresses one of the key architectural requirements.

🤖 Reviewed by Claude Code