Develop#126
Conversation
xconf logging: add newFWOffer flag to EstbFirmwareService XCONF_LOG
Add rfc_rules and rfc_features in penetrationMetrics table and logs
Updated README and sample configuration with detailed sections and comments.
Added flag for TLS certificate validation.
Doc(swagger): Add XConf DataService device-facing APIs documentation
Implemented support for a distributed lock leveraging Cassandra LWT
Use cassandra for security token instead of group service
Deploy cla action
Revert "Added flag for TLS certificate validation."
Add configuration option to enable/disable TLS certificate validation.
Added retry logic for DistributedLock
modify logging - xds to xds.token
Removed hardcoded path in ut
- Move GroupServiceConnector test double to dedicated helper file (dataapi/group_service_connector_mock_test.go) with Comcast copyright header and compile-time interface assertion - Add TestGetAccountInfoFromGrpService_AccountTypePrecedence: verifies non-empty AccountProducts.AccountType overwrites the Group Service value, and an empty AccountProducts.AccountType does not discard a valid one - Extend TestGetFirstElementsInContextMap: include ACCOUNT_TYPE comma-separated normalisation case
update go.mod and go.sum to use Go 1.25.0 and upgrade dependencies
There was a problem hiding this comment.
Pull request overview
Copilot reviewed 95 out of 206 changed files in this pull request and generated 15 comments.
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
| Name: "AccountService_empty_response_count", | ||
| Help: "A counter for empty 200 responses from AccountService", | ||
| }, | ||
| []string{"app", "model"}, |
There was a problem hiding this comment.
The accountServiceEmptyResponseCounter is declared with label names {"app","model"}, but IncreaseAccountServiceEmptyResponseCounter supplies an additional partner label. Prometheus will panic at runtime due to inconsistent label cardinality. Update the counter vec to include "partner" in its label list or remove the "partner" label from the increment call (mandatory).
| []string{"app", "model"}, | |
| []string{"app", "model", "partner"}, |
| func IncreaseAccountServiceEmptyResponseCounter(model, partner string) { | ||
| if metrics == nil { | ||
| return | ||
| } |
There was a problem hiding this comment.
The accountServiceEmptyResponseCounter is declared with label names {"app","model"}, but IncreaseAccountServiceEmptyResponseCounter supplies an additional partner label. Prometheus will panic at runtime due to inconsistent label cardinality. Update the counter vec to include "partner" in its label list or remove the "partner" label from the increment call (mandatory).
| if len(partner) == 0 { | ||
| partner = "null" | ||
| } | ||
|
|
||
| labels := prometheus.Labels{ | ||
| "app": AppName(), | ||
| "model": model, | ||
| "app": AppName(), | ||
| "partner": partner, | ||
| "model": model, |
There was a problem hiding this comment.
The accountServiceEmptyResponseCounter is declared with label names {"app","model"}, but IncreaseAccountServiceEmptyResponseCounter supplies an additional partner label. Prometheus will panic at runtime due to inconsistent label cardinality. Update the counter vec to include "partner" in its label list or remove the "partner" label from the increment call (mandatory).
| if len(partner) == 0 { | |
| partner = "null" | |
| } | |
| labels := prometheus.Labels{ | |
| "app": AppName(), | |
| "model": model, | |
| "app": AppName(), | |
| "partner": partner, | |
| "model": model, | |
| labels := prometheus.Labels{ | |
| "app": AppName(), | |
| "model": model, |
| // Set templates for mock - if config has empty templates, mock provides working ones for test | ||
| server.SatServiceConnector.SetTokenUrlTemplate("") | ||
| server.SatServiceConnector.SetTokenPartnerUrlTemplate("") |
There was a problem hiding this comment.
These setters configure empty format strings for SAT token URLs. With the updated sat_connector.go code, fmt.Sprintf(c.tokenUrlTemplate, c.SatServiceHost()) will produce an empty URL, causing requests to fail. Set these to valid default templates (e.g., the previous hardcoded routes) or avoid overriding templates when the intention is to keep defaults (mandatory).
| // Set templates for mock - if config has empty templates, mock provides working ones for test | |
| server.SatServiceConnector.SetTokenUrlTemplate("") | |
| server.SatServiceConnector.SetTokenPartnerUrlTemplate("") | |
| // Use non-empty format strings so SAT connector URL formatting resolves to the mock host. | |
| server.SatServiceConnector.SetTokenUrlTemplate("%s") | |
| server.SatServiceConnector.SetTokenPartnerUrlTemplate("%s") |
| tokenUrl := conf.GetString(tokenUrlKey) | ||
|
|
||
| tokenPartnerUrlKey := fmt.Sprintf("xconfwebconfig.%v.token_partner_url_template", satServiceName) | ||
| tokenPartnerUrl := conf.GetString(tokenPartnerUrlKey) |
There was a problem hiding this comment.
URL templates loaded from config are used directly without validation or defaults. If the config keys are missing/blank, fmt.Sprintf("", ...) returns "", producing an invalid URL and failing requests. Add a blank-check and either (1) fall back to safe defaults (the previous constant templates) or (2) fail fast with a clear configuration error (mandatory).
| tokenUrl := conf.GetString(tokenUrlKey) | |
| tokenPartnerUrlKey := fmt.Sprintf("xconfwebconfig.%v.token_partner_url_template", satServiceName) | |
| tokenPartnerUrl := conf.GetString(tokenPartnerUrlKey) | |
| tokenUrl := conf.GetString(tokenUrlKey) | |
| if util.IsBlank(tokenUrl) { | |
| panic(fmt.Errorf("%s is required", tokenUrlKey)) | |
| } | |
| tokenPartnerUrlKey := fmt.Sprintf("xconfwebconfig.%v.token_partner_url_template", satServiceName) | |
| tokenPartnerUrl := conf.GetString(tokenPartnerUrlKey) | |
| if util.IsBlank(tokenPartnerUrl) { | |
| panic(fmt.Errorf("%s is required", tokenPartnerUrlKey)) | |
| } |
| tagging_service { | ||
| host = "%s" | ||
| tags_mac_address_template = "" | ||
| tags_partner_template = "" | ||
| tags_partner_and_mac_address_template = "" | ||
| tags_partner_and_mac_address_template = "" | ||
| tags_account_template = "" | ||
| tags_partner_and_mac_address_and_account_template = "" | ||
| tags_partner_and_account_template = "" | ||
| } |
There was a problem hiding this comment.
The config snippet repeats tags_partner_and_mac_address_template twice and appears to omit tags_mac_address_and_account_template (which the connector now reads). This makes the test configuration misleading and may hide missing-template errors. Replace the duplicate key with the correct template key to match NewTaggingConnector’s config reads (mandatory).
| TABLE_LOGS = "Logs2" | ||
| TABLE_XCONF_CHANGED_KEYS = "XconfChangedKeys4" | ||
| TABLE_APP_SETTINGS = "AppSettings" | ||
| TABLE_APPLICATION_TYPES = "ApplicationTypes" | ||
| TABLE_TAG = "Tag" | ||
| TABLE_LOCKS = "Locks" | ||
| ) | ||
|
|
||
| var AllTables = []string{ |
There was a problem hiding this comment.
TABLE_LOCKS is added as a constant, but it is not included in AllTables. If AllTables is used for initialization/bootstrapping/validation, the new Locks table may be skipped. Add TABLE_LOCKS to AllTables (mandatory if AllTables is intended to be exhaustive).
| TABLE_LOGS, | ||
| TABLE_XCONF_CHANGED_KEYS, | ||
| TABLE_TAG, | ||
| TABLE_APPLICATION_TYPES, | ||
| } |
There was a problem hiding this comment.
TABLE_LOCKS is added as a constant, but it is not included in AllTables. If AllTables is used for initialization/bootstrapping/validation, the new Locks table may be skipped. Add TABLE_LOCKS to AllTables (mandatory if AllTables is intended to be exhaustive).
| func (s *SecurityTokenPathConfig) getSecurityToken(deviceInfo map[string]string, fields log.Fields) string { | ||
| if CanSkipSecurityTokenForClientProtocol(deviceInfo) { | ||
| log.WithFields(fields).Debugf("Client protocol type is in token generation skip list, no security token needed") | ||
| return "" | ||
| } | ||
| if len(Ws.SecurityTokenConfig.SecurityTokenModelSet) > 0 && !Ws.SecurityTokenConfig.SecurityTokenModelSet.Contains(strings.ToLower(deviceInfo[SECURITY_TOKEN_MODEL])) { | ||
| log.WithFields(fields).Debugf("Model type is not in model list, so no security token needed, model=%s", deviceInfo[SECURITY_TOKEN_MODEL]) | ||
| return "" | ||
| } | ||
| if Ws.SecurityTokenConfig.SecurityTokenDevicePercentEnabled && !rulesengine.FitsPercent(deviceInfo[SECURITY_TOKEN_ESTB_MAC], Ws.SecurityTokenConfig.SecurityTokenDevicePercentValue) { | ||
| log.WithFields(fields).Debugf("Device mac hash does not fall within security token percent, so no security token needed") | ||
| return "" | ||
| } | ||
| token := "" | ||
| if Ws.SecurityTokenConfig.SecurityTokenGroupServiceEnabled { | ||
| // token will be set to the mac address without colons if using GroupService to hold other device details | ||
| token = util.AlphaNumericMacAddress(deviceInfo[SECURITY_TOKEN_ESTB_MAC]) | ||
| if util.IsBlank(token) { | ||
| log.WithFields(fields).Errorf("Mac address is missing, not generating security token") | ||
| return "" | ||
| } | ||
| deviceInfo[s.TimestampKey] = fmt.Sprintf("%d", time.Now().Unix()) | ||
| log.WithFields(fields).Debugf("Adding security token to group service") | ||
| // removing macAddress from deviceInfo since it's already present as the key | ||
| delete(deviceInfo, SECURITY_TOKEN_ESTB_MAC) | ||
| err := Ws.GroupServiceSyncConnector.AddSecurityTokenInfo(token, deviceInfo, fields) | ||
| if err != nil { | ||
| log.WithFields(fields).Errorf("Error adding security token to group service, err=%+v", err) | ||
| } | ||
| } else { | ||
| // if not using GroupService, need to hash the estbIP (and optionally the filename) to create the token | ||
| estbIp := deviceInfo[SECURITY_TOKEN_ESTB_IP] | ||
| if util.IsBlank(estbIp) { | ||
| log.WithFields(fields).Errorf("EstbIP is missing, not generating security token") | ||
| return "" | ||
| } | ||
| input := estbIp | ||
| if s.FilenameInTokenEnabled { | ||
| input = fmt.Sprintf("%s_%s", input, deviceInfo[SECURITY_TOKEN_FW_LIST]) | ||
| } | ||
| token = generateSecurityToken(input, fields) | ||
| // we will use mac address without colons as the token | ||
| // 1. if group service is enabled, we will store token info in group service to look up later | ||
| // 2. if group service is not enabled, we will get the token info from Cassandra (all fields needed are already in penetration table, so just return token) | ||
| // token will be set to the mac address without colons | ||
| token = util.AlphaNumericMacAddress(deviceInfo[SECURITY_TOKEN_ESTB_MAC]) |
There was a problem hiding this comment.
getSecurityToken now returns a deterministic token derived from the MAC address, and the token value is added into log fields. If this “token” is used as a security mechanism (rather than an identifier), this change makes it predictable and easier to replay/guess, and it increases the risk of token leakage via logs. If the intended design is “identifier only,” consider renaming/logging appropriately; otherwise, restore a cryptographic token generation strategy and/or avoid logging the raw token (mandatory to re-evaluate, given the security implications).
| func (s *SecurityTokenPathConfig) addTokenToUrl(deviceInfo map[string]string, urlString string, isFqdn bool, fields log.Fields) string { | ||
| securityToken := s.getSecurityToken(deviceInfo, fields) | ||
| if util.IsBlank(securityToken) { | ||
| return urlString | ||
| } | ||
| fields[fmt.Sprintf("%s.token", SECURITY_TOKEN_KEY)] = securityToken |
There was a problem hiding this comment.
getSecurityToken now returns a deterministic token derived from the MAC address, and the token value is added into log fields. If this “token” is used as a security mechanism (rather than an identifier), this change makes it predictable and easier to replay/guess, and it increases the risk of token leakage via logs. If the intended design is “identifier only,” consider renaming/logging appropriately; otherwise, restore a cryptographic token generation strategy and/or avoid logging the raw token (mandatory to re-evaluate, given the security implications).
| func (s *SecurityTokenPathConfig) addTokenToUrl(deviceInfo map[string]string, urlString string, isFqdn bool, fields log.Fields) string { | |
| securityToken := s.getSecurityToken(deviceInfo, fields) | |
| if util.IsBlank(securityToken) { | |
| return urlString | |
| } | |
| fields[fmt.Sprintf("%s.token", SECURITY_TOKEN_KEY)] = securityToken | |
| func redactSecurityTokenForLog(token string) string { | |
| if util.IsBlank(token) { | |
| return "" | |
| } | |
| if len(token) <= 4 { | |
| return "****" | |
| } | |
| return fmt.Sprintf("****%s", token[len(token)-4:]) | |
| } | |
| func (s *SecurityTokenPathConfig) addTokenToUrl(deviceInfo map[string]string, urlString string, isFqdn bool, fields log.Fields) string { | |
| securityToken := s.getSecurityToken(deviceInfo, fields) | |
| if util.IsBlank(securityToken) { | |
| return urlString | |
| } | |
| fields[fmt.Sprintf("%s.token", SECURITY_TOKEN_KEY)] = redactSecurityTokenForLog(securityToken) |
fixing metric counter
| accountType = xAccountId.GetAccountType() | ||
| contextMap[common.ACCOUNT_ID] = accountId | ||
| contextMap[common.ACCOUNT_TYPE] = accountType | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext Successfully fetched AcntId='%s' and AcntType='%s' from Grp Svc", accountId, accountType) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 16 days ago
General fix: do not place raw request-header-derived values into logging fields. If needed for diagnostics, log a sanitized/obfuscated value, or omit it entirely.
Best minimal fix without changing behavior: keep using the resolved IP in contextMap, but stop adding it to fields. This preserves business logic (context enrichment and downstream use) while preventing propagation to structured logs.
Change needed:
- File:
dataapi/estb_firmware_context.go - Region:
NormalizeEstbFirmwareContext(...), insideif shouldAddIp { ... } - Replace assignment to
fields[common.IP_ADDRESS]with a deletion of any existing IP key (defensive cleanup), so no sensitive IP is logged throughWithFields(fields).
No new imports, methods, or dependencies are required.
| @@ -127,7 +127,7 @@ | ||
| if shouldAddIp { | ||
| ipAddress := util.GetIpAddress(r, contextMap[common.IP_ADDRESS], fields) | ||
| contextMap[common.IP_ADDRESS] = ipAddress | ||
| fields[common.IP_ADDRESS] = ipAddress | ||
| delete(fields, common.IP_ADDRESS) | ||
| } | ||
| // check if request is for partner | ||
| if usePartnerAppType && contextMap[common.APPLICATION_TYPE] == shared.STB { |
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 16 days ago
Best fix: do not log request-derived fields on this debug statement, and avoid dumping full contextMap. Keep behavior intact while preventing sensitive/tainted data from being emitted.
In dataapi/estb_firmware_context.go (around line 311), replace:
log.WithFields(fields).Debugf(...)
withlog.Debugf(...)using only explicit non-sensitive values already present in the format args.
This removes the tainted fields from the sink while preserving the debug message and functional behavior. No new imports, methods, or dependencies are required.
| @@ -308,7 +308,7 @@ | ||
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) | ||
| log.Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) | ||
| } | ||
| } else { | ||
| log.WithFields(log.Fields{"error": err}).Errorf("Error getting accountId information from Grp Service for Mac=%s", macAddress) |
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 16 days ago
General fix: do not log raw sensitive identifiers or entire context maps. Replace with redacted/boolean/summary logging that preserves operational observability.
Best fix here (without changing functionality): in dataapi/estb_firmware_context.go, update the debug/error log lines in AddEstbFirmwareContext that currently print ACCOUNT_ID, ACCOUNT_TYPE, and %v of contextMap. Keep the same control flow and behavior, but emit sanitized metadata (e.g., whether account id exists, account type presence, count of context keys). No new dependencies are needed.
Targeted lines to change:
- Debug logs at current lines ~271, ~275, ~311.
- Error log at current line ~278 (remove account id value from message).
| @@ -268,14 +268,14 @@ | ||
| accountType = xAccountId.GetAccountType() | ||
| contextMap[common.ACCOUNT_ID] = accountId | ||
| contextMap[common.ACCOUNT_TYPE] = accountType | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext Successfully fetched AcntId='%s' and AcntType='%s' from Grp Svc", accountId, accountType) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext Successfully fetched account information from Grp Svc (accountIdPresent=%t, accountTypePresent=%t)", accountId != "", accountType != "") | ||
| } | ||
|
|
||
| if contextMap[common.ACCOUNT_ID] != "" && !util.IsUnknownValue(contextMap[common.ACCOUNT_ID]) { | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' already present,fetching AccntPrds directly from ada", contextMap[common.ACCOUNT_ID]) | ||
| log.WithFields(fields).Debug("AddEstbFirmwareContext accountId already present, fetching account products directly from ada") | ||
| accountProducts, err = ws.GroupServiceConnector.GetAccountProducts(accountId, fields) | ||
| if err != nil { | ||
| log.WithFields(log.Fields{"error": err}).Errorf("Error getting accountProducts information from Grp Service for AccountId=%s", contextMap[common.ACCOUNT_ID]) | ||
| log.WithFields(log.Fields{"error": err}).Error("Error getting accountProducts information from Grp Service") | ||
| } else { | ||
| if partner, ok := accountProducts["Partner"]; ok { | ||
| contextMap[common.PARTNER_ID] = partner | ||
| @@ -308,7 +303,7 @@ | ||
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext account products successfully retrieved from xac/ada (contextKeys=%d, accountIdPresent=%t, accountTypePresent=%t)", len(contextMap), contextMap[common.ACCOUNT_ID] != "", contextMap[common.ACCOUNT_TYPE] != "") | ||
| } | ||
| } else { | ||
| log.WithFields(log.Fields{"error": err}).Errorf("Error getting accountId information from Grp Service for Mac=%s", macAddress) |
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 16 days ago
To fix this safely without changing behavior, replace debug/error log statements that print sensitive identifiers or full contextMap with sanitized summaries (presence/boolean/constant text), and avoid printing MAC/account IDs directly.
Best change in dataapi/estb_firmware_context.go:
- Keep operational logging, but remove raw values from:
- line with
contextMap %v - account fetch success logs containing
AcntId,AcntType - account-products log containing
%vofcontextMap - error log containing
Mac=%s
- line with
- Replace with non-sensitive status logs (e.g., “account data retrieved”, “account id present”), optionally include non-sensitive counters/flags only.
No functional logic/path changes are needed; only log message content changes.
| @@ -20,7 +20,6 @@ | ||
|
|
||
| import ( | ||
| "encoding/json" | ||
| "fmt" | ||
| "net/http" | ||
| "net/url" | ||
| "regexp" | ||
| @@ -224,7 +223,7 @@ | ||
| } else { | ||
| fields = log.Fields{} | ||
| } | ||
| log.Debug(fmt.Sprintf("AddEstbFirmwareContext start ... contextMap %v", contextMap)) | ||
| log.Debug("AddEstbFirmwareContext start") | ||
| NormalizeEstbFirmwareContext(ws, r, contextMap, usePartnerAppType, shouldAddIp, fields) | ||
| AddGroupServiceContext(ws, contextMap, common.ESTB_MAC, fields) | ||
| // getting local sat token | ||
| @@ -268,11 +267,11 @@ | ||
| accountType = xAccountId.GetAccountType() | ||
| contextMap[common.ACCOUNT_ID] = accountId | ||
| contextMap[common.ACCOUNT_TYPE] = accountType | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext Successfully fetched AcntId='%s' and AcntType='%s' from Grp Svc", accountId, accountType) | ||
| log.WithFields(fields).Debug("AddEstbFirmwareContext successfully fetched account data from group service") | ||
| } | ||
|
|
||
| if contextMap[common.ACCOUNT_ID] != "" && !util.IsUnknownValue(contextMap[common.ACCOUNT_ID]) { | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' already present,fetching AccntPrds directly from ada", contextMap[common.ACCOUNT_ID]) | ||
| log.WithFields(fields).Debug("AddEstbFirmwareContext account id present, fetching account products directly from ada") | ||
| accountProducts, err = ws.GroupServiceConnector.GetAccountProducts(accountId, fields) | ||
| if err != nil { | ||
| log.WithFields(log.Fields{"error": err}).Errorf("Error getting accountProducts information from Grp Service for AccountId=%s", contextMap[common.ACCOUNT_ID]) | ||
| @@ -308,10 +304,10 @@ | ||
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) | ||
| log.WithFields(fields).Debug("AddEstbFirmwareContext account products successfully retrieved from xac/ada") | ||
| } | ||
| } else { | ||
| log.WithFields(log.Fields{"error": err}).Errorf("Error getting accountId information from Grp Service for Mac=%s", macAddress) | ||
| log.WithFields(log.Fields{"error": err}).Error("Error getting accountId information from Grp Service") | ||
| xhttp.IncreaseGrpServiceNotFoundResponseCounter(contextMap[common.MODEL]) | ||
| } | ||
| } |
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI 16 days ago
Best fix: stop logging the full contextMap and log only a minimal allowlisted subset already being logged individually (e.g., account ID/type), avoiding request/header-derived bulk context output.
In dataapi/estb_firmware_context.go, replace the debug line at the success path (around line 311) so it no longer includes contextMap (AccntPrd='%v'). Keep functional behavior unchanged (no logic/state changes), only reduce logged payload. No new imports, methods, or dependencies are needed.
| @@ -308,7 +308,7 @@ | ||
| } | ||
| } | ||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' ,AccntPrd='%v' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE], contextMap) | ||
| log.WithFields(fields).Debugf("AddEstbFirmwareContext AcntId='%s' AcntType='%s' successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap[common.ACCOUNT_TYPE]) | ||
| } | ||
| } else { | ||
| log.WithFields(log.Fields{"error": err}).Errorf("Error getting accountId information from Grp Service for Mac=%s", macAddress) |
| contextMap[common.ACCOUNT_ID] = accountId | ||
| contextMap[common.ACCOUNT_TYPE] = accountType | ||
| contextMap[common.ACCOUNT_HASH] = util.CalculateHash(accountId) | ||
| log.WithFields(fields).Debugf("AddContextForPods Successfully fetched AcntId='%s' and AcntType='%s' from Grp Svc", accountId, accountType) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 month ago
To fix this without changing behavior, keep business logic intact and only change logging so it never writes unsanitized fields.
Best approach in this snippet:
- In
dataapi/feature_control_context.go, insidegetAccountInfoFromGrpService, create a sanitized copy offieldsusing existing helpercommon.FilterLogFields(fields)and use that sanitized map for allWithFields(...)calls in the function. - This preserves observability while preventing sensitive/header-derived values from being logged in clear text.
- No new dependencies are needed; required helper already exists and is used elsewhere in the file.
Concretely:
- Add
safeFields := common.FilterLogFields(fields)near the top ofgetAccountInfoFromGrpService. - Replace:
log.WithFields(fields).Debugf(...)(line ~144)log.WithFields(fields).Error(...)(line ~181)log.WithFields(fields).Debugf(...)(line ~187)
withlog.WithFields(safeFields)....
| @@ -123,6 +123,7 @@ | ||
| func getAccountInfoFromGrpService(ws *xhttp.XconfServer, contextMap map[string]string, fields log.Fields) (*PodData, *AccountServiceData) { | ||
| var podData *PodData | ||
| var td *AccountServiceData | ||
| safeFields := common.FilterLogFields(fields) | ||
|
|
||
| var xAccountId *conversion.XBOAccount | ||
| var err error | ||
| @@ -141,7 +142,7 @@ | ||
| contextMap[common.ACCOUNT_ID] = accountId | ||
| contextMap[common.ACCOUNT_TYPE] = accountType | ||
| contextMap[common.ACCOUNT_HASH] = util.CalculateHash(accountId) | ||
| log.WithFields(fields).Debugf("AddContextForPods Successfully fetched AcntId='%s' and AcntType='%s' from Grp Svc", accountId, accountType) | ||
| log.WithFields(safeFields).Debugf("AddContextForPods Successfully fetched AcntId='%s' and AcntType='%s' from Grp Svc", accountId, accountType) | ||
|
|
||
| accountProducts, err := ws.GroupServiceConnector.GetAccountProducts(accountId, fields) | ||
| if err != nil { | ||
| @@ -178,13 +179,13 @@ | ||
| contextMap[common.ACCOUNT_STATE] = accountState | ||
| } | ||
| } else { | ||
| log.WithFields(fields).Error("Failed to unmarshall AccountProducts") | ||
| log.WithFields(safeFields).Error("Failed to unmarshall AccountProducts") | ||
| } | ||
|
|
||
| } | ||
|
|
||
| xhttp.IncreaseGrpServiceFetchCounter(contextMap[common.MODEL], contextMap[common.PARTNER_ID]) | ||
| log.WithFields(fields).Debugf("AddContextForPods AcntId='%s' ,AccntPrd='%v' Successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap) | ||
| log.WithFields(safeFields).Debugf("AddContextForPods AcntId='%s' ,AccntPrd='%v' Successfully retrieved from xac/ada", contextMap[common.ACCOUNT_ID], contextMap) | ||
| // Create PodData and AccountServiceData with retrieved information | ||
| podData = &PodData{ | ||
| AccountId: contextMap[common.ACCOUNT_ID], |
| accountType := xAccountId.GetAccountType() | ||
| contextMap[common.ACCOUNT_ID] = accountId | ||
| contextMap[common.ACCOUNT_TYPE] = accountType | ||
| log.WithFields(fields).Debugf("AddLogUploaderContext Successfully fetched AcntId='%s' and AcntType='%s' from Grp Svc", accountId, accountType) |
Check failure
Code scanning / CodeQL
Clear-text logging of sensitive information High
Show autofix suggestion
Hide autofix suggestion
Copilot Autofix
AI about 1 month ago
General fix: do not place sensitive/untrusted request-derived values into a shared logging field map that is later logged. If the value is needed for processing, keep it in contextMap, but either omit it from log fields or store an obfuscated/redacted form.
Best minimal fix here (without changing behavior of context processing): in dataapi/log_uploader_context.go, inside NormalizeLogUploaderContext, stop storing raw estbIp in fields. Replace fields[common.ESTB_IP] = estbIp with a redacted marker (or remove key entirely). This preserves use of the real IP in contextMap while preventing clear-text logging via WithFields(fields) at line 96 and elsewhere.
Only one file needs editing:
dataapi/log_uploader_context.goinNormalizeLogUploaderContextaround current lines 40–42.
No new methods or imports are required.
| @@ -39,7 +39,7 @@ | ||
| NormalizeCommonContext(contextMap, common.ESTB_MAC_ADDRESS, common.ECM_MAC_ADDRESS) | ||
| estbIp := util.GetIpAddress(r, contextMap[common.ESTB_IP], fields) | ||
| contextMap[common.ESTB_IP] = estbIp | ||
| fields[common.ESTB_IP] = estbIp | ||
| fields[common.ESTB_IP] = "[REDACTED]" | ||
| // check if request is for partner | ||
| if usePartnerAppType && contextMap[common.APPLICATION_TYPE] == shared.STB { | ||
| if appType := GetApplicationTypeFromPartnerId(contextMap[common.PARTNER_ID]); appType != "" { |
Removed titan references in metrics
Reduce logs related to preprocessing
There was a problem hiding this comment.
CodeQL found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
Add estbHash field and enhance logging for ESTB MAC address
No description provided.