testconsole: 401 Unauthorised when attempting to retrieve automated test runs

[dave-sampson13]

Awesome task!

I'm using the testconsole to test my configuration & handlebars template. If I issue myself a PAT with full access, everything works fine. However, if I attempt to restrict the permissions in the PAT to those I would expect it to need, it only goes so far before encountering a 401 Unauthorised when attempting to retrieve details of automated test runs (there are none in my project, by the way), which then seemingly causes it to lose all the work items it had previously successfully retrieved, and consequently the markdown file produced is largely empty.

I tracked the issue down to

AzurePipelines/Extensions/XplatGenerateReleaseNotes/V3/ReleaseNotesFunctions.ts

Line 542 in 1ca0b2c

export async function getTestsForBuild(

, specifically

AzurePipelines/Extensions/XplatGenerateReleaseNotes/V3/ReleaseNotesFunctions.ts

Line 550 in 1ca0b2c

let buildTestResults = await (testAPI.getTestResultsByBuild(teamProject, buildId));

which uses testAPI.getTestResultsByBuild and hates me & the PAT I rode in on.

The doco is fairly limited on what access this extension method & the APIs it calls needs, so running a bit blind, but my work-around for now has been to comment out the majority of this function (because I don't have any automated tests anyway), and then everything works just fine. Ideally though I wouldn't need to do that, & instead it'd be good to:

1. Know exactly what permissions the PAT I use with the testconsole needs so I don't need to create a PAT with full access (indeed some Azure DevOps organisations I work with don't even allow the creation of full access PATs)
2. More gracefully handle this case of the 401, so that although it might result in no tests being retrieved, at least it wouldn't result in the whole object structure being empty, particularly the work items, not sure why this should impact them

Thanks for your help, let me know if you need any more information!

[dave-sampson13]

These are the permissions I've given my PAT:

• Build: Read
• Code: Read, Status
• Load Test: Read
• Packaging: Read
• Release: Read
• Test Management: Read
• Work Items: Read

[rfennell]

There is a very recent document about how Microsoft have added PAT scopes to the API and documented them. That said, it is not obvious which scope is missing. The first step is to find the actual API call being used by the SDK, or just use a process of elimination.

There is as you say the option to gracefully handle the 401, or a technique I have used before is to add a flag to disable the call to get the test data.

[dave-sampson13]

Thanks @rfennell . The Azure DevOps API docs don't appear to cover this particular API call (resultsByBuild) & also not sure which exact version of the API will be being used, but a general query of test results appears to require the write scope (which is bizarre in itself). I tried adding this scope however & still no dice. Indeed, I've tried adding every scope that can be added via the Azure DevOps UI, and even with every scope under the sun (but not a "full access" PAT), still get the error. So it would appear that there's some scope that cannot be added via the UI, that a "full access" PAT gets.

[github-actions]

This issue is stale because it has been open 30 days with no activity. Remove stale label or comment or this will be closed in 5 days

[Stan-Stani]

Thanks @rfennell . The Azure DevOps API docs don't appear to cover this particular API call (resultsByBuild) & also not sure which exact version of the API will be being used, but a general query of test results appears to require the write scope (which is bizarre in itself). I tried adding this scope however & still no dice. Indeed, I've tried adding every scope that can be added via the Azure DevOps UI, and even with every scope under the sun (but not a "full access" PAT), still get the error. So it would appear that there's some scope that cannot be added via the UI, that a "full access" PAT gets.

@dave-sampson13 Thank you for the work around.

Bumping as this is still an issue.

[rfennell]

I will move this to be a known issue, I fear we are still limited by what PATs are 'setable'