[Connectors] Refactor YETI to API v2 payload and auth schemas by sanjib2006 · Pull Request #3736 · intelowlproject/IntelOwl

sanjib2006 · 2026-06-02T16:56:57Z

Closes #3707

Description

Implemented YETI JWT authentication and access token handling.
Updated YETI API endpoints and request payload schemas to align with the current v2 API requirements.
Added unit tests covering :
- Authentication failure scenario.
- Missing access token in authentication response.
- Successful file-sample submission and processing flow.

Type of change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue).

Checklist

I have read and understood the rules about how to Contribute to this project
The pull request is for the branch gsoc-2026/connectors
I have inserted the copyright banner at the start of the file: # This file is a part of IntelOwl https://github.com/intelowlproject/IntelOwl # See the file 'LICENSE' for copying permission.
Linters (Ruff) gave 0 errors. If you have correctly installed pre-commit, it does these checks and adjustments on your behalf.
I have added tests for the feature/bug I solved (see tests folder). All the tests (new and old ones) gave 0 errors.

sanjib2006 · 2026-06-02T17:00:28Z

Screenshots:

sanjib2006 · 2026-06-02T17:09:28Z

+
        if self._url_key_name and self._url_key_name.endswith("/"):
            self._url_key_name = self._url_key_name[:-1]
-        url = f"{self._url_key_name}/api/v2/observables/"


/api/v2/observables/ in current version of yeti this endpoint accepts tags, value and type only

Check here

sanjib2006 · 2026-06-02T17:17:48Z

        # create observable with `obs_value` if it doesn't exists
        # new context, tags, source are appended with existing ones
+
+        url = f"{self._url_key_name}/api/v2/observables/extended"


/api/v2/observables/extended we can send extra context through this endpoint only

Check here

sanjib2006 · 2026-06-02T17:19:30Z

+            "observable": {
+                # there are type mismatches between YETI and IntelOwl
+                # so for now we are not senging the type to YETI
+                # "type": obs_type,


Without providing the type it defaults to jarm. Check the screenshot.

sanjib2006 · 2026-06-02T17:47:44Z

Regarding the type:

These are the supported types in YETI.

I am not sure about how v1 was working but I have checked v2 properly.

If we do not send type in the payload then it defaults the type to jarm (I tried a file and an ip got same)

The current implementation has a obs_type var:

IntelOwl/api_app/connectors_manager/connectors/yeti.py

Lines 17 to 25 in 38ebe10

    
           def run(self): 
        
               # get observable value and type 
        
               if self._job.is_sample: 
        
                   obs_value = self._job.analyzable.md5 
        
                   obs_type = "file" 
        
               else: 
        
                   obs_value = self._job.analyzable.name 
        
                   obs_type = self._job.analyzable.classification

But there are some mismatches .classification returns "ip" and not "ipv4" or "ipv6" but yeti does not recognize ip type (it recognizes ipv4 or ipv6)

IntelOwl/api_app/choices.py

Lines 105 to 112 in 38ebe10

    
           class Classification(models.TextChoices): 
        
               IP = "ip" 
        
               URL = "url" 
        
               DOMAIN = "domain" 
        
               HASH = "hash" 
        
               GENERIC = "generic" 
        
               FILE = "file"

So we have to do something for ip, domain and hash, otherwise will get errors for these unknown types.
Well YETI has a guess_type function which determines the observable type itself (by sending "type": "guess") but currently the endpoints I am working on i.e. .../ and .../extended/ do not call it(I have tested that also).

sanjib2006 · 2026-06-02T17:54:57Z

I will add the related tests and do something for the type tomorrow. I have provided links to the official source wherever needed because the YETI docs do not contain info about these things.

… condition

sanjib2006 · 2026-06-04T14:05:18Z

            "report": f"{settings.WEB_CLIENT_URL}/jobs/{self.job_id}",
            "status": "analyzed",
-            "date": str(self._job.finished_analysis_time),
+            "date": str(self._job.received_request_time),


getting NULL (check the first screenshot) probable reason is race condition so I replaced it with the received_request_time

sanjib2006 · 2026-06-04T14:09:36Z

Screenshots:

Tests

with type parameter

sanjib2006 · 2026-06-04T14:15:24Z

            obs_type = self._job.analyzable.classification

+        # convert obs_type to YETI's expected types if possible
+        if obs_type == "ip":


these three were conflicting

ip - converting it to ipv4 or ipv6 (generic if any problem)

domain to hostname

about hash there are multiple types so for now I am making it generic

sanjib2006 · 2026-06-04T14:17:30Z

            "status": "analyzed",
-            "date": str(self._job.finished_analysis_time),
+            "date": str(self._job.received_request_time),
            "description": f"IntelOwl's analysis report for Job: {self.job_id} | {obs_value} | {obs_type}",


Since we changed obs_type to match the YETI, this context will also contain the YETI types and not Intelowl

sanjib2006 · 2026-06-04T14:29:53Z

Hello @mlodic!
This is done. Please have a look whenever you get time.

mlodic

a question: why the IP address is classified as JARM in this case?

mlodic · 2026-06-04T17:41:08Z

+
+        # auth
+        auth_url = f"{self._url_key_name}/api/v2/auth/api-token"
+        auth_headers = {"x-yeti-apikey": self._api_key_name}


both here and in the subsequent headers request we can customize it to add "IntelOwl" as user agent. This is common practice to help detecting integrations requests.

I have added the user agent in the latest commit

mlodic · 2026-06-04T17:43:18Z

+
        # create context
        context = {
            "source": "IntelOwl",


should the source removed here too in favor of an user-agent string?

Without providing source this becomes blank (see 2nd and 3rd), similar in the observable page

sanjib2006 · 2026-06-05T17:56:57Z

a question: why the IP address is classified as JARM in this case?

I checked it, the intended response if type parameter is not provided is to raise an error but this endpoint is working without sending type because of a syntax bug

Check this line

The parameter here should be discriminator and not discriminant (spelling mistake) and this leads to trying all models and it just picks the one which validates first which in my case (the observable I tested ) matches JARM first which does not has any validator actually (anything can match to jarm in yeti currently)

I have raised an issue for this yeti-platform/yeti#1273

refactor(connectors): update YETI to API v2 payload and auth schemas

f662854

sanjib2006 commented Jun 2, 2026

View reviewed changes

sanjib2006 linked an issue Jun 2, 2026 that may be closed by this pull request

[Connectors] - Fix Yeti API v2 Authentication #3707

Open

sanjib2006 added 3 commits June 3, 2026 20:25

refactor(connectors): YETI observable type handling

e5ced1b

fix(yeti): update context date to received_request_time to avoid race…

b8f7300

… condition

test(yeti): add unit tests for YETI connector auth and response handling

46ad92e

sanjib2006 commented Jun 4, 2026

View reviewed changes

sanjib2006 marked this pull request as ready for review June 4, 2026 14:30

sanjib2006 requested a review from mlodic June 4, 2026 14:49

mlodic reviewed Jun 4, 2026

View reviewed changes

YETI: Add IntelOwl as User-Agent to request headers

b38d456

Uh oh!

Conversation

sanjib2006 commented Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Checklist

Uh oh!

sanjib2006 commented Jun 2, 2026

Uh oh!

sanjib2006 Jun 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanjib2006 commented Jun 2, 2026

Uh oh!

sanjib2006 commented Jun 2, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanjib2006 commented Jun 4, 2026

Tests

with type parameter

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanjib2006 commented Jun 4, 2026

Uh oh!

mlodic left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

sanjib2006 commented Jun 5, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

sanjib2006 commented Jun 2, 2026 •

edited

Loading

sanjib2006 Jun 2, 2026 •

edited

Loading