Object detection adversary in LightningModule by mzweilin · Pull Request #115 · IntelLabs/MART

mzweilin · 2023-03-24T17:22:07Z

What does this PR do?

This PR adds support to object detection adversary. As the implementation is modality-aware, it should not be hard to implement RGB-Depth adversary.

mart \
experiment=ArmoryCarlaOverObjDet_TorchvisionFasterRCNN \
fit=false \
+trainer.limit_test_batches=1 \
trainer=gpu \
+attack@model.modules.input_adv_test=object_detection_rgb_mask_adversary \
+model.test_sequence.seq005=input_adv_test \
model.test_sequence.seq010.preprocessor=["input_adv_test"]

Type of change

Please check all relevant options.

Improvement (non-breaking)
Bug fix (non-breaking)
New feature (non-breaking)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

Testing

Please describe the tests that you ran to verify your changes. Consider listing any relevant details of your test configuration.

pytest tests

Before submitting

The title is self-explanatory and the description concisely explains the PR
My PR does only one thing, instead of bundling different changes together
I list all the breaking changes introduced by this pull request
I have commented my code
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I have run pre-commit hooks with pre-commit run -a command without errors

Did you have fun?

Make sure you had fun coding 🙃

dxoigmn

See comments. What the merge plan here? Do you want to merge this into #103 before we merge #103?

mart/attack/enforcer.py

dxoigmn · 2023-04-01T14:25:02Z

mart/attack/enforcer.py

    def __init__(self, **modality_constraints: dict[str, dict[str, Constraint]]) -> None:
        self.modality_constraints = modality_constraints
+        # Prepare for modality_dispatch().
+        self.modality_func = {


Maybe self._enforce_modality? What's the benefit of creating a partial here ?

It does feel like the partial is unnecessary given that you can pass self._enforce and pass modality in modality_dispatch.

Good idea. I have revised modality_dispatch() to support modality_func() that accepts modality as a keyword argument.

mart/attack/perturber.py

dxoigmn · 2023-04-01T14:37:38Z

mart/attack/perturber.py

+    def configure_perturbation(self, input: torch.Tensor | tuple | tuple[dict[str, torch.Tensor]]):
+        def create_and_initialize(data, *, input, target, modality="default"):
+            # Though data and target are not used, they are required placeholders for modality_dispatch().
+            pert = torch.empty_like(input, requires_grad=True)


I think you need to specify dtype=torch.float to override input's dtype. If input is uint8, we probably don't want the perturbation to uint8 too.

Fixed.

I guess this won't affect mixed precision training?

That's a good point. I'm not sure and it isn't really something I have thought about.

dxoigmn · 2023-04-01T15:07:39Z

mart/utils/modality_dispatch.py

+__all__ = ["modality_dispatch"]
+
+
+def modality_dispatch(


If you want to get really fancy you could use functools.singledispatch.

Does this obviate the need for Projector and Composer to be batch aware? If so, I would remove that code.

Yes, we can make Projector and Composer simpler with modality_dispatch.

Can you remove the batch code from Projector and Composer then?

dxoigmn · 2023-04-01T15:08:39Z

mart/attack/perturber.py

-            pert = torch.empty_like(inp, dtype=torch.float, requires_grad=True)
-            self.initializer(pert)
+    def configure_perturbation(self, input: torch.Tensor | tuple | tuple[dict[str, torch.Tensor]]):
+        def create_and_initialize(data, *, input, target, modality="default"):


Can you make "default" a constant somewhere?

I made it a class constant.

dxoigmn · 2023-04-01T15:12:28Z

mart/attack/perturber.py

+        # Recursively configure perturbation in tensor.
+        # Though only input=input is used, we have to fill the placeholders of data and target.
+        self.perturbation = modality_dispatch(
+            modality_func, input, input=input, target=input, modality="default"


Why does target=None not work? target=input is confusing...

If input is a list or tuple, target=None won't work.

Can you make it work? I'm guessing you need something that zips well with input? You can use target = cycle([None]) if target is None.

dxoigmn · 2023-04-01T15:14:39Z

mart/attack/perturber.py

+    @property
+    def parameter_groups(self):
+        """Extract parameter groups for optimization from perturbation tensor(s)."""
+        param_groups = self._parameter_groups(self.perturbation)
+        return param_groups


Why is this a property? It is only used for testing purposes? If so, then there's probably something wrong with the tests...

The property decorator is removed.

dxoigmn · 2023-04-01T15:15:40Z

mart/attack/perturber.py

-            self.perturbation = create_and_initialize(input)
+            raise ValueError(f"Unsupported data type of input: {type(pert)}.")
+
+    def project(self, perturbation, *, input, target, **kwargs):


Rename to project_ to adhere to in-place naming scheme.

dxoigmn

See comments.

mart/attack/adversary.py

dxoigmn · 2023-04-06T15:00:49Z

mart/attack/composer.py

        return input * (1 - mask) + perturbation * mask
+
+
+class MaskAdditive(Composer):


Inherit from Additive?

dxoigmn · 2023-04-06T15:03:59Z

mart/attack/perturber.py

+        gradient_modifier: GradientModifier | dict[str, GradientModifier] | None = None,
+        projector: Projector | dict[str, Projector] | None = None,
        objective: Objective | None = None,
+        optim_params: dict[str, dict[str, Any]] | None = None,


Should we just make optimizer a Callable | dict[str, Callable]? That will be more uniform like the existing changes.

There is no difference between multiple optimizers and a single optimizer with multiple parameter groups. We just have to make sure we step the multiple optimizer approach in sync.

It will probably simplify a lot of the code changes to the parameters.

dxoigmn · 2023-04-06T15:05:26Z

mart/attack/perturber.py

 class Perturber(pl.LightningModule):
    """Peturbation optimization module."""

+    MODALITY_DEFAULT = "default"


I would make this a module-level constant. Will get rid of the self. prefix in self.MODALITY_DEFAULT.

Actually, this constant should be moved to modality_dispatch.py and you should import it from there.

dxoigmn · 2023-04-06T15:07:18Z

mart/utils/modality_dispatch.py

+    *,
+    input: Tensor | tuple | list[Tensor] | dict[str, Tensor],
+    target: torch.Tensor | dict[str, Any] | list[dict[str, Any]] | None,
+    modality: str = "default",


This should be MODALITY_DEFAULT.

dxoigmn · 2023-04-06T15:09:15Z

mart/attack/perturber.py

+        # When an Adversary takes input from another module in the sequence, we would have to specify kwargs of Adversary, and model would be a required kwarg.
+        outputs = model(**batch, model=None)


Is there an example test where this happens?

dxoigmn · 2023-04-06T15:09:51Z

mart/utils/modality_dispatch.py

+__all__ = ["modality_dispatch"]
+
+
+def modality_dispatch(


Can you remove the batch code from Projector and Composer then?

dxoigmn added 30 commits March 13, 2023 14:20

Remove NoAdversary

f36d8c1

Remove NoAdversary from CIFAR10 adversarial training

dcf7114

Remove NoAdversary from RetinaNet model

75886b6

Fix COCO_TorchvisionFasterRCNN_Adv experiment

52aa94d

Remove NoAdversary tests

bbed2d6

First stab at treating adversary as LightningModule

f3f7b1b

style

0e16460

bugfix

e25af6f

Integrate Perturber into LitPerturber

36295b5

Integrate objective to LitPerturber

c68d9a8

Cleanup use of objective function to compute gain

689da74

bugfix

3d70dde

Make adversarial trainer silent

2990cab

style

d5e17f7

Move threat model into LitPerturber

b8f8761

Make attack callbacks plain PL callbacks

d39c5c1

comment

5b7ee68

Remove Perturber

f980315

comment

90fb9ab

Better silence

00aefee

Integrate LitPerturber into Adversary

86bbc13

Uncombine Adversary into Adversary and LitPerturber

7a12bf8

cleanup

ad1872a

Move silence into utils

1aa4fc0

bugfix

d09c5b8

bugfix

78701a4

cleanup

8143b5d

Enable dependency injection on Adversary

7bfc98f

Make dependency injection backwards compatible

eaf1607

Replace max_iters with trainer.limit_train_batches

bf5df50

mzweilin added 2 commits March 31, 2023 11:03

Comment.

33acb97

Make image_visualizer modality-aware.

3c6a7f7

mzweilin marked this pull request as draft March 31, 2023 18:16

mzweilin marked this pull request as ready for review March 31, 2023 18:16

Enable visualizer.

b070723

mzweilin requested a review from dxoigmn March 31, 2023 18:17

This was referenced Mar 31, 2023

Modality-aware composer #126

Closed

Make Projector batch aware #120

Merged

mzweilin added 2 commits March 31, 2023 15:30

Clean up nop().

3a86b69

Cleanup.

7bbe347

mzweilin mentioned this pull request Mar 31, 2023

Update GradientModifier to be an in-place operation #123

Merged

14 tasks

dxoigmn requested changes Apr 1, 2023

View reviewed changes

mzweilin added 9 commits April 3, 2023 10:08

project() -> project_().

b86cdc9

Remove property decorator for parameter_groups().

b68d308

Make sure we create float perturbation tensors.

e961307

Update type annotation.

aa95721

Fix type annotation and comment.

877a00e

Extend modality_dispatch() to support modality_func(modality=?).

6bd0399

MODALITY_DEFAULT = "default"

f33dbc8

Clean up.

9eedad8

Clean up.

3710d48

mzweilin requested a review from dxoigmn April 3, 2023 18:34

mzweilin added 5 commits April 3, 2023 12:21

Merge branch 'main' into objdet_adv

0911769

Allow Adversary not in the first place of a sequence

9ce0ea2

Fix tests.

655f4e0

Allow target is None in modality_dispatch().

48cb56f

Remove target=input.

fa27723

dxoigmn requested changes Apr 6, 2023

View reviewed changes

dxoigmn mentioned this pull request Apr 14, 2023

Replace tuple with Iterable[torch.Tensor] #134

Merged

13 tasks

Base automatically changed from adversary_as_lightningmodule to main June 2, 2023 19:33

		return input * (1 - mask) + perturbation * mask


		class MaskAdditive(Composer):

		# When an Adversary takes input from another module in the sequence, we would have to specify kwargs of Adversary, and model would be a required kwarg.
		outputs = model(**batch, model=None)

Conversation

mzweilin commented Mar 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What does this PR do?

Type of change

Testing

Before submitting

Did you have fun?

Uh oh!

dxoigmn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

dxoigmn Apr 1, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dxoigmn Apr 3, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

dxoigmn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

mzweilin commented Mar 24, 2023 •

edited

Loading

dxoigmn Apr 1, 2023 •

edited

Loading

dxoigmn Apr 3, 2023 •

edited

Loading