[16.0][IMP] Add support for migration scan between repository (modules moved to new repos)

[sebalix]

This supports the migration scan for modules that have been moved to another repository (like mail modules moved from OCA/social to OCA/mail starting from 18.0):

This commit also adds new technical flags and migration states to improve module qualification during a migration:
- Moved to standard
- Moved to OCA
- Moved to generic repo (specific modules moved to generic repo)

Migration with such states won't trigger a migration scan as modules could be different. E.g. l10n_eu_oss Odoo module is not the same than OCA one starting from 15.0, and OCA renamed its module l10n_eu_oss_oca from this version, either to complete std implementation, or to propose another one. So these states will help integrators to identify such modules.

Later we could add a feature to set a given module as renamed from a given version, like l10n_eu_oss renamed to l10n_eu_oss_oca, so the module won't be qualified with Moved to standard state anymore as it's just a renaming. But such feature will require to also improve oca-port to handle such case to perform a migration scan.

Project module migration status will be updated accordingly as well:

[hparfr]

I didn't read carefully this PR yet.

I think it can bee cool if we can know between 2 major versions (16->18):

• module rename
• change of repository
• discontinuation of a module (because not needed anymore | or merged into another (=rename)
• split of module

Odoo Standard

Example: coupon, gift_card -> loyalty
Expected: coupon: removed, loyalty added. We don't care about migration for std modules.

OCA

Renaming:
Example: delivery_carrier_customer_info -> partner_delivery_info
About migration: it' a problem of oca-port.

Repository change:
OCA/stock-logistics-warehouse/
stock_quant_package_dimension -> OCA/stock-logistics-tracking/
stock_quant_package_dimension

Split:
(14) base_delivery_carrier_label -> (16) base_delivery_carrier_label + (16) delivery_carrier_account

Publishing on OCA:

(14) custom_module -> (16) oca_module

How does oca-port handle all these cases ?

IMO, having just a "rename" link between two modules in diffrent versions is ok here.

[hparfr]

moved_to_standard does it exists ?

May be just moved is enough ?

[sebalix]

Yes, module l10n_eu_oss (while it's not really a move, more Odoo that creates its own, and OCA needed to rename its own with suffix _oca), same for knowledge module, Odoo creates its own enterprise module starting from 16.0, forcing OCA to rename it document_knowledge.

So I agree it's not really a "move" functionally speaking, it's the technical module name that has been moved only, but that highlight something we should take care during a migration.

[sebalix]

@hparfr I agree with all your remarks. But some of these data cannot easily be retrieved only by scanning code (or we cannot be confident with them to automate the process if we start to parse PR description/commit content to understand what happened - even module renaming is difficult to see with how git is working, Mercurial/hg handles a metadata that a file has been renamed, while git tries to show that by comparing file content, generating false-positive if you change the files at the same time).

So basically how I see things:

1. grab accurate data from code scanning (we should be confident in them), and highlight potential issues like with these "Moved to X" states (they are the exceptions)
2. human intervention to complete/fix these data (here it starts to be interesting to get a community work to spread the load) => that's the next step, able to qualify that:
   1. a module has been renamed, but the implementation is the same, so migration scan is still possible to forward patches
   2. a module has been replaced by/ merged in another one (OCA or std), migration scan is not possible anymore, we are not talking about the same module here

For now oca-port is not able to handle module renaming, but it's on the roadmap. I still don't know if oca-port should record such metadata in .oca/oca-port/ hidden folder in repositories, but this has to be stored one way in another in odoo_repository data models, either automatically from oca-port results if any, and/or manually by users.

[hparfr]

On OCA.

We have the issue for instance for stock_quant_package_dimension moved from OCA/stock-logistics-warehouse/ to OCA/stock-logistics-tracking/

Someone can try to migrate the module OCA/stock-logistics-warehouse#18.0 If it's the same name, we may have a warning by pypi. But if there is module rename, we may never see it.

I think of two implementations:

A) On the repo:
add a new command on oca-github-bot. Like
/ocabot migration stock_quant_package_dimension from OCA/stock-logistics-warehouse/stock_quant_package_dimension

It will create on OCA/stock-logistics-warehouse/ # Issue Migration to version 18.0 a line to indicate the module has been moved here.

B) On a global repo:
Let's imagine a new repo on OCA/module-mapping.
With only one branch, and one file

[18.0]
OCA/stock-logistics-tracking/stock_quant_package_dimension: from OCA/stock-logistics-warehouse/stock_quant_package_dimension
[17.0]
odoo/loyalty: from odoo/coupon, odoo/gift_card

it can be then used for generate openupgrade#apriori
For oca modules, it should be edited only by oca-github-bot during merge.

[sebalix]

I like the idea to get a mapping file somewhere in OCA to consolidate this knowledge that can be used by any tool 👍
Do you want to open the discussion with the community?

EDIT: we should use a JSON or YAML format with such file so pre-commit can validate its format

[hparfr]

Do you know where i can put this discussion ?

[sebalix]

On the contributor mailing list I guess

[hparfr]

I thinked a lot about this.

One first easier step is first to declare the change.

OCA/oca-github-bot#317

In a second time, leverage the tagged PR by ocabot to generate a list of the change and publish it in a new repo.
I did not create a discussion here yet. But I have an idea of how to do it.

EDIT: we should use a JSON or YAML format with such file so pre-commit can validate its format

I found something a bit akward but lot more simple and no subject to conflicts: just create folders and files.

Like this:

 tree
.
├── 16.0
│   ├── account
│   │   └── odoo
│   │       └── odoo
│   │           └── oca
│   │               └── account-financial-tool
│   │                   ├── account_balance_line
│   │                   └── account_force-removal
│   ├── account_edi_facturx
│   │   └── odoo
│   │       └── odoo
│   │           └── odoo
│   │               └── odoo
│   │                   └── account_edi_ubl_cii
│   ├── account_edi_ubl
│   │   └── odoo
│   │       └── odoo
│   │           └── odoo
│   │               └── odoo
│   │                   └── account_edi_ubl_cii
│   ├── account_edi_ubl_bis3
│   │   └── odoo
│   │       └── odoo
│   │           └── odoo
│   │               └── odoo
│   │                   └── account_edi_ubl_cii
│   ├── account_reconciliation_widget
│   │   └── oca
│   │       └── account-reconcile
│   │           └── oca
│   │               └── account-reconcile
│   │                   └── account_reconcile_oca
│   ├── base_delivery_carrier_label
│   │   └── oca
│   │       └── delivery-carrier
│   ├── loyalty
│   │   └── odoo
│   │       └── odoo
│   │           └── odoo
│   │               └── odoo
│   │                   └── coupon
│   └── payment_test
│       └── odoo
│           └── odoo
│               └── odoo
│                   └── odoo
│                       └── payment_demo
├── 17.0
│   └── base_delivery_carrier_label
│       └── oca
│           └── delivery-carrier
└── 18.0
    ├── base_delivery_carrier_label
    │   └── oca
    │       └── delivery-carrier
    │           └── oca
    │               └── delivery-carrier
    │                   └── base_delivery_carrier_label
    ├── delivery_carrier_account
    │   └── oca
    │       └── delivery-carrier
    │           └── oca
    │               └── delivery-carrier
    │                   └── base_delivery_carrier_label
    ├── partner_delivery_info
    │   └── oca
    │       └── delivery-carrier
    │           └── oca
    │               └── delivery-carrier
    │                   └── delivery_carrier_customer_info
    └── stock_quant_package_product_packaging
        └── oca
            └── stock-logistics-tracking
                └── oca
                    └── stock-logistics-workflow
                        └── stock_quant_package_product_packaging

the layout is:
`(d) branch / (d) module name / (d) actual org / (d) actual repo / (d) source org / (d) source repo / (f) old module name

(d) : directory
(f) : file (empty)

A ci workflow can then just generate a listing directory .html and an apriori (openupgrade) file (the algo may be quite stupid).
In scripts, know if a module already exist in one branch, it just a check if HTTP_200 == curl to github.com/oca/lookuprepo/tree/master/module_name/

[sebalix]

I like the idea, the structure is a bit akward like you said but the usage is simple.
So if I translate correctly this example:

├── 16.0
│   ├── account
│   │   └── odoo
│   │       └── odoo
│   │           └── oca
│   │               └── account-financial-tool
│   │                   ├── account_balance_line
│   │                   └── account_force-removal

That means: 15.0 version of modules account_balance_line + account_force_removal have been included in account starting from 16.0. Right? source_org and source_repo folders always refer to the previous version.

EDIT: But how to use that structure easily to answer the question "how this {module} of version {x} was migrated in next version?", we only have as input the name of the module, and its current version. The current structure implies to know at first
the answer 🤔

EDIT: I would change the source_org / source_repo / old_module_name to new_org / new_repo / new_module_name, so we refer to the next version, not the previous one.

[hparfr]

That means: 15.0 version of modules account_balance_line + account_force_removal have been included in account starting from 16.0. Right?

Yes.
It took example of https://github.com/OCA/OpenUpgrade/blob/16.0/openupgrade_scripts/apriori.py#L90-L92

source_org and source_repo folders always refer to the previous version

Not exactly.
source_org and source_repo folders always refer to a previous version

For instance, if module delivery_carrier_customer_info (v16 but not v17) becomes partner_delivery_info in v18. We will have the structure:

NOTE: I didn't state it in my first example, but I think know it's better to always complete this file: standard migration, module addition, and of course module rename / module merge ...

├── 16.0
    ├── delivery_carrier_customer_info
    │   └── oca
    │       └── delivery-carrier
├── 17.0
└── 18.0
    ├── partner_delivery_info
    │   └── oca
    │       └── delivery-carrier
    │           └── oca
    │               └── delivery-carrier
    │                   └── delivery_carrier_customer_info

I do not know the good practice when someone needs it in v17.
Should she use the name delivery_carrier_customer_infoorpartner_delivery_info` ?

I think it's better to keep the old repo / old name in this case, but it's not necessary to enforce it. I guess both will work within apriori.py.

EDIT: But how to use that structure easily to answer the question "how this {module} of version {x} was migrated in next version?", we only have as input the name of the module, and its current version. The current structure implies to know at first the answer 🤔

The structure tells you the actual situation and is never changed. Like I said just previously, if some module is not migrated every version, you don't konw where it will in the intermediate version if it's ported lated (v17: delivery_carrier_customer_info or partner_delivery_info).

With the following timeline:

• 1. just after the merge of delivery_carrier_customer_info in v15
• 1. just after the merge of delivery_carrier_customer_info in v16
• 1. just after the merge of partner_delivery_info in v18
• 1. just after the merge of partner_delivery_info in v17

1. delivery_carrier_customer_info in v15

$ find . -name 'delivery_carrier_customer_info'
15.0/delivery_carrier_customer_info/oca/delivery-carrier

We know the module at this time only exists in v15

• 1. delivery_carrier_customer_info in v16

$ find . -name 'delivery_carrier_customer_info'
15.0/delivery_carrier_customer_info/oca/delivery-carrier
16.0/delivery_carrier_customer_info/oca/delivery-carrier

• 1. partner_delivery_info in v18

$ find . -name 'delivery_carrier_customer_info'
15.0/delivery_carrier_customer_info/oca/delivery-carrier
16.0/delivery_carrier_customer_info/oca/delivery-carrier
18.0/partner_delivery_info/oca/delivery-carrier/oca/delivery-carrier/delivery_carrier_customer_info

At this time, we know the module has not been migrated in v17 yet.
We need to now know if partner_delivery_info was existing at this time:

$ find . -name 'partner_delivery_info'
18.0/partner_delivery_info/oca/delivery-carrier/oca/delivery-carrier/delivery_carrier_customer_info

So here, we know it's the first occurence of partner_delivery_info; It's not a merge of the modules, just a rename.

• 1. partner_delivery_info in v17

$ find . -name 'delivery_carrier_customer_info'
15.0/delivery_carrier_customer_info/oca/delivery-carrier
16.0/delivery_carrier_customer_info/oca/delivery-carrier
17.0/partner_delivery_info/oca/delivery-carrier/oca/delivery-carrier/delivery_carrier_customer_info
18.0/partner_delivery_info/oca/delivery-carrier/oca/delivery-carrier/delivery_carrier_customer_info

Here, we see they did not enforce the good practice and renamed the module in v17, but they not forget to tell us !

I think we will add a ci step, it will generate a apriori.py, some reports, yaml... for easy parsing and inverse lookup when you need it.

EDIT: I would change the source_org / source_repo / old_module_name to new_org / new_repo / new_module_name, so we refer to the next version, not the previous one.

I didn't get it :)

[sebalix]

Ok, so this structure is answering the question perfectly if we use find as in your example (we get the whole picture for a given module), my concern was about the initial example with curl, where you need to know all parameters first to generate the query (which is not useful).

But whatever, as soon as we have these info gathered somewhere, we can fetch them and re-consolidate these data the way we want to ease requests (like odoo_repository downloading and parsing this file from time to time to update links between modules).

Sounds good 👍

[sebalix]

Will merge this one, and will open another PR to address module renaming:

IMO, having just a "rename" link between two modules in diffrent versions is ok here.

It will also leverage the renaming of module with oca-port:

• [IMP] Add support for module renaming oca-port#74

Current PR has been tested live for weeks now, working fine.