Support nested Processing Groups

[hijakk]

Hello, I took a look at the nifi-api-deploy utility as recommended to scarpacci on Stack Overflow, and it definitely seems to be on the right track.
We have fairly complex flows, and the code as written didn't seem to play nicely with undeploying processors/connections at the root level processor group, or the controllers.

Controller undeployment fails with:

Undeploying Controller Service: Test - DBCPConnectionPool
[ERROR] HTTP call failed. Status code: HTTP/1.1 409 Conflict: This NiFi instance has been updated by 'anonymous'. Please refresh to synchronize the view.
Caught: java.lang.AssertionError: Terminated script execution. Expression: null
java.lang.AssertionError: Terminated script execution. Expression: null
    at NiFiDeploy$_run_closure2.doCall(NiFiDeploy.groovy:526)
    at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:503)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1142)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1076)
    at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
    at groovyx.net.http.RESTClient.put(RESTClient.java:163)
    at groovyx.net.http.RESTClient$put$0.call(Unknown Source)
    at NiFiDeploy._changeControllerServiceState(NiFiDeploy.groovy:510)
    at NiFiDeploy.stopControllerService(NiFiDeploy.groovy:470)
    at NiFiDeploy$_handleUndeploy_closure5.doCall(NiFiDeploy.groovy:78)
    at NiFiDeploy.handleUndeploy(NiFiDeploy.groovy:74)
    at NiFiDeploy.run(NiFiDeploy.groovy:539)

Undeployment of root connected processor groups fails with (manually deleting the controller service, else the above error is encountered and the process halts):

Loading Process Groups from NiFi
Undeploying Controller Service: Test - DBCPConnectionPool
Undeploying Process Group: root
[WARN] No such process group found in NiFi
Undeploying Process Group: Group 1
[ERROR] HTTP call failed. Status code: HTTP/1.1 409 Conflict: Cannot delete Process Group because Input Port LocalPort[name=Input,id=abc123] has at least one incoming connection from a component outside of the Process Group. Delete this connection first.
Caught: java.lang.AssertionError: Terminated script execution. Expression: null
java.lang.AssertionError: Terminated script execution. Expression: null
    at NiFiDeploy$_run_closure2.doCall(NiFiDeploy.groovy:526)
    at groovyx.net.http.HTTPBuilder$1.handleResponse(HTTPBuilder.java:503)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1142)
    at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:1076)
    at groovyx.net.http.HTTPBuilder.doRequest(HTTPBuilder.java:515)
    at groovyx.net.http.RESTClient.delete(RESTClient.java:219)
    at groovyx.net.http.RESTClient$delete$1.call(Unknown Source)
    at NiFiDeploy$_handleUndeploy_closure6.doCall(NiFiDeploy.groovy:106)
    at NiFiDeploy.handleUndeploy(NiFiDeploy.groovy:91)
    at NiFiDeploy.run(NiFiDeploy.groovy:539)

Manually deleting and redeploying seemed to work alright.
It also seems to auto start processor groups, even if all contained processors are marked as halted. This appears to be a result of line 352.

[aperepel]

Hi @hijakk , thanks for the feedback. I'll try comment below, but feel free to break out into individual tracks/issues if necessary.

Auto-start processor groups - this line came in later, and yes, it will start them unconditionally. I was basing it on the logic that one does want things to run after deployment. If you could describe your expectations for that example you had, we can revisit.

Root-level things. I personally discourage putting processors on the root level (and tell everyone and their grandmas whenever I can). Nothing but mess, always put things in a Processing Group to keep them manageable.

Having said that, I simply didn't focus on flows which had processing groups interconnected (simple assumption was the PG as a self-contained sub-flow). I definitely waited for someone to show interest in developing this area before investing more time. Could you please share a sample template (maybe file a new issue?).

[hijakk]

Ah, sure thing - Regarding the auto start processor groups - we've got multiple nifi hosts running different flows. If we're looking to use this in a prod environment where we're in a maintenance window, it can be potentially disruptive to have some components like Nifi start after deploying when other services are in an unknown state. The tolerance for this in Nifi is admirable, but services starting on their own after they'd been stopped is something that make the ops guys' eyes twitch in bad ways.

I'll try canning the entire flow into a parent process group to see if it behaves more nicely in terms of undeploy/redeploy.

Is the controller undeploy symptomatic of the root level stuff, or is it something else?

Quick edit:
The root level processors and interconnects/etc are more a matter of convenience than anything else. I think that if the feedback is "put everything that communicates to each other into a single group" as the best practice, we are definitely not married to having stuff in root.

[aperepel]

Hi,

Controller services are a little different, they don't 'suffer' from the root group concerns. I'd like to hear more about an issue you ran into.

If nesting groups and interconnects satisfies your requirements - perfect! Avoiding direct root placement makes things easier. Although I do believe the site-to-site use cases will still require ports to be placed at the root level, so, let's see how much interest there is...

[hijakk]

template_nested_group_example.zip
I'm seeing odd behavior when it comes to dealing with nested subprocess groups. Initially I had (abstractly) a set of two process groups at the root level (those contained in "my giant process group"). I put them both into the parent "my giant process group", ensured they run like I want, and then exported the template.

It looks as though the TemplateInspector.groovy (and/or the NiFiDeploy.groovy) utility doesn't handle nested processor groups? In the attached example, no processors are listed as existing inside the "my giant process group" group, which technically there aren't at the parent process group level - but "my giant process group" contains two further process groups that each have one processor each.

While this example would be entirely reasonable to run as two processors in a processor group one level below root, the complexity of the real flows I'm dealing with is definitely helped by having sub processes to keep the logical flow a nice picture.

If the best practice is to have atomic process groups at the root level containing everything else, the ability to have nested process groups is fairly important.

[aperepel]

I see, makes total sense. Let's rename this issue as 'Support nested Processing Groups' and create a separate one for any Controller Services issue you saw.

[hijakk]

Awesome, thanks. Starting up another issue for the controller service problem