Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failure to update configuration with multiple dependent controller services and processors #39

Open
ennui93 opened this issue Jan 23, 2018 · 0 comments
Open

Failure to update configuration with multiple dependent controller services and processors #39

ennui93 opened this issue Jan 23, 2018 · 0 comments

Comments

@ennui93
Copy link
Contributor

ennui93 commented Jan 23, 2018 •
edited
Loading

Looks like there has been a regression in version 1.1.32 (ce60cb1) when updating a controller service with a dependent processor, when said controller service is itself dependent on another controller service (i.e. processor 1 -> controller service 2 -> controller service 1, where -> is a dependency). This appears to have originated in commit 1c475d5. Following the steps from issue #35 now results in:

$ java -jar nifi-config/target/nifi-deploy-config-*.jar -m updateConfig -b "root > controller_service_dependencies" -c controller_service_dependencies.json -n http://localhost:8079/nifi-api
2018-01-23 09:57:23 | INFO  | [main] c.g.h.n.c.Main:109 - Starting config_nifi 1.1.32-SNAPSHOT on mode updateConfig
2018-01-23 09:57:23 | INFO  | [main] c.g.h.n.c.Main:132 - Communicate with nifi 1.3.0
2018-01-23 09:57:23 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:66 - Processing : controller_service_dependencies.json
2018-01-23 09:57:23 | INFO  | [main] c.g.h.n.c.s.ProcessorService:56 -  GetFile (33b7ba02-6f96-3915-38aa-a9c999512279) is already STOPPED
2018-01-23 09:57:23 | INFO  | [main] c.g.h.n.c.s.ProcessorService:56 -  ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) is already STOPPED
2018-01-23 09:57:23 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:76 - [root, controller_service_dependencies] is stopped
2018-01-23 09:57:23 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:288 - Update config processor ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:320 - Processor ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) have config updated 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:288 - Update config processor GetFile (33b7ba02-6f96-3915-38aa-a9c999512279) 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:320 - Processor GetFile (33b7ba02-6f96-3915-38aa-a9c999512279) have config updated 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ProcessorService:56 -  ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) is already STOPPED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:288 - Update config processor ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:320 - Processor ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) have config updated 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:211 -  JsonRecordSetWriter (3efc8893-ab75-35d9-d2d6-d1c247ac4aab) is replaced by (3efc8893-ab75-35d9-2e66-82ea3ef8ac85)
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ProcessorService:56 -  ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) is already STOPPED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  JsonRecordSetWriter (3efc8893-ab75-35d9-2e66-82ea3ef8ac85) update for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-5-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  JsonRecordSetWriter (3efc8893-ab75-35d9-2e66-82ea3ef8ac85) waiting for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-5-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  JsonRecordSetWriter (3efc8893-ab75-35d9-2e66-82ea3ef8ac85) is DISABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:76 -  JsonRecordSetWriter (3efc8893-ab75-35d9-2e66-82ea3ef8ac85) is UPDATED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:76 -  JsonRecordSetWriter (3efc8893-ab75-35d9-d2d6-d1c247ac4aab) is UPDATED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:211 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9125-493416ff8f6f) is replaced by (8d17b40b-5cd3-3134-da58-49dbb5c2f58e)
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ProcessorService:56 -  ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) is already STOPPED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:288 - Update config processor ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:320 - Processor ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) have config updated 
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.UpdateProcessorService:211 -  CSVReader (7db4fe70-133f-3576-945d-f720958368db) is replaced by (7db4fe70-133f-3576-fc87-0e015daa8412)
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ProcessorService:56 -  ConvertRecord (55f2cf94-a952-343a-5c72-471ed80adcfd) is already STOPPED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) update for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-12-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) waiting for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-12-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) is DISABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:76 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) is UPDATED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  JsonRecordSetWriter (3efc8893-ab75-35d9-d2d6-d1c247ac4aab) update for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-13-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  JsonRecordSetWriter (3efc8893-ab75-35d9-d2d6-d1c247ac4aab) waiting for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-13-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  JsonRecordSetWriter (3efc8893-ab75-35d9-d2d6-d1c247ac4aab) is DISABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:214 -  JsonRecordSetWriter (3efc8893-ab75-35d9-d2d6-d1c247ac4aab) trying removing
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:216 -  JsonRecordSetWriter (3efc8893-ab75-35d9-d2d6-d1c247ac4aab) is removed
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9125-493416ff8f6f) update for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-14-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9125-493416ff8f6f) waiting for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-14-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9125-493416ff8f6f) is DISABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:214 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9125-493416ff8f6f) trying removing
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:216 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9125-493416ff8f6f) is removed
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  CSVReader (7db4fe70-133f-3576-945d-f720958368db) update for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-15-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  CSVReader (7db4fe70-133f-3576-945d-f720958368db) waiting for DISABLED
2018-01-23 09:57:24 | INFO  | [pool-15-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  CSVReader (7db4fe70-133f-3576-945d-f720958368db) is DISABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:214 -  CSVReader (7db4fe70-133f-3576-945d-f720958368db) trying removing
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:216 -  CSVReader (7db4fe70-133f-3576-945d-f720958368db) is removed
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9f89-ea8075fffac6) update for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-16-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9f89-ea8075fffac6) waiting for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-16-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-9f89-ea8075fffac6) is ENABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  JsonRecordSetWriter (3efc8893-ab75-35d9-2e66-82ea3ef8ac85) update for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-17-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  JsonRecordSetWriter (3efc8893-ab75-35d9-2e66-82ea3ef8ac85) waiting for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-17-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  JsonRecordSetWriter (3efc8893-ab75-35d9-2e66-82ea3ef8ac85) is ENABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-da58-49dbb5c2f58e) update for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-18-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-da58-49dbb5c2f58e) waiting for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-18-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  AvroSchemaRegistry (8d17b40b-5cd3-3134-da58-49dbb5c2f58e) is ENABLED
2018-01-23 09:57:24 | INFO  | [main] c.g.h.n.c.s.ControllerServicesService:110 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) update for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-19-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) waiting for ENABLED
2018-01-23 09:57:24 | INFO  | [pool-19-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) is ENABLING
2018-01-23 09:57:26 | INFO  | [pool-19-thread-1] c.g.h.n.c.s.ControllerServicesService:114 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) waiting for ENABLED
2018-01-23 09:57:26 | INFO  | [pool-19-thread-1] c.g.h.n.c.s.ControllerServicesService:116 -  CSVReader (7db4fe70-133f-3576-fc87-0e015daa8412) is ENABLED
2018-01-23 09:57:26 | INFO  | [pool-21-thread-1] c.g.h.n.c.s.ControllerServicesService:139 - 3efc8893-ab75-35d9-d2d6-d1c247ac4aab cannot be enabled because it is not valid: '' is invalid because Failed to run validation due to java.lang.IllegalStateException: Could not find any Processor, Reporting Task, or Controller Service with identifier 3efc8893-ab75-35d9-d2d6-d1c247ac4aab
Exception in thread "main" com.github.hermannpencole.nifi.config.model.ConfigException: com.github.hermannpencole.nifi.swagger.ApiException: Conflict
	at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runTimeout(FunctionUtils.java:40)
	at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runWhile(FunctionUtils.java:65)
	at com.github.hermannpencole.nifi.config.service.ControllerServicesService.setStateReferencingControllerServices(ControllerServicesService.java:127)
	at com.github.hermannpencole.nifi.config.service.UpdateProcessorService.updateControllers(UpdateProcessorService.java:176)
	at com.github.hermannpencole.nifi.config.service.UpdateProcessorService.updateByBranch(UpdateProcessorService.java:88)
	at com.github.hermannpencole.nifi.config.Main.main(Main.java:137)
Caused by: com.github.hermannpencole.nifi.swagger.ApiException: Conflict
	at com.github.hermannpencole.nifi.swagger.ApiClient.handleResponse(ApiClient.java:1104)
	at com.github.hermannpencole.nifi.swagger.ApiClient.execute(ApiClient.java:1027)
	at com.github.hermannpencole.nifi.swagger.client.ControllerServicesApi.updateControllerServiceReferencesWithHttpInfo(ControllerServicesApi.java:1011)
	at com.github.hermannpencole.nifi.swagger.client.ControllerServicesApi.updateControllerServiceReferences(ControllerServicesApi.java:996)
	at com.github.hermannpencole.nifi.config.service.ControllerServicesService.lambda$setStateReferencingControllerServices$1(ControllerServicesService.java:137)
	at com.github.hermannpencole.nifi.config.utils.FunctionUtils.runWhile(FunctionUtils.java:54)
	at com.github.hermannpencole.nifi.config.utils.FunctionUtils.lambda$runWhile$0(FunctionUtils.java:65)
	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:514)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1167)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:641)
	at java.base/java.lang.Thread.run(Thread.java:844)

Perhaps this issue was always present, but only noticeable now since nifi-config attempts to limit the scope of controller services it enables/disables in 1c475d5?
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@ennui93