In my newly upgraded HA environment we had an issue failing backwards on some of our servers -- the Recommendations engine kept stopping with logfiles that look like this:
*** SolarWinds Recommendations Service v1.2.0.4674, .Net Runtime v4.0.30319 *** 2018-05-22 07:17:39,271 [SolarWindsRecommendations.OnStart] INFO SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details. 2018-05-22 07:17:40,083 [SolarWindsRecommendations.OnStart] WARN SolarWinds.Orion.Common.Helpers.CryptoHelper - Unable to load SolarWinds.Orion.Security.dll from execution directory. System.IO.FileNotFoundException: The system cannot find the file specified. (Exception from HRESULT: 0x80070002) at System.Reflection.RuntimeAssembly.nLoadFile(String path, Evidence evidence) at System.Reflection.Assembly.LoadFile(String path) at SolarWinds.Orion.Common.Helpers.CryptoHelper..cctor() 2018-05-22 07:17:40,083 [SolarWindsRecommendations.OnStart] WARN SolarWinds.Orion.Common.Helpers.CryptoHelper - Trying to load SolarWinds.Orion.Security.dll from Orion install directory... 2018-05-22 07:17:40,099 [SolarWindsRecommendations.OnStart] INFO SolarWinds.Orion.Common.ConnectionStringProvider - SqlWinAuthImpersonate=true 2018-05-22 07:17:40,427 [SolarWindsRecommendations.OnStart] INFO SolarWinds.InformationService.Contract2.InfoServiceProxy - Non-used connection was opened. Information for developers. No impact on product functionality. See verbose log for more details. 2018-05-22 07:17:40,911 [SolarWindsRecommendations.OnStart] INFO SolarWinds.Recommendations.ServiceContainer.ServiceManager - DataProviderService started successfully. 2018-05-22 07:17:41,130 [SolarWindsRecommendations.OnStart] INFO SolarWinds.Recommendations.ServiceContainer.ServiceManager - EngineService started successfully. 2018-05-22 07:17:41,568 [SolarWindsRecommendations.OnStart] INFO SolarWinds.Recommendations.Orchestration.Notifications.SubscriptionService - Starting subscription service 2018-05-22 07:17:41,614 [SolarWindsRecommendations.OnStart] INFO SolarWinds.Orion.Swis.PubSub.MessageBus.MessageBusConnectionProvider - Orion.PubSub.PubSubOverMessageBusEnabled == false, so not attempting message bus connection. (Change in /orion/admin/advancedconfiguration/global.aspx) 2018-05-22 07:17:42,599 [SolarWindsRecommendations.OnStart] ERROR SolarWinds.Recommendations.Orchestration.Notifications.SubscriptionService - Adding swis subscriptions failed. Is swis running? System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Create failed, check fault information. Object reference not set to an instance of an object. (Fault Detail is equal to InfoServiceFaultContract [ System.NullReferenceException: Object reference not set to an instance of an object. at SolarWinds.InformationService.ChangeBroker.PersistentNotificationServiceForwarder.GetSubscriptionKeyFromUri(String subscriptionId) at SolarWinds.InformationService.ChangeBroker.PersistentNotificationServiceForwarder.Subscribe(Subscription subscription, NetworkCredential credentials, String[] tags) at SolarWinds.InformationService.ChangeBroker.SubscriptionProvider.SubscriptionProvider.CreateSubscription(IDictionary`2 columns) at SolarWinds.InformationService.Addons.DataProvider.<Create>d__61.MoveNext() at System.Collections.Generic.List`1.InsertRange(Int32 index, IEnumerable`1 collection) at SolarWinds.InformationService.Core.CrudProcessor.<CreateLocal>d__21.MoveNext() at SolarWinds.InformationService.Core.CrudProcessor.<CreateInternal>d__18.MoveNext() at System.Linq.Enumerable.First[TSource](IEnumerable`1 source) at SolarWinds.InformationService.Core.CrudPro...). 2018-05-22 07:17:42,599 [SolarWindsRecommendations.OnStart] FATAL SolarWinds.Recommendations.Service.RecommendationsService - Unhandled exception caught. System.Reflection.TargetInvocationException: Exception has been thrown by the target of an invocation. ---> System.ServiceModel.FaultException`1[SolarWinds.InformationService.Contract2.InfoServiceFaultContract]: Create failed, check fault information. Object reference not set to an instance of an object. ...
The FATAL error is like 27, Caused by the ERROR on line 15, but the hint is on line 14 as to what is going wrong. for some reason PubSubOverMessageBusEnabled was set to false on a random selection of servers.
to check this visit the advanced settings page (which is where the URL us trying to send you at ... https://your-server/Orion/Admin/AdvancedConfiguration/ServerSpecific.aspx
Check the settings and make sure they are set the same on all of your servers... I can't imagine a case when not setting these all the same would be the right thing to do
Hope this helps someone