Service Incident
2023-03-31T23:11:00Z
We have finished notifying affected customers.
2023-03-31T20:36:00Z
On Tuesday March 28, 2023, a Netwrix PolicyPak Cloud customer reported finding an unknown computer registered in one of their other software products. Netwrix personnel quickly investigated this report. An initial investigation of this behavior determined that some of this customer’s PolicyPak policy configuration had been applied to a computer belonging to another customer. Netwrix did not and has not found any evidence that the service has been subject to an attack or compromised by an adversary.
As an initial step to limit further impact, Netwrix decided to stop the PolicyPak Cloud policy synchronization service while identifying and mitigating the cause. Upon identifying the suspected cause, Netwrix implemented mitigations for the issue and restored service. The service is now operating normally. Netwrix is carefully monitoring the service for performance and reoccurrence of the issue.
In its detailed investigation, Netwrix has identified the likely cause as a data corruption issue within a major development framework used by PolicyPak Cloud. Netwrix is coordinating with the third party to confirm and fix this issue.
In parallel with the investigation into the root cause, Netwrix conducted a forensic investigation to determine the scope of customers and computers potentially affected by the issue. Netwrix is in the process of notifying customers whose computers were potentially affected and will update this advisory when notifications are complete.
2023-03-30T21:09:00Z
We are observing improvement in service health and performance, as the mitigation work has stabilized. Most PolicyPak Cloud clients should be successfully synchronizing and installations completing successfully.
2023-03-30T13:30:00Z
We are continuing our work to fully restore service. We believe that we have identified the root cause in a third-party software component utilized by the policy synchronization service. We are coordinating with the third party to confirm.
This issue has necessitated implementing mitigations within the PolicyPak Cloud policy synchronization service to work around the suspected third-party issue. The service has been operating in a degraded state while these mitigations deploy. While we are presently observing throughput levels consistent with those prior to the service incident, there is an increased volume of agent requests that will subside as agents successfully check-in and re-establish a normal polling interval.
We believe that normal service levels should resume within the next 24-36 hours. There may be periods of additional degradation as we continue to optimize the mitigation. We will continue to provide updates as the situation evolves.
2023-03-30T14:33:00Z
Our detailed investigation is ongoing. While the PolicyPak Cloud end-user application is operating normally, the policy synchronization service is operating in a degraded state. Customers may notice the following impacts:
-
Customers may notice that policy synchronization is delayed and agents are encountering timeouts; synchronizing agents will retry until successful.
-
Manual installation of new agents may fail to complete. Silent installation of agents will retry until successful.
Our teams are diligently working to address the service degradation. We estimate 2-3 hours to restore the policy synchronization service to full capacity.
2023-03-30T00:38:00Z
While our detailed investigation into the root cause of the service issue is ongoing, we believe we have identified the root of the issue and deployed a mitigation. We therefore have resumed the policy synchronization service of PolicyPak Cloud. We will continue to monitor and provide updates when available.
2023-03-29T19:25:00Z
PolicyPak Cloud policy synchronization with managed endpoints remains unavailable while we continue to investigate this service incident.
2023-03-29T16:45:00Z
We continue to investigate the PolicyPak Cloud service incident. At this time, policy synchronization with managed endpoints is unavailable.
2023-03-29T04:30:00Z
We are continuing to investigate the service incident with PolicyPak Cloud. At this time, policy synchronization with managed endpoints has been restored and we are continuing to monitor.
2023-03-28T13:55:00Z
We are currently investigating a service incident with PolicyPak Cloud. At this time, policy synchronization to managed endpoints is unavailable. We will post additional updates here as they become available.
Please see this PolicyPak KB article for information on how PolicyPak behaves during a cloud service outage.