HiveCPQ is experiencing issues with outgoing e-mail

Major incident HiveCPQ application
2023-09-03 18:53 UTC · 2 days, 18 hours, 25 minutes

Updates

Post-mortem

On Friday, September 1st, we noticed a significant increase in outgoing emails from the info@hivecpq.com address. We employ strict security rules and immediately blocked all outgoing email traffic until we could assess the situation further.

Since then, we have been collaborating with our email provider to identify the cause of this increase and implement measures to prevent any future security breaches in our email system. This process required thorough checks and careful investigation to ensure that no potential risks are overlooked.

We want to assure you that only emails sent from the info@hivecpq.com address were affected and were sent to random email addresses. The e-mail system cannot access any data stored within HiveCPQ.

It is important to note that no data was compromised during this incident.

Following our ISO-27001 guidelines, we always conduct an internal examination after an incident takes place. The measures we have taken so far include resetting all passwords, resetting 2 factor authentication methods, rotating api keys and increased email monitoring via Datadog (an industry standard monitoring solution).

At this moment the HiveCPQ platform is sending out emails again. Unfortunately due to this incident our email reputation dropped significantly, which caused some email providers to automatically block emails coming from HiveCPQ. This reputation will increase over time to our usual 99% and we are in active communication with big e-mail providers like GMail and Outlook to help us fixing the issue.

We highly advise all customers to setup custom domains in HiveCPQ. After completing this setup, HiveCPQ will send emails using your domain and not the hivecpq.com one. This will solve issues for customers that experience issues with blocked emails. More information on the steps required to set this up can be found on our documentation page. This feature will not incur additional costs to your subscription.

You have the option to subscribe to future updates regarding this incident by visiting the HiveCPQ status page.

If you have any questions or concerns regarding this incident, please do not hesitate to contact our support team at support@hivecpq.com. We truly appreciate your continued support, and once again, we apologise for any inconvenience this incident may have caused.

September 7, 2023 · 09:59 UTC
Resolved

Our e-mail provider resolved the issue. New and queued mails are delivered.

September 6, 2023 · 13:15 UTC
Investigating

We have made the necessary changes to allow e-mail for password rest to be sent. We are now working on enabling e-mail for the other types of mail from the HiveCPQ platform.

September 6, 2023 · 12:42 UTC
Investigating

We have implemented an alternative e-mail provider and testing out the solution.

September 6, 2023 · 04:21 UTC
Investigating

We are expecting an update from our e-mail provider to re-enable our e-mail settings and resolve the issue.

September 5, 2023 · 13:39 UTC
Update

We are expecting this issue to be resolved within the next 4 hours.

September 5, 2023 · 05:03 UTC
Update

We have identified the issue with our mail provider and are working to resolve it with them.

September 4, 2023 · 12:52 UTC
Issue

We are currently experiencing issues with our e-mail provider. We are actively looking into the issue. At this point all outgoing e-mails from HiveCPQ are not send, but held into a buffer until the issue is resolved.

September 3, 2023 · 18:53 UTC

← Back