Dear customers,
Regarding recent connection problems that started on Tuesday we would first like to say that everything is now fully working again. We noticed the issues just before noon on Tuesday and directly put all efforts towards solving the issue. More or less the entire day from that point was spent trying to troubleshoot and resolve this.
Our servers are located on several data centers to minimize the impact if one data center is experiencing problems. Unfortunately, this time, the monitoring systems did not detect any problems with the servers which is also the reason that it took some time to make the connection to that specific data center. Late Tuesday night we moved our servers to another data center which made the service available again.
On Wednesday we received a call from our hosting provider who takes full responsibility for the issues. During a network update in one of their data centers some ports were blocked, which affected the traffic to our servers. Only a few of their customers were affected, unfortunately Telldus was one of them. It took a while for them to locate the network issue and resolve it and they are still in the process of analyzing the reason behind it. After that process they will take measures to change routines to avoid this kind of issues in the future.
Availability is top priority for us and we are continuously working on solutions to improve that even further. Specifically in this case we will investigate why our monitoring systems did not detect this particular issue.
This is a quote from our hosting provider (in Swedish):
"I samband med en uppdatering av nätverksfunktionalitet i ett av våra datacenter så blockerades trafik på en standard port för virtuella nätverk. I vår riskbedömning inför uppdateringen hade vi inte noterat att det här var en känd risk, vi kan inte se någon rimlig anledning till varför den här spärren existerar så hanterar vi det som en bugg i mjukvaran och arbetar med vår leverantör för att reda ut det. Analysen kring varför det här hände är inte klar så kan vi inte dra några större slutsatser ännu. Felet var svårt att lokalisera eftersom det drabbade ett fåtal kunder. Vi jobbar på att reda ut hur vi kan förändra våra rutiner för att kunna identifiera liknande problem snabbare i framtiden."
