This Service Bulletin Update is in regards to the Emergency Maintenance that was declared yesterday morning. ImageSilo engineers worked diligently to identify the issue and found it to be the result of a direct violation of the ImageSilo Acceptable Use Policy. Once identified, the issue was addressed with the appropriate parties, and we have subsequently taken measures to ensure this situation will be mitigated in the future.

We apologize for any inconvenience this performance degradation may have caused. As always, we sincerely appreciate your business.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

This Service Bulletin Update is in regards to the Emergency Maintenance that was declared earlier this morning. ImageSilo engineers have upgraded the FTP database server, which resolved the connection issues customers experienced, and are continuing to monitor the FTP service.

We apologize for any inconvenience this performance degradation may have caused. As always, we sincerely appreciate your business.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

This morning at 0600 MDT ImageSilo engineers began the process to move FTP services to AWS.  This process was completed at 1400 MDT.  All FTP services have been restored at this time.

During this move the IP address for the FTP servers was changed.  Engineers planned this to minimize or eliminate any customer impact from these changes, but in some unusual cases customers may experience issues connecting to the servers.

We apologize for any inconvenience this may have caused you.  Our goal every day is to meet or exceed your expectations and we appreciate the feedback you supply us in order for us to achieve that goal.  We greatly appreciate your business!

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

Yesterday at 17:24 MDT, ImageSilo engineers were alerted to a DNS-related issue with a non-ImageSilo system. As they began troubleshooting the issue, some of the external monitoring systems that monitor ImageSilo alerted to similar issues. Engineers quickly traced the issue to UltraDNS, our global DNS provider. UltraDNS, the largest DNS provider in the world, responsible for providing DNS services to many of the web’s most prominent sites, was victim to a DDOS (distributed denial of service) attack. UltraDNS took mitigation steps to block the attack and restored service by 17:42 MDT. During this entire time, the security of your data was never at risk and ImageSilo was operating normally. Users in certain geographical locations may not have been able to resolve the login.imagesilo.com DNS entry if their server’s copy had happened to expire during that 18 minute period of time. The fact that the TTL on that record is set to 24 hours, coupled with our belief that this was a geographical incident, leads us to believe that this affected very few customers (possibly none). UltraDNS has been stable since yesterday at 17:42 MDT and, at this time, we have no reason to expect that to change.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

The purpose of this notification is to inform you that the performance issue reported earlier have been resolved and ImageSilo has been processing at 100% of expectation since approximately 10:30 MDT.  We apologize for any inconvenience this may have caused you.  Our goal every day is to meet or exceed your expectations and we appreciate the feedback you supply us in order for us to achieve that goal.  We greatly appreciate your business!

ImageSilo Administration

A handful of customers have notified us that they have been experiencing slower than expected performance while retrieving documents on ImageSilo this morning. Whereas we work every day to meet or exceed our customers’ expectations, this Service Bulletin is to let you know that we are investigating the issue. We will update you as information becomes available. As always, we appreciate your business.

This morning (Tuesday, July 24, 2012) , at 07:39 AM MDT, ImageSilo’s database servers began rejecting database connections from the web servers. This resulted in users being unable to access ImageSilo’s services. Engineers were immediately alerted and began troubleshooting the issue. Engineers took the appropriate steps and service was fully restored at 07:51 AM MDT, 12 minutes later. Since then, engineers have been investigating the cause of the issue. Microsoft, the database software vendor, has been engaged. At this time, it appears this issue may have been caused by a bug in the database software for which there is a hot fix. However, the investigation is ongoing and it appears that there is still a significant amount of work to be done. Once completed, engineers will take the necessary steps address any underlying issues. If any of those actions will affect customers, a maintenance notification will be issued.

We apologize for any inconvenience caused by this morning’s interruption. As always, we greatly appreciate your business. Rest assured that we are as vigilant as ever to make sure we earn your business every day.

ImageSilo Administration

This Service Bulletin Update is in regards to the Emergency Maintenance that was declared earlier this morning. ImageSilo engineers have identified the root cause of the extreme system performance issues experienced this morning. The issue lies in a bug in the load-balancers that handle traffic for ImageSilo. In this bug, the on-board SSL (encryption/decryption) offloading card can stop working properly, resulting in the extremely slow performance we experienced this morning. Engineers took steps necessary to temporarily address the bug and full performance was restored by approximately 10:17 AM (MST). During this weekend’s previously scheduled maintenance, ImageSilo engineers will take steps to permanently address the bug.

We apologize for any inconvenience this performance degradation may have caused. As always, we sincerely appreciate your business.

ImageSilo Administration

This Service Bulletin Update is in regards to the network issue that occurred on September 19, 2011. That day, an overseas ISP, Saudi Telecom, began sending invalid BGP data (in this case, routes with invalid attributes). BGP is the core routing protocol used throughout the internet. Unfortunately, it has a number of known, significant shortcomings (as we saw on the 19th). One of ImageSilo’s ISPs, Level3, was amongh hundreds of ISPs throughout the world that were hit particularly hard by the event. Any customer that had a connection to Level3, regardless of having redundant connections to other ISPs, was unable to properly route traffic until Level3 was de-peered (removed as a connected ISP). While general internet outages are clearly outside of Digitech Systems’ control, in keeping with our policy of open and honest communications with our customers, we wanted to ensure our customer were fully aware of nature and wide-reaching scope of Monday’s internet challenges. This will be the final Service Bulletin Update relating to the September 19, 2011 incident.

ImageSilo Administration

SunGard continues to work torwards a resolution with multiple ISPs affecting multiple SunGard facilities around the nation. Multiple ISPs have been reporting an issue with internet routing. As for those ISPs affecting ImageSilo, SunGard de-peered from Level3 at approximately 14:30 (MDT) and, as of now (approximately seventy minutes later), our connectivity appears stable. Level3 is reporting an issue with bad routes from an overseas vendor and is taking corrective action now. We believe that the bad routes from the overseas vendor possibly “poisoned” the BGP routes on Level3 and those routes propagated to other vendors (i.e. AT&T, Verizon, etc). Needless to say, the worldwide affects of these issues with the ISPs has been far-reaching.

We will continue to monitor the situation and, once the crisis is completely resolved, work to get a more complete explaination of today’s events out to our customers. We apologize for the inconvenience today’s issues have caused. As always, we appreciate your business.

ImageSilo Administration