ImageSilo Service Bulletin Update

This Service Bulletin Update is in regards to the Emergency Maintenance that was declared earlier this morning. ImageSilo engineers have identified the root cause of the extreme system performance issues experienced this morning. The issue lies in a bug in the load-balancers that handle traffic for ImageSilo. In this bug, the on-board SSL (encryption/decryption) offloading card can stop working properly, resulting in the extremely slow performance we experienced this morning. Engineers took steps necessary to temporarily address the bug and full performance was restored by approximately 10:17 AM (MST). During this weekend’s previously scheduled maintenance, ImageSilo engineers will take steps to permanently address the bug.

We apologize for any inconvenience this performance degradation may have caused. As always, we sincerely appreciate your business.

ImageSilo Administration

ImageSilo Emergency Maintenance Notification – Jan. 20, 2012

Start Date: January 20, 2012 (Friday)
Start Time: 10:06 (MST)
Finish Date/Time: Not yet determined

Purpose: ImageSilo engineers are investigating a performance issue that is causing extremely sluggish performance of the ImageSilo system.

Customer Impact: At this time, the issue appears to be specifically limited to SSL traffic (not the web or application servers themselves). We are engaging vendors to assist in troubleshooting and addressing the problem. We anticipate failing over firewalls and load-balancers as part of those troubleshooting steps. Customer impact will be unknown. We will post another update as more information becomes available.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

ImageSilo Maintenance Notification – Jan. 20, 2012

Start Date: January 20, 2012 (Friday)
Start Time: 20:00 (MST)
Finish Date: January 23, 2012 (Monday)
Finish Time: 04:00 (MST)

Purpose: Apply security updates and patches to multiple ImageSilo systems.

Customer Impact: ImageSilo engineers will be applying security updates and patches to a number of ImageSilo systems. Expected impact is approximately four periods of high latency (up to two minutes each) as domain authentication fails over to redundant hosts, a few FTP session interruptions (which should automatically re-connect if using Data Transfer Manager), and two periods of limited access/interruption (up to five minutes each) for database fail-overs.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

ImageSilo Maintenance Notification – Dec. 2, 2011

Start Date: December 2, 2011 (Friday)
Start Time: 20:00 (MST)
Finish Date: December 5, 2011 (Monday)
Finish Time: 04:00 (MST)

Purpose: ImageSilo Engineers will be performing general maintenance on load balancers, firewalls, and storage systems. Additionally, an upstream internet service provider will be upgrading core router software.

Customer Impact: The following activities will be performed during this maintenance:

  • Maintenance on load balancers that will require multiple reboots: no customer impact is expected.
  • Maintenance on firewalls that will require multiple reboots: customer impact is expected to be limited to two periods of short (less than one minute) network interuption.
  • Maintenance on storage systems to modify mirroring configuration: no customer impact is expected.
  • Upstream internet service provider upgrading core routers (Sunday): customer impact is expected to be limited to a two minute period of higher than usual latency.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

ImageSilo Emergency Maintenance Notification – Nov. 3, 2011

Start Date: November 3, 2011 (Thursday)
Start Time: 20:00 (MDT)
Finish Date: November 4, 2011 (Friday)
Finish Time: 08:00 (MDT)

Purpose: An upstream network provider must depeer a redundant ISP connection on a local internet router to reduce the service effect of carrier vendor maintenance.

Customer Impact: Some customers may experience minor latency while traffic converges during the maintenance period.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

 

ImageSilo Maintenance Notification – Oct. 1, 2011

Start Date: October 1, 2011 (Saturday)
Start Time: 08:00 (MDT)
Finish Date: October 2, 2011 (Sunday)
Finish Time: 17:00 (MDT)

Purpose: Replace network equipment outside the ImageSilo network.

Customer Impact: ImageSilo engineers will be replacing switches and other networking equipment outside of the ImageSilo network. None of the changes are within the ImageSilo network and no customer impace is expected.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

ImageSilo Service Bulletin Update

This Service Bulletin Update is in regards to the network issue that occurred on September 19, 2011. That day, an overseas ISP, Saudi Telecom, began sending invalid BGP data (in this case, routes with invalid attributes). BGP is the core routing protocol used throughout the internet. Unfortunately, it has a number of known, significant shortcomings (as we saw on the 19th). One of ImageSilo’s ISPs, Level3, was amongh hundreds of ISPs throughout the world that were hit particularly hard by the event. Any customer that had a connection to Level3, regardless of having redundant connections to other ISPs, was unable to properly route traffic until Level3 was de-peered (removed as a connected ISP). While general internet outages are clearly outside of Digitech Systems’ control, in keeping with our policy of open and honest communications with our customers, we wanted to ensure our customer were fully aware of nature and wide-reaching scope of Monday’s internet challenges. This will be the final Service Bulletin Update relating to the September 19, 2011 incident.

ImageSilo Administration

ImageSilo Service Bulletin Update

SunGard continues to work torwards a resolution with multiple ISPs affecting multiple SunGard facilities around the nation. Multiple ISPs have been reporting an issue with internet routing. As for those ISPs affecting ImageSilo, SunGard de-peered from Level3 at approximately 14:30 (MDT) and, as of now (approximately seventy minutes later), our connectivity appears stable. Level3 is reporting an issue with bad routes from an overseas vendor and is taking corrective action now. We believe that the bad routes from the overseas vendor possibly “poisoned” the BGP routes on Level3 and those routes propagated to other vendors (i.e. AT&T, Verizon, etc). Needless to say, the worldwide affects of these issues with the ISPs has been far-reaching.

We will continue to monitor the situation and, once the crisis is completely resolved, work to get a more complete explaination of today’s events out to our customers. We apologize for the inconvenience today’s issues have caused. As always, we appreciate your business.

ImageSilo Administration

ImageSilo Service Bulletin Update

ImageSilo continues to experience network connectivity issues. Numerous SunGard facilities around the nation are being impacted by this network event. ImageSilo engineers have identified what they believe to be an issue with Level3’s network and have been attempting to pass this along to the numerous high-level SunGard engineers that are working the issue in order to get Level3 de-peered from the SunGard network. Another notification will be sent out when further information is available.

We apologize for length of time it took to get the initial ImageSilo Service Bulletin out earlier as the same intermittent network issues affecting ImageSilo were separately impacting our ability to update the notification site and send notification emails.

ImageSilo Administration

ImageSilo Service Bulletin

ImageSilo is currently experiencing technical difficulty. Our engineers are currently working to resolve the issue and we expect ImageSilo to be fully restored shortly. Please do not call support or ImageSilo Administration regarding this service bulletin as all personnel are working to resolve the issue. A Service Bulletin Update will be posted as additional information becomes available. Thank you for your patience.

ImageSilo Administration