ImageSilo continues to experience network connectivity issues. Numerous SunGard facilities around the nation are being impacted by this network event. ImageSilo engineers have identified what they believe to be an issue with Level3’s network and have been attempting to pass this along to the numerous high-level SunGard engineers that are working the issue in order to get Level3 de-peered from the SunGard network. Another notification will be sent out when further information is available.

We apologize for length of time it took to get the initial ImageSilo Service Bulletin out earlier as the same intermittent network issues affecting ImageSilo were separately impacting our ability to update the notification site and send notification emails.

ImageSilo Administration

ImageSilo is currently experiencing technical difficulty. Our engineers are currently working to resolve the issue and we expect ImageSilo to be fully restored shortly. Please do not call support or ImageSilo Administration regarding this service bulletin as all personnel are working to resolve the issue. A Service Bulletin Update will be posted as additional information becomes available. Thank you for your patience.

ImageSilo Administration

On Friday, June 24, 2011 (at approximately 14:06 MDT), some ImageSilo users may have received an error stating that “The disk is full” when attempting view documents. We do not believe that this affected all ImageSilo users that were actively using the system at the time. The issue was caused by an extremely abnormal usage pattern that caused the storage for the cache location (where documents are temporarily cached to provide secure access to documents without exposing the entire document store to the internet) to be quickly consumed and not released. Engineers were alerted by our internal monitoring systems. By14:22 MDT (18 minutes later), the issue was fully rectified.

ImageSilo engineers have taken steps to ensure this issue doesn’t reoccur in the future, including increasing the cache size exponentially. We sincerely appreciate your business and apologize for any inconvenience this may have cause those who experienced the error.

ImageSilo Administration

This Service Bulletin Update is in regards to the Emergency Maintenance that was declared earlier this afternoon (on March 24, 2011). After thorough investigation and discussion with the load balancing hardware vendor, ImageSilo engineers were able to rectify the issue. Engineers will continue to closely monitor the systems.

ImageSilo Administration

 This Service Bulletin Update is in regards to the Emergency Maintenance that was declared on March 3, 2011 and continued on March 7, 2011. Last night, ImageSilo engineers applied the suggested Microsoft hotfix to the database clusters in order to address the performance-related bug. They have been monitoring the performance of the systems since that time. Based on data gathered since the hotfix application, the performance issue appears to be resolved. Engineers will continue to closely monitor the systems.

ImageSilo Administration

This Service Bulletin Update is in regards to the Emergency Maintenance that was declared on March 3, 2011 and continued on March 7, 2011. ImageSilo engineers have been working continuously with Microsoft to identify the cause of the performance issues related to ImageSilo’s database clusters. After evaluating a significant amount of data, Microsoft has made two recommendations – (1) make a change to a setting to allow SQL Server’s cache plan to be optimized for ad hoc queries, and (2) apply a hotfix to address a bug related to the performance issues (the hotfix was released a couple weeks ago, but has not made it into standard updates).

The change to optimize the cache plan was made at approximately 11:50 MST. Immediately following the change, there was an expected further degradation of performance (while the cache plan was flushed and rebuilt), which lasted about 15 minutes. Afterwards, performance returned to normal and currently appears to remain good. Engineers will continue to monitor performance going forward.

This evening, at 20:00 MST, ImageSilo engineers will be applying the suggested hotfix to fix the associated performance-related bug. This will require failing over the database clusters. As such, two short periods of inaccessibility (up to 5 minutes each) are expected this evening during the failover.

Questions/Comments: Please feel free to contact ImageSilo Administration with any questions or comments.
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

This Service Bulletin Update is in regards to the Emergency Maintenance that was declared this afternoon (January 26, 2011). ImageSilo engineers have taken measures to address the issues identified earlier with the network switching infrastructure. One of those steps required hard restarts of all parts of the network equipment, including all redundant equipment, simultaneously. At this time, the equipment is not exhibiting any of the previously identified symptoms. ImageSilo engineers will continue to closely monitor the systems while working with the hardward vendor to identify the root cause of the issue and take preventative measures to prevent any future occurence.

If you have any questions or comments, feel free to contact ImageSilo Administration.

ImageSilo Administration
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

 This is an update pertaining to the ImageSilo Service Bulletin issued the evening of Feb 5, 2010 and the subsequent Emergency Maintenance opened the following morning. Full service and full redundancy has been restored and operational since yesterday. Two separate issues came to light: (1) a fairly seldom seen Microsoft SQL Server bug and (2) a hardware failure with fibre equipment linking a backup SQL server to its database storage array. Below is a more detailed outline of the events of Feb 5 and 6:

On Feb 5, 2010, at 7:38 PM MST ImageSilo’s internal monitoring systems alerted engineers of an issue with a SQL database cluster. Initially (as communicated in the Feb 5 Service Bulletin) , it was thought that this issue had affected all ImageSilo customers. Instead, it turned out that the issue had affected only a single customer’s database. However, by 7:44 PM MST, the issue began affecting other databases. ImageSilo engineers immediately made the decision to manually fail the database cluster over to its redundant partner node. As a result of this issue and the resultant failover (which took considerably longer than expected), access interuption for all customers began at 7:45 PM MST and service was restored by 7:51 PM MST (except for the single customer database that was still being affected).

Engineers worked through the night, enlisting the help of escalated Microsoft support, to determine the cause of the problem. By early Saturday morning (Feb 6), it was determined that the cause of the initial problem was a well known (within Microsoft) but seldom seen bug in SQL Server. An Emergency Maintenance Notification was issued to provide the oppportunity to apply a Microsoft-supplied patch to address the bug, as well as to identify and address a new issue that appeared on the database node that was carrying the problematic node’s load (which was also the cause of the extended period of time taken for the cluster failover the previous evening).

By 11:57 AM MST (Feb 6), engineers had identified that the current database node was experiencing an issue with one of the fibre links connecting it to its database storage array. The hardware vendor immediatley dispatched parts to replace the failed/failing equipment and the bad link was fully restored and tested by 2:02 PM MST. Engineers then went back to work to apply the Microsoft patch to address the SQL Server bug. By 8:59 PM MST the patch had been installed on all cluster nodes and multiple tests had been performed to ensure that the software and hardware was operating as expected.

As of the time of this writing (1:34 PM MST, Feb 7), the issues appear to have been completely addressed by the patch applied and the replacement of the failed hardware. If the status should change, another Service Bulletin Update will be posted. Otherwise this issue is considered closed.

As always, we appreciate your continued support and look forward to continuing to provide you with the high level of service you’ve come to enjoy as an ImageSilo customer. If you have any questions or comments, feel free to contact ImageSilo Administration.

ImageSilo Administration
866.374.3569 (303.493.6900)
siloadmin@imagesilo.com

This evening (Feb 5, 2010), at  7:38 PM MST, ImageSilo encountered a database issue that resulted in an interuption of service. ImageSilo engineers immediately responded to the issue and forced a failover to backup services. Backup service was fully operational and restored by 7:51 PM. ImageSilo engineers will continue working to identify and address the cause of the database issue (as well as the cause of a lack of automatic failover). A Service Bulletin Update will be posted as additional information becomes available. Thank you for your patience.

ImageSilo Administration