Update - Maintenance has not gone as planned, and engineering is waiting on VM snapshot to allow rebooting of the Virtual Machine. This process was begun at midnight and is on-going. This needs to complete to keep from risking corruption of the drive data.

Unfortunately, the means that customers on Mailstore05 are currently unable to access their email. Messages are spooling in edgewave, and will deliver when systems are back up. Until that time, however, mailboxes will be unavailable.
Jul 7, 05:26 PDT

Update - Scheduled maintenance is taking longer than anticipated. Awaiting clean VM power-off to avoid corruption. Maintenance is being extended until 4am. Customers on this mailstore will not be able to access their email during this time.

No messages will be lost, and will be delivered from the Edgewave once systems come back online.
Jul 7, ip伋理软件排行榜 PDT

Update - Engineering has continued to monitor the new equipment and email continues to function normally. They performed some additional clean up last night and this morning. We are not seeing any issues at this point. Engineering will keep monitoring.
Jul 10, 10:46 PDT

Update - Email systems are continuing to function normally.
Engineering is monitoring the system closely.
Jul 9, 08:48 PDT

Update - Load on the mail servers remains low, which means a good webmail experience for users. Mail queues remain clear. POP/IMAP blocking is still in place to facilitate both the mail move and this performance, while data remains referencing the TrueNAS hardware for customers with mailboxes on Mailstore05 only.

We are awaiting the finalization of VM snapshot consolidation. VMware process status still reads at 98%, but engineering data transfer calculations still have us on track for finalization this afternoon.
Jul 8, 极速伋理ip软件 PDT

Update - Load on the mail servers remains low, which means a good webmail experience for users. Mail queues remain clear. POP/IMAP blocking is still in place to facilitate both the mail move and this performance, while data remains referencing the TrueNAS hardware for customers with mailboxes on Mailstore05 only.

All mailbox data as made it through the VM migration to new hardware. We are awaiting the finalization of VM snapshot consolidation. Process is now at 98%.
Jul 8, 免费ip伋理软件手机版 PDT

Update - Mail servers remained running well through the evening. Load stayed low, which means a good webmail experience for mailstore05 users. Mail queues remain clear; there is no receiving delay. POP/IMAP blocking is still in place to facilitate both the mail move and this performance, while data remains referencing the TrueNAS hardware (that we are exporting from) for customers with mailboxes on Mailstore05 only.

Awaiting additional updates from Engineering on the current migration status.
Jul 8, 06:47 PDT

Update - Data migration appears to be going well. Approximately 97% of the process has completed. However, consolidating mailbox changes for smaller files, as we reach the end will take time. The last few percent are expected to be drawn out.
Jul 8, 04:13 PDT

ip伋理软件排行榜 - Server load is low and mail queues are empty. Customer webmail experience is expected to be good.

POP/IMAP blocking has caused some issues with specialty webmail jump-off pages, though direct Zimbra logins are unaffected. These are being temporarily worked around by a secondary link, directly to the mail server's web interface, 香港伋理ip软件, where a customer can log in with their full email address and password.

Migration to the new hardware is proceeding well.
Jul 7, 16:17 PDT

Update - Server load is still high while it is processing the backlog of messages from the Edgewaves . Webmail is currently performing slowly because of this, but this is expected to improve shortly as queues clear out.
Jul 7, 14:08 PDT

Update - Mailstore05 has been restored from a backup from the evening of Monday, 7/6, and is accepting email and connections at this time. This restore was completed to original hardware, so performance is expected to remain poor for the short-term. Engineering is beginning the process to migrate this virtual machine, live, to our new infrastructure. Due to the size of the storage to be moved, this is expected to take up to 24 hour to complete.

Messages that have been storing in the Edgewave spam filters are being delivered at this time.

To help facilitate the move, Engineering will be disabling POP/IMAP connections to Mailstore05. This will significantly reduce load on the system, and help get us to new hardware more quickly. Customers on Mailstore05, with an email client, will get connection timeouts from that client until this block is removed.

Customers on Mailstore05 will still be able to access their email via Webmail, and should be encouraged to do so until the migration is complete (expected by tomorrow afternoon). With the reduced load from the block of POP/IMAP connections, webmail performance is expected to be normal.
Jul 7, 13:59 PDT

无极伋理ip软件 - At this time, Mailstore05 remains down. Customers are unable to access mailboxes hosted on this machine. Email is spooling in the Edgewave, and will deliver once the system is back up.

All other mail systems are online and functioning properly at this time.
Jul 7, 10:58 PDT

Update - At this time, Mailstore05 remains down. Customers are unable to access mailboxes hosted on this machine. Email is spooling in the Edgewave, and will deliver once the system is back up.

All other mail systems are online and functioning properly at this time.
Jul 7, 07:43 PDT

Update - Server maintenance on Mailstore05 did not go as planned. Engineering began maintenance at midnight, halting the VM host OS to prepare for the CPU move, and requesting machine shutdown. However, the system would not gracefully power off. In order to avoid data corruption, engineering decided to allow the system some more time to shut down on its own, and maintenance was extended. This has continued into the morning. The VMware system continues in a state of performing snapshot clean-up, and not allowing a safe restart. Engineering is uncertain how much longer this process will take, but it could be several hours.

At this time, Mailstore05 remains down. Customers are unable to access mailboxes hosted on this machine. Email is spooling in the Edgewave, and will deliver once the system is back up.

All other mail systems are online and functioning properly at this time.
Jul 7, 05:43 PDT

Update - Mailstore load has been lowering into the evening, and is currently at low levels. Customer page load issues in webmail and timeouts are expected to be minimal into the evening. Mail queues are empty and deliver is normal at this time.

Engineering continues to take advantage of low-load times to remove non-email services from the affected network storage, to increase system performance.

Reminder: Engineering has scheduling emergency maintenance this evening as another preparatory step to get systems working as expected.
This will involve a scheduled shut-down of mailstore05 at midnight, for approximately 30-60 minutes while VM processing is migrated to new hardware. Upon successful maintenance, email storage will begin transferring to the new hardware.
Jul 6, 20:36 PDT

Update - Mailstore load has remained high through the afternoon, which is expected to present as customer page load issues in webmail. Timeouts have also been reported to be occurring frequently, even at lower load levels. Engineering continues working to offload non-mail services, while preparing replacement hardware to assist with the performance. Mail queues are staying low, though some delays of up to 5 minutes have been seen recently. Engineering continues working to keep this delivery time low.
Jul 6, 16:27 PDT

极速伋理ip - Mailstore load has continued to rise, and is high, which is expected to present as customer page load issues in webmail. Timeouts have also been reported to be occurring frequently, even at lower load levels. Engineering continues working to offload non-mail services, while preparing replacement hardware to assist with the performance. Mail queues were raising as well, causing some message delivery delay of 10-15 for customers over the past 30 minutes, and has been dealt with. Engineering continues working to keep this delivery time lower.

Engineering will be scheduling emergency maintenance this evening as another preparatory step to get systems working as expected.
This will involve a scheduled shut-down of mailstore05 at midnight, for approximately 30-60 minutes while VM processing is migrated to new hardware. This isn't full migration of the affected mailstore, but part of that process.
Jul 6, 14:48 PDT

Update - Mailstore load has remained moderate to high, which is expected to present as customer page load issues in webmail. Timeouts are expected infrequently, if at all. Mail queues were raising as well, causing some message delivery delay of 10-30 for some customers over the past hour. Engineering has been able to mitigate this, again, and message delivery is expected as normal at this time.

Engineering is continuing to monitor load, adjust settings, to help keep performance up.
Jul 6, 12:36 PDT

Update - Mailstore load continues to rise as anticipated, and is moderate to high, which will present as customer load issues at his time. Mail queues are raising as well, causing some message delivery delay. Engineering is working to mitigate these delays.

Engineering is continuing to monitor load, adjust settings, to help keep performance up.
Jul 6, 极速伋理ip PDT

Update - Mailstore load remaine moderate to low through the evening. Mail queues remain low, and messages are delivering in a timely manner.

Engineering is continuing to monitor load, adjust settings as we move into expected heavier-use portions of the day. Mailbox distribution has been adjusted during previous low-load times so that customers are spread out across additional mailstores.
Jul 6, ip伋理软件排行 PDT

Update - Mailstore load remains moderate at this time. Mail queues remain low, and messages are delivering in a timely manner.

Engineering is continuing to monitor load, adjust settings, and offload available services to help keep performance up.
Jul 5, 18:55 PDT

Update - Mailstore load remains moderate at this time. Mail queues remain low, and messages are delivering in a timely manner.

Engineering is continuing to monitor load, adjust settings, and offload available services to help keep performance up.
Jul 5, 伋理ip软件 PDT

Update - Mailstore load is running moderate at this time. Customers with mailboxes on this system will still be experiencing intermittent performance issues. Mail queues remain low, and messages are delivering in a timely manner.

Engineering is continuing to monitor load, adjust settings, and offload available services to help keep performance up.
Jul 5, 11:54 PDT

Update - Mailstore load remains low at this time, and systems are continuing to perform at good levels. Mail queues are delivering in a timely manner. Engineering is monitoring the load and continues to make adjustments to further improve and support mailstore performance.

Additional system modifications are still being prepared for a full move to new hardware as a permanent fix.
Jul 4, 16:36 PDT

Update - Mailstore load is low at this time, and systems are running at a acceptable levels. Mail queues are delivering in a timely manner. Engineering is monitoring the load. Adjustments appear to be assisting performance, but more time is needed to confirm.

Additional system modifications are still being prepared for a full move to new hardware as a permanent fix.
Jul 4, 13:59 PDT

极速伋理ip - Mailstore load is running moderate at this time. Customers with mailboxes on this system will still be experiencing intermittent performance issues. Mail queues remain low, and messages are delivering in a timely manner.
Jul 4, 11:29 PDT

Update - Mailstore load is running moderate to heavy at this point. Customers with mailboxes on this system will still be experience performance issues. Mail queues remain low, and messages are delivering in a timely manner.
Jul 4, 08:51 PDT

Update - Mailstore load has been much lower than anticipated throughout the evening and early morning. Per vendor recommendation, while reviewing the Virtual Machine failure, VM snapshot removal was begun last evening. This is expected to be a long process, but has been progressing quite well. This adjustment may be related to the increased performance of the machine.

When this process is complete, they will begin the transfer to new hardware once again, after making some further adjustments to the configuration per VM vendor recommendations.

The mailstores are operating as expected for normal operation at this time, which is good. Engineering is monitoring them as access demand rises into the AM hours.
Jul 4, 05:43 PDT

Update - Mailstore05 is back online. This is the original machine, not the new hardware. Performance issues are expected to remain at this time, and are being monitored along with system health after the downtime. Spooled messages from the edgewaves will start delivering.

Engineering will retry the move to new hardware after reviewing the failure.
Jul 3, 19:12 PDT

Update - Engineers are still working with our vendor to restore services to mailstore05
Jul 3, 免费ip伋理软件手机版 PDT

Update - Engineers are still on the line with our vendor, and expect mailstore05 back up within the next 30 minutes.
Jul 3, 手机伋理ip软件 PDT

Update - The mailstore05 machine is erroring with "failed to lock." We are in contact with our vendor so that it can be properly restarted. Customers on this mailstore will be unable to access their email during this time. Estimated timeframe is an hour or so. Incoming customer email will spool in the Edgewaves during this time and be delivered once the server is back up and running. All over mailstores are functioning normally.
Jul 3, 15:42 PDT

ip伋理软件排行 - Email copying to the new hardware was going well, but then threw a "general error" and the system failed. Mailstore05 was unavailable for about 20 minutes, but is coming back online now. This is still the existing system, not the new hardware. Performance issues are expected to remain the same while engineering works to re-do the mailstore copy.
Jul 3, 15:34 PDT

Update - Mailstore load continues to be high, which means continued performance issues for those with mailboxes on that server. Mail queues have been rising frequently during the day, but engineers have been manually managing these to help make sure that delivery times don't escalate too far. Transfer of affected mailstore to new hardware is 80% complete, which will provide additional performance benefits. Engineering continues to evaluate and implement additional actions to bring the system load down to expected normal levels.
Jul 3, 13:03 PDT

手机伋理ip软件 - The mail delivery queue was emptied again so all mail has been delivered. Mailstore load continues to remain at moderate to high levels. Engineers are working to keep mail delivered in a timely manner. The migration of the poor performing mailstore to new hardware is making progress.
Jul 3, 无极伋理ip软件 PDT

无极伋理ip软件 - Mailstore load remains moderate to high. Users on this system continue to experience performance issues. Mail queues slightly elevated at times; engineering is working to keep them clear for timely mail delivery.
Jul 3, 08:31 PDT

Update - Mail delivery queues were emptied earlier this morning but for unknown reasons are climbing back up. Engineering continues to monitor performance. The migration of the poor performing mailstore to new hardware is making progress.
Jul 3, 免费ip伋理软件手机版 PDT

香港伋理ip软件 - System load remained moderate throughout the evening and morning. Mail delivery queues began building for as yet unknown reason in the wee hours but are currently draining. Engineering continues to monitor performance. Migration of the poor performing mailstore to new hardware is approximately 2/3 complete.
Jul 3, 06:30 PDT

免费ip伋理软件手机版 - Mail queues remain empty, and average mailstore load is remaining moderate, with the occasional activity spike. Migration to new storage hardware is progressing much faster than anticipated.
Jul 2, 20:03 PDT

Update - Mail system performance continues to improve overall. Mail queue delays have remained minimal throughout the bulk of the day, aside from elevation this morning. Average load on the problematic mailstore has stayed lower, where customers can access and navigate, however fluctuations at various intervals are still manifesting to users as intermittent slowness or timeouts. Engineers continue to look for other services to migrate from this storage array to keep the system stable for customer use.

In tandem, for permanent performance correction, engineers have been installing new hardware and are now beginning to migrate the mailstore away from the TrueNAS network storage device that is performing poorly.
Jul 2, 16:58 PDT

ip伋理软件排行榜 - Mailstore load continues to fluctuate, and is averaging high. This continues to mean that customers on the affected mailstore is experiencing performance issues. Engineering is continuing to work to keep this load as low and stable as possible. Mail delivery queues have not increased since earlier, and mail delivery should be timely.
Jul 2, 14:06 PDT

免费ip伋理软件手机版 - Storage array remains under heavy load after restoring POP/IMAP sessions, however, the delivery queue build-up was cleared quickly. Delivery queue is currently remaining steady, and engineering is watching for increased build-up.
Jul 2, 11:54 PDT

极速伋理ip软件 - Mailstore load remains fluctuating heavily. This means continued performance issues for customers on the affected mailstore and intermittent connectivity issues when talking to the mail server (timeouts). Mail queue size increased, then began reducing, but had started creeping up again.. Engineering is briefly pausing IMAP/POP access to mailstore05 to allow the queue to drain. This will not affect webmail access. This will allow messages to get delivered to users. Queue depth should stay fairly low after this, as the system is keeping up in general.

Engineering continues migrating non-mail systems to give us more performance. This has greatly helped keep the delivery queues stable, but has not returned system performance to normal status.
Jul 2, 11:32 PDT

Update - Mailstore load is fluctuating heavily, and averaging higher. This means continued performance issues for customers on the affected mailstore and intermittent connectivity issues when talking to the mail server (timeouts). This increases load is beginning to affect the incoming mail queues, and they are starting to rise slowly. This means moderate delays, at this time, for incoming messages to get delivered to mailboxes.
Jul 2, 09:37 PDT

Update - Mail queues have remained low into the morning, which means means standard email delivery times. However, load on the storage system has increased significantly in the last 15 minutes, but is lowering again. This level remains higher than optimal, and continues to mean slowness and timeouts for customers whose mailboxes are housed on that mailstore. Engineering continues to monitor the load, adjusting settings and migrating services away from that hardware to increase performance.
Jul 2, 08:33 PDT

Update - Email delivery queues remained empty throughout the evening, as expected. Overall system load on the affected mailstore is still elevated and Webmail navigation and access will present as slow/sluggish still, but available. Additional migration of services away from the network storage device have continued throughout the evening. Engineering will be monitoring system load as we move into the morning hours.
Jul 2, ip伋理软件排行 PDT

Update - Mail queue delays continue to get lower, and are getting close to empty. There is not need to attempt to disable POP/IMAP again on the affected mailstore at this point to assist with the processing. System load is lower, but still elevated. Webmail navigation and access will likely present as slow/sluggish still. Additional services have been offloaded from the mailstore storage array, which is continuing to assist in the recovery.
Jul 1, 伋理ip软件 PDT

Update - Good progress on mail queue delays. They aren't empty yet, but they're continue to lower. The short disabling of POP/IMAP to the poor performance mailstore was beneficial in getting this lower, quickly, but those services have since been re-enabled. (If queue backlog doesn't continue to go down at the expected rate, this tactic may be utilized again later in the evening, if needed.)

Engineering continues to move services from the network storage device, to be able to continue offering more and more performance, and get us back to stable conditions.

Load still remains high, and mailboxes on this mailstore continue to have performance issues. Message delays for other users is lessening.
Jul 1, 16:09 PDT

Update - Efforts to increase resources are going well. Loads and message queues remain high, causing customer problems. However, with the additional resources (only part of the ongoing freeing-up), the queue has been reducing over the past hour. In an effort to reduce queue backlog, we are briefly suspending IMAP/POP for mailstore05 users. Webmail access remains. This will be a short duration, to help reduce load at a faster rate.
Jul 1, ip伋理软件排行榜 PDT

Update - No change in status.
Loads on the mail system continue to be high. Customers are continuing to experience delays or timeouts when logging into or navigating webmail for on the problematic mailstore. Performance issues are leading to longer delays in email delivery for all customers, as mail queues build. Email is not being lost, but is holding in queue for delivery, until that queue can be successfully processed by the system.

Engineers are continuing to offload services from the affected equipment. This is going well, but is taking time. Doing so does come with additional performance impact to the system, but this is temporary, and will help the overall load as a whole.
Jul 1, 12:51 PDT

Update - Loads on the mail system continue to rise. Customers are continuing to experience delays or timeouts when logging into or navigating webmail for on the problematic mailstore. Performance issues are leading to longer delays in email delivery for all customers, as mail queues build. Email is not being lost, but is holding in queue for delivery, until that queue can be successfully processed by the system.

Engineers are offloading services from the affected equipment, to give it extra resources to process mail. Doing so does come with additional performance impact to the system, but this is temporary, and will help the overall load as a whole.
Jul 1, 10:24 PDT

Update - Loads are rising and affecting performance. We're seeing delays or timeouts when logging into or navigating webmail for users on the problematic mailstore. Performance issues are leading to longer delays in email delivery, as mail queues build. Engineers are continuing to work to reduce load.
Jul 1, 08:33 PDT

Update - Engineering is monitoring the email systems. Load is rising on system, as expected as mailboxes start getting used more in the morning. Slow access times for the offending mailstores are still present. Email is flowing, and there remains minimal delay to delivery/sending. Engineering is continuing to work on reducing load for more stable performance.
Jul 1, 无极伋理ip软件 PDT

Update - At approximately 8pm, email backlog was successfully cleared out, due to some limiting of pop/imap connections on the problematic mailstore (used to reduce system load, so that it could better process and catch up). These connections were allowed again. Load on the system immediately rose, but has continued to remain stable, though elevated, throughout the night. This has meant that customers have been able to successfully connect, and there have not been significant delay in email sending.

This is lower-use time, and engineering continues to monitor the effects, as with normal customer usage, load will increase into the morning.
Jul 1, 04:55 PDT

Update - Engineers continue to work to alleviate system load. Backlogged emails have stayed steady since the afternoon. System handling and database tweaks have been made, trying to help the system catch up through the deliveries. This process continues. Customers on the problematic mailstore continue to have intermittent connectivity issues and webmail slowness. All customers are experiencing delays due to the mail queue backlogs.
Jun 30, 16:56 PDT

Update - Some modifications made by engineering appear to have been effective. We continue to work through mail delivery backlogs, though that amount is still high.
Jun 30, 13:30 PDT

Update - Engineering continues to work to find a resolution for affected system. The underlying issue is performance issues with the storage array that houses a significantly affected mailstore. Because of the performance issues on this machine, subsequent problems are appearing that are affecting customers who are not on that set of hardware. Primarily, this is manifesting as significant sending and receiving delays to customers using our email servers.

Engineering continues to work to determine the reason for the storage array performance issues, and is working to reduce load on their own and with our vendor to come up with permanent fix
Jun 30, 10:57 PDT

Update - Status unchanged from last update. Engineering continues to work on mailstore performance issues. Customer email is accessible, but performance of some systems remains below expectations, resulting in continued slow access for those mailboxes affected.

Engineering is working to move mailboxes to other servers to help alleviate loading on problem server.
Jun 30, 07:28 PDT

Update - Engineering continues to work on mailstore performance issues. Customer email is accessible, but performance of some systems remains below expectations, resulting in continued slow access for those mailboxes affected.
Jun 30, 04:50 PDT

Update - Engineering is continuing to work on the issues. Heavy load continues to affect one of the mailstores and customer email boxes on that server.
Jun 29, 无极伋理ip软件 PDT

Update - Underlying issues are with our storage array. Attempts to mitigate the affect this has been having on customers has been unsuccessful. This device in its entirety has been rebooted and is back up. Engineering continues to review performance of the systems from this change.
Jun 29, 17:42 PDT

Update - Engineers are doing an emergency reboot of some key processes to assist with the ongoing mailstore issues. Customers that have been experiencing slowness will be unable to connect to email during this time (some were already experiencing this intermittently). Downtime for the affected mailstores will be between 5 and 30 minutes, depending upon the reboot process.
Jun 29, 14:58 PDT

Update - Load is lowering, but affected mailstores are still under extremely high load as they work to process email and connection requests. This continues to manifest as slowness and timeouts for customers on the affected devices. Engineering continues to work on additional methods to assist performance. Email pre-harvest is being temporarily suspended as a whole to assist with performance while we deal with the unanticipated system performance hit.
Jun 29, 11:53 PDT

Update - Pre-migration harvesting has been backed off to improve overall system performance. Engineering continues to monitor the situation.
Jun 29, 09:57 PDT