Shiny Mix - Second Life® Information Overload » Tags » SLGrid

Hi everyone, another server deploy is upon us!  The schedule for the rolling restart is thus:

• 2010-02-10, 7a - 12p: Pilot Regions[1]
• 2010-02-16, 7a - 12p: Even numbered sims
• 2010-02-17, 7a - 12p: Odd numbered sims

Regions should be down no more than 30 minutes.  If yours is down longer than that, please leave a comment with the region name and we’ll look into it.  This thread will be updated with the status of the deploy when it begins.

For more information, comments, and updates, visit the Discussion Page.

[12:02 PM PST] This issue has been resolved and inworld activities can resume

[11:55 AM PST] We are currently experiencing problems with possible tp failures and general inworld services. Please refrain from rezzing no copy items and and making purchases. We are investigating the problem and will post updates here.

We’re installing a few upgrades and restarting PJIRA beginning at 1:00 a.m. Pacific tomorrow morning.

Even if PJIRA seems accessible during this time, you may lose edits. Please refrain from updating any PJIRA pages during the maintenance window.

If the scope or scheduling of this work changes, we’ll update you here.

[6 February 2010 - 12:05am PST] The issues have been resolved and are working. Thank you for your patience.

[5 February 2010 - 11:25pm PST] Forums.secondlife.com is having intermittent issues. We are working to resolve the problem. Please watch this blog for further updates.

JIRA and Wiki are both back to normal. Thank you for your patience.

[5 February 2010 - 2am PST] [UPDATE] Wiki is currently unavailable for maintenance also.

[4 February 2010 - 11:50pm] JIRA are currently unavailable. We are working on the issue now. Please watch this blog for further details.

[Resolved 5:15pm PST] We have restored normal working order to all in-world services. Thank you for you patience and understanding.

[Posted 4:37pm PST] We are currently experiencing some issues related to in-world services. We’re tracking down the source as rapidly as possible. Please do not attempt L$ operations or manipulate no-copy items until we issue an all-clear. We will update this post as we know more.

[Resolved @ 8:09 am] This issue has been resolved and logins have been restored. Once again, thank you for your understanding.

There have been a few reports of Xstreet experiencing some difficulties in regards to logins. We are currently investigating the situation. Thank you for your patience while we work towards a resolution.

[Resolved] Monday, February 8th 6.16am This has now been resolved and all services are running as normal. If you have any issues please contact Support for further help.

Monday, February 8 2010 at 5:00 a.m. Pacific, we’ll be closing logins and disabling major inworld services for ninety minutes of upgrade work on our databases.

During the maintenance, logins, account registrations, XStreetSL, and the land store will be unavailable.  The Knowledge Base will remain accessible.

Residents who are in world will not be disconnected when maintenance begins, but will lose access to services such as

mapping,
teleport,
inventory,
object and L$ transfers, and
LindeX.

We’ll warn in world several minutes before the outage begins, and we’ll post any changes to the scope and schedule of the work here as they develop.

We appreciate that this will inconvenience a number of our Residents. The maintenance work is to enhance the stability and performance of our database-backed services.

[Resolved 7:33pm PST] We have resolved the issue with this small subset of regions. If you are aware of a region that is still not back online, please contact support.

[Feb 2 6:20pm PST] An extremely small set of regions are having issues coming back online when being restarted. We are working on resolving the issue. If you are aware of a region that has gone offline and has not come back online within 30 minutes, please contact support.

[Resolved Feb. 3 @ 5:17 am] All stipends should have now paid out. Thank you for your patience while we finished this process.

[Updated 7:38pm PST] Stipends are still in the process of paying out. If all goes accordingly, the next update will be when the stipends have successfully finished being paid. Please watch this post for updates.

[Posted 1:30pm PST] Due to the Inworld service disruption this morning, stipend payments have been delayed. We are currently in the process of paying out all due stipends and the process should finish over the next few hours. We will update this post if anything changes and when the stipends have finished paying out.

[6:15 a.m. Pacific] Services have been restored inworld and transactions with L$ and no-copy objects are safe.

[5:36 a.m. Pacific] We’ve cleared a cluster of issues, but there’s one slowdown left in our database. We hope to issue an all-clear very shortly.

[4:50 a.m. Pacific] Inworld services have been disrupted for a number of Residents. We’re assessing as rapidly as possible. Please do not attempt L$ operations or manipulate no-copy items until we issue an all-clear.

We’ll keep you posted.

[RESOLVED] 5:22pm PST  -  This issue have been corrected and resolved.

We are currently working on some database issues that have affected teleports.  As a precaution, please refrain from inworld transactions or rezzing objects.  We will update this post.

As we did last week, we’ll be performing up to three hours of network maintenance tomorrow, 2 Feb, beginning at 6 a.m. Pacific. Residents may experience isolated cases of TP and transaction failure.

We appreciate your patience during this necessary work.

[Resolved] 1/29/2010 2:03 pm PST  The issue with the regAPI has been resolved.

1/29/2010 12:35 pm PST
Some residents have been experiencing failures with the Second Life regAPI for certain requests.  We are working on a resolution right now. Please check back to this blog for updates.

[RESOLVED] 6:51 pm pst, 1/28/2010.  The Second Life Jira website (https://jira.secondlife.com) is back online.

The Second Life Jira website (https://jira.secondlife.com) is down.  We are working on a fix to get it back up as soon as possible.  Please check back to this blog for status updates.

[28 Jan 2010 - 3:10am PST] Issues with voice have been resolved. If you are experiencing problems, please relog.

[28 Jan 2010 - 2:50am PST]  We are currently experiencing voice issues in Second Life. It appears to be affecting a small number of our residents and we are working with the engineers to correct the problem Please keep checking back here for updates.

The Second Life Auction website ( http://usd.auctions.secondlife.com/ ) will be down for one hour Wednesday January 27th, 2010 between 8:00pm and 9:00pm PST for maintenance.  Please watch this blog for further status updates.

[24 January 2010 8:30PM Pacific] There are currently technical issues with Group Payouts which will be delayed. Our teams are working on resolving this issue as soon as possible. Please watch this blog for any further details.

[RESOLVED 11:00AM PST January 23 2010] We have successfully completed Server 1.34.3 deployment, we thank you for your patience.

[8:50pm Pacific, January 22 2010] Due to a potent sim-crashing bug, Server 1.34.3 will be deployed tonight / tomorrow.  The roll will begin around 21:00 Pacific.

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

[22nd January 2010 1.54AM PST] The landstore is back online and is available to be used. If you notice any errors please contact Customer Support. Thank you.

[21 January 2010 10:30PM PST] We are currently experiencing issues with the Landstore, and will be unavailable until further notice. We will post more details as they become available; check back for updates.

[21 January 2010, 11PM Pacific] [RESOLVED] Hardware issues have now been fixed and functioning fully. Thank you for your patience.

[9:35pm Pacific] We’re currently experiencing some hardware issues that will affect some residents. Symptoms include inability to log in, and inability to access inventory or rez objects while inworld. We’re working to restore full functionality as soon as possible and will post here when we have more information.

[11:00 AM PST]  Monday, February 8, 2010 at 5:00 a.m. PST, we will be closing logins and disabling major inworld services for ninety minutes of additional upgrade work on our databases.

During the maintenance, logins, account registrations, XStreetSL, and the land store will be unavailable, along with Support Ticketing, Live Chat and telephone support.

The Knowledge Base will remain available.

Residents who are in world will not be disconnected when maintenance begins, but will lose access to services such as

» mapping,
» teleport,
» inventory,
» object and L$ transfers, and
» LindeX.

We will  warn in world several minutes before the outage begins, and  post any changes to the scope and schedule of the work here as they develop.

We appreciate that this will inconvenience a number of our Residents. The maintenance work is to enhance the stability and performance of our database-backed services.

[9:03 AM PST][RESOLVED] The equipment has been fixed. Log ins and registrations should be back to normal.

[8:41 AM PST]  We are currently experiencing an equipment failure. It will affect many log ins and new registrations. We will post updates here as we have them.

[Resolved} Our support system vendor will be restarting the system at 9 p.m. Pacific this evening. Support tickets and chat will be unavailable for about five minutes.

As always, if the scope or scheduling of the work changes, we'll update here.

[9:33AM PST][RESOLVED] The grid rolling restart is finished.

At approximately 10pm PST, we will begin a rolling restart of the grid for some hardware moves.  This should last approximately 8 hours and should not affect logins or other resident activities outside of the restarting regions.

[Resolved @ 7:13 am PST] Stipend payments are now back on track and should be completed. Thank you for your patience.

Some residents may experience their stipends being paid out slightly later than usual. We are aware of this and working towards a resolution. We apologize for any inconvenience caused. Thank you for your understanding.

[POSTED Jan. 15th 3:20 PM PST] We will be conducting support portal maintenance this evening starting at 10:00 PM PST.  This will be an approximately 30 minute maintenance update.  During this time, residents will not be able to create tickets or speak to a live chat agent.  For abuse issues, please continue to use our abuse report system.  We apologize for any inconvenience this may cause.  Please follow this blog for any updates.

[Resolved] Maintenance has been completed. Thank you for your patience.

We will be conducting some routine server maintenance that may briefly affect some regions and residents.  We apologize for any inconvenience this may cause.  Please watch this blog for any new updates as they become available.

[1:48 PM PST]  [RESOLVED] Vivox has informed us that this issue has been resolved and log ins to voice have returned to normal.

[1:28 PM PST]  At this time Vivox is investigating an issue with the SIP proxy system.  This is causing intermittent failure to log into voice.  Residents in voice in a given location will continue to have voice service, but they may experience voice failures if they attempt to teleport or switch between spatial voice and group voice. We will post further updates as we get them.

[Resolved @ 6:21am PST] The maintenance has completed. Logins and registrations have been re-enabled. Thank you for your patience.

Starting at 6 a.m. PST, all login and registrations will be offline due to maintenance.  Logins will not be available during that time. This will not affect our residents currently in-world.  Maintenance should take approximately 30 minutes.  We apologize for any inconvenience.

[11:38 AM PST] [Resolved]  Stipends have completed the run for this week.

[07:41 AM PST] There may be a delay for some residents receiving their stipends today.   Please check this page for more updates.

[Resolved 4:55pm PST] All services have returned to normal. Thank you for your patience.

[Posted 3:20pm PST] Our billing, technical, and outworld support teams are limited at the moment due to a power outage. Concierge phone and live chat and premium live chat support is still available. We will post an update here when this issue is resolved.

[9:45am Pacific]  Support Tickets and Live Chat now restored.

[9:21am Pacific]  A fault has occurred with our support portal and both tickets and Live Chat are currently unavailable.  Telephone support remains unaffected.  We are working to resolve this issue and will update you here with further information.

[10 Jan 2010, 5:20pm PST] [RESOLVED] All services have returned to normal.

[10 Jan 2010, 4:07pm PST] Another power outage has again limited our billing, technical, and outworld support teams. We will post here as more information becomes available. Our apologies for any inconvenience.

[09 Jan 2010, 10:30pm PST] [RESOLVED] All services have returned to normal.

[09 Jan 2010, 8:58pm PST] [UPDATED] Another power outage has again limited our billing, technical, and outworld support teams. We will post here as more information becomes available.

[09 Jan 2010, 8:39pm PST] [RESOLVED] Power has been restored and the support teams are available again.

[09 Jan 2010, 8:25pm PST] Our billing, technical, and outworld support teams are limited at the moment due to a power outage. Concierge phone and live chat and premium live chat support is still available. We will post an update here when this issue is resolved.

[January 9 2010 9:05pm Pacific] [RESOLVED] Server maintenance is complete, thank you for your patience.

[Posted Jan. 9th 2010 3:46PM PST] We will be conducting some routine server maintenance that may briefly affect some regions and residents.  We apologize for any inconvenience this may cause.  Please watch this blog for any new updates as they become available.

[POSTED 11:42 AM PST]  Information that was recently uploaded to our Public Issue Tracker after the recent database maintenance may have been lost.  This is being looked by our technicians.  Please keep an eye on this post for additional information and status updates as they are available.

[Resolved Jan. 8 2010 1:57 AM PST] The missing data issues from Wednesday’s JIRA outage and rollback should now be resolved.  In tandem with residents re-posting their lost activity, we’ve recovered enough logging information to manually restore all remaining contributions.

We sincerely apologize again for any inconvenience this may have caused anyone while using JIRA, and thank all the residents who re-posted their contributions. Please contact support if you discover that some information is still missing or if you need any further assistance, and we will do our best to help you.

[Resolved Jan. 7 2010, 3:16 am PST] The JIRA Service has been re-enabled and any previous changes that were lost have now been restored.   This required a restore from yesterday afternoon. Anything that was posted in the JIRA between 13:30 and 20:00 PST on January 6th has been lost and may need to be reapplied. We sincerely apologize for any inconvenience caused while we resolved this matter.

[Updated 9:42pm PST] We are still working on bringing the JIRA service back up and running. Thank you for your patience while we resolve this issue. We will continue to update this blog post with any updates as they become available.

[Updated 8:00pm PST] The issues we were investigating earlier related to JIRA updates not showing on their respective pages has lead us to disable access to JIRA until further notice. Please check this blog for more updates as they become available. We apologize for the inconvenience.

[Posted 7:30pm PST] We are currently experiencing some issues with the JIRA service where recent updates are not showing on their respective pages. We are currently investigating the issue and will update this blog when we know more.

[Completed 7:23pm PST] All of our support tools are available again. Thank you for your patience while we completed this maintenance.

[Posted 4:33pm PST]  One of our internal support tools will be unavailable for the next couple of hours for maintenance. Residents making Support requests via phone or Live Chat may be requested to defer some issues to tickets for resolution or to call back after the maintenance has finished.

[January 3, 2010 5:30 pm PST] This has been resolved and service has been restored for logins and inventory. Thank you for your patience.

[3:11pm PST] An isolated number of residents who have been experiencing the login and inventory issues will be logged out from inworld and will experience a short delay logging back in. We will keep this delay as minimum as possible. Please refer back to this blog for updates.

[3:02pm PST] We are still experiencing isolated issues with logins and inventory. Please refer back to this blog for updates.

[1:01pm PST] We are currently experiencing isolated login issues. You may see extended login times, difficulty logging in or errors in logins. We are also seeing related isolated inventory issues where a small group of residents will only see their default library of inventory. Please refer back to the blog for updates.

[RESOLVED 6 Jan 6:30am Pacific] Maintenance has now completed. All services should once again be available. Thank you!

[Update 6 Jan 5:00am Pacific] Maintenance has begun. Please check this page for updates.

[Edited 5 Jan 09 10:30 a.m. added land store and XStreetSL to list of disabled services]

[Bumped 4 Jan 09 2 p.m. Pacific]

Wednesday January 6 2010 at 5:00 a.m. Pacific, we’ll be closing logins and disabling major inworld services for ninety minutes of upgrade work on our databases.

During the maintenance, logins, account registrations, XStreetSL, and the land store will be unavailable, along with Support Ticketing, Live Chat and telephone support.

The Knowledge Base will remain available.

Residents who are in world will not be disconnected when maintenance begins, but will lose access to services such as

• mapping,
• teleport,
• inventory,
• object and L$ transfers, and
• LindeX.

We’ll warn in world several minutes before the outage begins, and we’ll post any changes to the scope and schedule of the work here as they develop.

We appreciate that this will inconvenience a number of our Residents. The maintenance work is to enhance the stability and performance of our database-backed services.

[Resolved 1.30am PST] This is now Resolved. Rolling restarts have completed.

[Update 4:40pm PST - Server 1.34.2 was ready earlier than expected and thus we have begun deploying it now. We will update this blog post with any further developments.

Server 1.34.2 will be deployed tonight in order to fix a severe sim-crashing bug.  The roll will be commencing as soon as possible, which is expected to be around 18:00 PST.  Status updates will be posted here and in the forums.

• 2009-12-31, ASAP (~18:00 Pacific) - 100% sim roll

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

For questions / comments, please refer to the forums: [forums.secondlife.com]

[30 Dec 7:27pm PST] We will be performing maintenance on JIRA starting at 12am PST. The outtage will be very brief. Please refer back to this post for any updates.

[UPDATED/RESOLVED 30th December 6.40am PST] The maintenance is now finished and all residents should be able to login as normal. If you can not login, please contact support.

[UPDATED 30th December 6.00am PST] Maintenance is starting now. We will update when it is finished.

[29th December 4:45pm PST] We will be performing emergency maintenance on December 30th between 6 - 6:30am PST. We are performing this maintenance in response to login issues that occured on Dec 29th. This will effect a small portion of residents. Those particular residents will not be able to log in or will be logged off if they are inworld during the maintenance. Please refer back to this blog for further updates.

[UPDATED/RESOLVED] 4:21pm PST] Logins are back to normal.

[UPDATED 3:35pm PST] The majority of logins have returned to normal. Residents may find difficulty with logging in; however, it should be successful after a few attempts. We are continuing to work on and monitor the issue. We will update this blog with any new developments.

[UPDATED 3:04pm PST] We have resolved the initial issue; however, logins for all accounts may be slow or fail currently. We expect this to last approximately 15 - 20 minutes. Please continue to watch this blog for further updates.

[UPDATED 2:36pm PST] We are in the midst of resolving this issue, which will have resulted in those affected being logged out of Second Life. We anticipate about an hour before those affected will be able to log back in. We will update this blog when further information is available.

[POSTED 2:19pm PST] We are currently experiencing log in issues which affect a small percentage of Residents. This will prevent the ability to log in, although it’s possible some of those affected may be able to log in after several attempts. We are working to resolve this as quickly as possible. Please watch this blog for further updates as they become available.

[8:50pm Pacific] [RESOLVED] Phone support issues have been resolved.

[7:50pm Pacific] Phone support is available again, however there are still intermittent connectivity issues at this time. If your call will not connect or if it connects with poor quality, please retry again after a short time. We will post here when more information is available.

[6:10pm Pacific] We’re currently experiencing some connectivity issues with our phone support system. Billing and password reset phone support is unavailable at this time. The Concierge phone support line is still available. We are working on this issue and will post here when we have more information.

[7:08pm PST] All inworld issues have been resolved.

[POSTED 4:57pm PST] We are currently experiencing some issues which may prevent residents from logging in to Second Life. This issue is also affecting updates of in-world information, including online status, transactions and other related functions. Please refrain from making any transactions or handling no-copy objects until this blog has been resolved.

[5:30 AM PST] Phone support has been restored!

[4:30 AM Pacific] We have temporarily lost phone support via our service provider. We have taken steps to contact them and hope to have phone support back up as soon as possible. We will keep you posted and let you know as soon as phone support has been restored.

[RESOLVED 2:07pm PST] All related issues have been resolved. If you have issues logging in, please try again in a few minutes. All other functions should have returned to normal functionality at this time.

[UPDATED 12:56pm PST] This issue is affecting updates of in-world information, including the in-world balance display. Please refrain from making any transactions or handling no-copy objects until this blog has been resolved.

[POSTED 12:38pm PST] We are currently experiencing some login issues which may prevent residents from logging in to Second Life. We are working to resolve this as quickly as possible and will update this blog post as information becomes available.

[POSTED 9:26am PST] Over the next two days, starting today, we are conducting maintenance on our servers which will affect some regions.  There will be occasional 5 minute restarts while this maintenance occurs.

[RESOLVED 9:19am PST] The Second Life Blogs service has completed and the blogs are now available.

[8:43 AM PST] The Second Life Blogs are temporarily down for unscheduled service. Please continue to check back here for updates.

[RESOLVED 2:41pm PST] The issues have all been resolved. XStreet SL purchases are re-enabled and all in-world functions should have returned to normal as well.

[UPDATED 12:14pm PST] We are continuing to work on this issue. In the meantime, XStreet SL purchases have been disabled as to avoid any undue complications. We will update this blog when we have more information.

[POSTED 11:26am PST] There are currently some issues with XMLRPC functions. This may result in payments to XStreet terminals not fully processing or possible delayed/failed deliveries, some script issues, and some vendor/server issues… as examples. We are working to resolve this as soon as possible and will update this blog post as details became available.

[RESOLVED 3:32pm PST] While the issue of region delays has been resolved, there may be some regions that are getting up to speed a bit longer than others.  We ask that you please be patient as they will be available shortly.

[UPDATED 12:27pm PST] Some progress has been made; however, work continues on this issue and will be resolved as soon as possible. We will update this blog when more details are available.

[POSTED 9:41am PST] This morning, various regions are suffering a delay in returning from a crash or restart. We are working to resolve the issue as soon as possible, in the meantime we are manually addressing regions which are having this issue. Please watch this blog post for further updates as they become available.

[02:23 AM Pacific] [RESOLVED] The support portal is now back online again, you will be able to use live chat and tickets as normal.

[11:58 PM Pacific] We have temporarily lost our online support portal. We hope to have support back up as soon as possible. We will keep you posted and let you know as soon as support has been restored.

[Posted 2:55pm PST] Tonight starting around 12:00am PST we will be conducting maintenance on PJIRA (http://jira.secondlife.com/) and the wiki (http://wiki.secondlife.com/). During this maintenance, there will probably be some interruption to both services. We expect the maintenance to last approximately one hour. We apologize for any inconvenience.

[COMPLETED 7:14pm PST] All server maintenance restarts have been completed. The team who handled the restarts were able to finish in half the allotted time.

Over the next two days, we will be conducting maintenance on our servers which may affect some regions.  There will be occasional 5 minute restarts while this maintenance occurs.

[5.33am Pacific] The rolling restart for today is about to commence. We will update when this has finished

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.
For questions / comments, please refer to the forums: [forums.secondlife.com].

[RESOLVED] 07:00am PDT - All issues have now been resolved with the Lindex purchases.

[UPDATE - 05:30am PDT]  Some residents who are making Lindex purchases appear to be getting charged $2000 each time so again please refrain from making Lindex purchases until further notice.

[03:05am PDT]  Some residents maybe experiencing some difficulties purchasing currency inworld.

We are aware of the issue and working towards a speedy resolution. For the time being, please do not make any Lindex purchases as your payment method will be charged, but your purchase will not be delivered.

Thank you for your understanding.

Please check back shortly for further information.

[Updated 7:03pm PST] We have completed the rolling restarts needed to deploy server version 1.34.1.

[Updated 5:18pm PST] The rolling restarts for this evening are about to commence. We will update when this has completed.

[Updated 12:00pm PST] The rolling restarts for the morning are finished. We will continue the process tonight starting around 5:00pm PST.

[Updated 5:33am PST ] The rolling restart for today is about to commence. We will update when this has finished.

We will be deploying server version 1.34.1 to the main grid tomorrow due to a content threatening bug. The rolling restart is scheduled as follows:

• 2009-12-15, Tuesday, 5:30am - Deploy to as many sims hosts as possible until 12:00pm
• 2009-12-15, Tuesday, 5:00pm - Deploy to any remaining sim hosts

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.
For questions / comments, please refer to the forums: [forums.secondlife.com].

[3:30 AM PST]  Scheduled maintenance to our Support Portal is complete!

[December, 13 2009 3:00 AM PST] Scheduled maintenance to our Support Portal is underway.

On Sunday (13th December) at approximately 3:00 AM PST our Support Portal will be down for scheduled maintenance.  During this time residents will be unable to submit tickets or access Live Chat. Concierge and Billing support by phone will still be available.

[RESOLVED] 11 Dec 09 11:50PM Issues with live chat have now been resolved.

11 Dec 09 6:45PM Intermittant issues with live chat. If you are having problems connecting to an agent, please be patient while it connects as it may take longer than normal, or try again after a short time.

[Resolved 11:39am PST] Access to Live Chat and support tickets have been fully restored. We apologize for any inconvenience this may have caused.

[UPDATE] Some Residents are unable to submit tickets as well.  We’re chasing down the issue as quickly as possible.

Some Residents may experience some issues accessing live chat. We are aware of this issue and working towards a resolution. Ticket submissions are still working as they should. We apologize for any inconvenience.

We have had reports of old IMs, some as old as 2 months, reappearing in chat as if they were just resent. We want to assure residents we believe this is a technical problem and not associated with anything intentional or malicious. We are investigating the cause and if you wish to send in a support ticket with your experience we will pass that information on to the investigative team.

[Updated 5:55am PST 09-Dec-09] We are continuing to work to resolve this issue and will update as soon as we have further information.

[Updated 3:26pm PST] We have made some progress on this issue and are continuing to work on resolving this as quickly as possible. In the meantime, if you find that you are in a region in which you cannot view proper About Land information, as well as crashing when trying to teleport out, please contact support.

There are a number of regions which are experiencing an issue where there is no information presented in About Land. Affected regions may result in crashing when trying to TP out.  We are working to resolve this issue and will update with more information as it becomes available.

[Resolved 10:55am PST] This issue has been resolved and you should be able to view groups, as normal, once again.

We are aware that some residents maybe experiencing some issues with groups not showing inworld. This situation is being investigated and we expect a resolution as quickly as possible. Thank you for your patience.

[RESOLVED] 7 Dec 09 11:50PM PST - All issues have now been resolved.

[Update] 7 Dec 09 10:40 PM PST - Lindex, Registrations, Logins, XStreet are up and working. Landstore is still currently unavailable. Please watch this blog for further updates.

[Update] 7 Dec 09 - Logins and Landstore are currently Disabled

7 December 09 10:10PM - We are currently experiencing some internal issues, this includes but not limited to L$ transactions, rezzing of items, teleports, region crossings. Please dont rez any objects that are of value being no copy items until the all clear is given.  Please watch this blog for further updates.

[7:58 am PST] Live chats have been restored to full working order. Thank you for your patience.

[04:25am PST] We are experiencing some issues with our live chat support. We are investigating the issue and will post here with updates when we have more information. Ticket submissions is working correctly.

[7:44 AM PST] Service has been fully restored. We apologize for any inconvenience caused.

[7:00AM PST] A handful of Second Life websites are unavailable at the moment.  We are looking into this and will post any updates here as they become available.

[Resolved 1:30pm PST] These issues have since been completely resolved.

This morning several of our web services were down due to a service interruption. Unfortunately the status blog was one of those affected. The services are once again up and running. We apologize for any inconvenience.

[12:27pm Pacific] RESOLVED - Our live chat and ticket portal is back up.

[12:17pm Pacific] Our live chat and ticket portal is currently down. We’re investigating the issue and will post here with updates when we have more information.

[COMPLETED] 2009-12-10 13:37 PST: Rolling Restarts are complete * Update @ 2009-12-10 07:09 am PST: The deploy to odd numbered sim hosts is about to commence. * Update @ 2009-12-09 12:27 pm PST: The deployment to even numbered sim hosts has completed. * Update @ 2009-12-09 7:37 am PST: The deploy to even numbered sim hosts is about to commence. * Update @ 2009-12-08 19:23 PST: The deployment to the pilot sim hosts has been completed. At this time, we are sticking to the original schedule. * Update @ 2009-12-08 17:48 PST: The forementioned bug has been fixed and we are starting the pilot roll now.  If there are no other issues, we will stick to the original schedule for the rest of the roll. * Update @ 2009-12-08 11:07 PST: The centrals are complete.  The sim roll is being delayed due to a recently discovered bug.  New schedule will be posted when its available. The rolling restart is scheduled as follows:

• [Completed ] 2009-12-08, Tuesday, 7am - Deploy to central servers
• [Completed ] 2009-12-08, Tuesday, 5:45pm - Deploy to pilot sim hosts
• [Completed ] 2009-12-09, Wednesday, 7am - Deploy to even numbered sim hosts
• [Completed ] 2009-12-10, Thursday, 7am - Deploy to odd numbered sim hosts

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Please refer to the forums for the most up to date information, including release notes, deploy updates, and comments:

[forums.secondlife.com]

[Resolved 11:23am PST] All login issues for residents have been fixed.

[Updated 11:12pm PST] We are still experiencing login issues for some residents. We are continuing to work on resolving this issue as soon as possible and will update this blog post once we have further information.

[Updated 10:08pm PST] We are still experiencing login issues for some residents. We are continuing to work on resolving this issue as soon as possible and will update this blog post once we have further information.

[Updated 9:04pm PST] We are still experiencing login issues for some residents. We are continuing to work on resolving this issue as soon as possible and will update this blog post once we have further information.

[Updated 7:22pm PST] We are still experiencing login issues for some residents. We are continuing to work on resolving this issue as soon as possible and will update this blog post once we have further information

[Updated 5:31pm PST] We are still working on resolving this issue; however, some people may have found they have been able to log in successfully. For accounts which are not able to login, you may see an error message, “The inventory system is unavailable”. This is related to the work being done and does not signify any issues with your inventory.

[Posted 2:15pm PST] We are currently experiencing login issues for some residents. If you have tried to login and find that it stalls at the Loading bar, you should consider that account as affected by this issue. We are working to resolve this issue as soon as possible and will update this blog post once we have further information.

[7 Dec 09, 4:25 p.m. Pacific]

We’re bumping this post from last week because there’s still a little work to be done to complete this project. At 7 p.m. Pacific tonight and tomorrow, we’ll be migrating about 800 regions per night. Each affected region will experience a standard restart, including a five-minute warning.

Each phase will conclude by 7 a.m. the following morning.

[1 Dec 09, 4:25 p.m. Pacific]

Beginning at roughly 7 p.m. Pacific this evening, and at about 9 p.m. Pacific tomorrow and Thursday nights, we’ll be migrating about 800 regions per night to new hosts as we perform some work on a few server racks.

Each overnight phase should be concluded by about 5 a.m. Pacific.

Residents and visitors within those regions will receive the customary five-minute warning, followed by a normal restart cycle. Please note that while we’ll work to minimize the chances, it is possible that some regions may be restarted more than once during the maintenance.

As always, we’ll update here with any changes to the scope or scheduling of the project.

[RESOLVED] 13:40PM PST Maintenance for XstreeSL has been completed and the site is now functioning normally.

[UPDATE 08:08AM PST]  Maintenance has just began for XStreetSL

As Siz posted this past weekend, XStreetSL will be offline for approximately five hours beginning at 8 a.m. Pacific on Wednesday, 2 December.

We’ll keep you updated here on any scope and schedule changes for the work.

Once XStreetSL is available again, please report any unusual behavior by opening a support ticket. We’ll also be checking the forums for reported issues.

[Resolved 3:31pm PST] The stipend payments have finished earlier than expected and all payments due should now be paid. If you show that you did not receive an expected stipend, please contact support.

[Updated 1:35pm PST] Stipends are now on their way to processing for all who have not yet received them. We will resolve this blog once they have finished deployment, which is expected to be late tonight.

We are aware of a current situation where some stipends have not yet been paid. We are working to resolve this as quickly as possible and will update this blog post as soon as we have additional information or a resolution has been reached.

[RESOLVED 14:15PM PST] Problems with asset issues have been resolved. Logins, script performance and loading of assets should be returned to normal.

We are currently experiencing isolated, yet dispersive, issues with teleports, incorrect permissions, failed loading of assets, scripts not performing and related incidents. This may also have an affect on logins. If you find this to be an issue, please try logging in directly to a different region. We also suggest refraining from performing any transactions which involve L$ or no-copy assets, particularly if you are in an affected region.

We are working to resolve this as soon as possible and will update this blog post as information becomes available.

[4:05pm Pacific] RESOLVED - The maintenance is complete and logins are fully functional again.

[3:45pm Pacific] The maintenance is taking long than expected, and some residents may still be experiencing login issues. We will update this post as more information becomes available.

[2:30pm Pacific] A small number of residents may be impacted by login issues while we perform emergency maintenance. We expect this to be resolved within 30 minutes. We will post here when the issue is resolved.

[RESOLVED] 12:55pm PST The payment system problems that occurred earlier today have been resolved.

We are aware of problems with our or payment services provider. Problems residents may notice are delays in updating credit card information or buying L$ through the Lindex. Users of PayPal should see no such delays. We are in contact with our provider and are working to fix the problem as soon as possible.

[22 November 2009, 6:45pm Pacific] REOPENED - Some group details are again not showing in search. Our technical teams are looking into the issue and we will update this post as more information becomes available.

[21 Nov - 6PM PST] Group search issues have now been fixed.

We are aware that at present clicking on groups found in Search results in a “Page Not Available” error. We are investigating and will have an update as soon as we know more or have resolved the issue.

[19 November 09 - 5:15 a.m. Pacific] All in world functions and transactions should be available including new account registration, Second Life logins, and XStreet logins.

[19 November 09 - 5:01 a.m. Pacific] We have started the database cutover and it should take no longer than 30 minutes to complete.  All in world functions and transactions should be available. However, new account registration, Second Life logins, and XStreet logins will be unavailable.

[18 November 09 - 4:00 p.m. Pacific]

Beginning at 7 p.m. Pacific this evening, we’ll be adjusting the performance of roughly 1300 mainland regions, a few at a time.  Tomorrow at 7 p.m. Pacific, we’ll tune another 1300 regions to complete the maintenance.

The only visible impact to Residents within those regions should be a standard restart. There will be a five-minute advance warning for each restarting region.

Also, tomorrow morning beginning at 5 a.m. Pacific, we’re doing a quick database cutover. It should take no longer than thirty minutes to complete. During the cutover, all in world functions and transactions should be available. However, new account registration, Second Life logins, and XStreet logins will be unavailable.

During this scheduled maintenance, starting at 5:00am PST and projected to last until 5:30am PST, logins to Second Life and XStreetSL will be blocked and new registrations will not be available while we carry out a minor upgrade to a central database server. All in-world functions should continue to work normally.

[1:47am PDT] Service to support tickets has been restored. The support portal should be functioning normally. Thank you for your patience.

[00:51] AM PDT]  Live chat is now available from the support portal.   Support tickets are still down at the moment.

[00:17 AM PDT] Our support portal is currently down.   Residents will not be able to access live chat or submit tickets at this time.  Phone lines however, are still available.

[Nov 18 10:25 PDT] Maintenance has completed. Thank you for your patience.

[Nov 18 10:25pm PDT] We are performing emergency maintenance currently. There will be isolated login failures and possible log outs. Please monitor this blog post for updates.

[Resolved] 1:02pm PST - This issue has been resolved.

We are experiencing some issues that affects saving notecards and uploading images.  This issue is being addressed and should be fixed shortly.  We will update any changes.

[Nov 16, 10:30pm PST] Issues with LindeX have been resolved. Sorry for any inconvience this might have caused.

[Nov 16, 7:00pm PST] We are currently experiencing some issues with LindeX that may result in a delay with LindeX transactions being processed for some residents. We are working on resolving the issue as soon as possible. Please watch this blog for any additional information.

[2:22 PM PST] Voice issues are now resolved.

[1:21 PM PST] We are currently experiencing voice issues in Second Life. It appears to be affecting a small segment of our residents and we are working with the engineers to correct the problem Please keep checking back here for updates.

There have been some residents who may have experienced their tier not having been billed last night as it should have. This should be resolved by tomorrow morning.

We apologize for any inconvenience caused.

[Completed 11:18am PST, Nov 18] The server deploy for odd numbered sim hosts has been completed. This concludes the 1.32.1 server update.

[Update 7:00AM PST Nov. 18, 2009] Server deploy for odd numbered sim hosts has begun.

[Resolved 11:40AM PST] The Rolling Restart of the even numbered sim hosts is complete.

[Update 7:00AM PST Nov. 17, 2009] Server deploy for even numbered sim hosts has begun.

We will be deploying server version 1.32.1 to the main grid during the week of November 17th

. The rolling restart is scheduled as follows:

• 2009-11-17, Tuesday, 7am - Deploy to even numbered sim hosts
• 2009-11-18, Wednesday, 7am - Deploy to odd numbered sim hosts

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Please refer to the forums for the most up to date information, including release notes, deploy updates, and comments:
[forums.secondlife.com]

[Resolved,10:21am Pacific] The rolling restarts that began last night at 8:20pm PST have now been completed.

[8:20pm Pacific] We are beginning tonight’s rolling restarts. Each region will have a 5 minute warning prior to shutting down and will only be offline for a short time. We will post here when the restarts are complete.

[Resolved 06:57am PST, 13 November 2009] The maintenance work has been completed.

***

We’ve scheduled a quick server move for tomorrow (Friday, 13 November) at 6 a.m.  During the move, features involving currency or L$ will be unavailable.  These include:

• LindeX transactions,
• updating payment info at the Website, and
• upgrading to Premium membership.

We anticipate that all services will be restored by 6:30 a.m., possibly sooner.  If the scope or scheduling of the work change, we’ll update you here.

[12 Nov 7:54am PDT] Resolved - isolated inventory issues have been resolved.

[12 Nov 7:43am PDT] We are currently experiencing isolated inventory issues. This issue will effect a small portion of residents. If you are having issues seeing or rezzing inventory, we are working to correct this issue as quickly as possible. Refer back to this blog for updates.

[Resolved 1:06pm PST] Both the region restart issues and transactions are now resolved. This should also resolve any related asset upload issues.

[Updated 12:41pm PST] We have identified the cause of the transaction issues and are applying a fix to affected regions. This may cause residents in those regions to be logged out without warning.

We are currently experiencing issues where some regions are not returning after a restart (or crash). This is currently being addressed as efficiently as possible. In isolated incidents, there may also be issues with asset uploads and/or transactions.

We will update this blog as more information becomes available.

[Updated 7:00am PST - Nov 12, 2009] Inventory database maintenance has begun. We will resolve this post when it has completed.

We are arranging a partial outage on Thursday November 12, 2009, for maintenance to an inventory database. Only a small group of residents will be affected by this between 7:00am and 7:30am (PST). Affected residents will be logged out, if in-world, at the time; as well as being unable to log in.

We are about to begin another rolling restart of up to 640 regions. All regions will receive a 5 minute warning before shutting down, and we will update this post once we’re done.

[5:00am Pacific - Nov. 11, 2009] Rolling restarts have completed.

[7:30pm Pacific - Nov. 10, 2009] We are now commencing rolling restarts of approximately 750 regions. We will post here when the restarts are complete.

[5:56pm Pacific] The system clock issues have been resolved, and you should now be able to log in normally.

[5:15PM Pacific] We’re currently experiencing an issue with logins. You may see a message that indicates you cannot log in due to the system clock being set incorrectly. As a temporary workaround, setting your computer clock to Pacific Standard Time should allow you to login.

We’re working on resolving this issue as quickly as possible and will post here with updates.

[1:20AM] The Support Portal is now back online.

[12:40am - 7 November 2009] The Support Portal is currently experiencing issues with both Live Chat and the ticketing system. We are working to resolve this as soon as possible. Please watch this blog for information as it becomes available.

[Nov 5, 7:12am PDT] We will be starting the rolling restarts for odd numbered sim hosts server version 1.32.

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Please refer to the forums for the most up to date information, including release notes, deploy updates, and comments:
[forums.secondlife.com]

[Resolved - 11:35 AM PST] Today’s rolling restarts are complete. [7:51 AM PDT] We will be starting the rolling restarts for even numbered sim hosts server version 1.32. As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Please refer to the forums for the most up to date information, including release notes, deploy updates, and comments:
[forums.secondlife.com]

[RESOLVED 3 November 2009, 12:43am PDT] The Support Portal is now back online.

[3 November 09  12:23am] The Support Portal is currently experiencing issues with both Live Chat and the ticketing system. We are working to resolve this as soon as possible. Please watch this blog for information as it becomes available.

[RESOLVED 30 October 2009, 1:00am PDT] The Support Portal is now back online.

[POSTED 12:30AM PDT] The Support Portal is currently experiencing issues with both Live Chat and the ticketing system. We are working to resolve this as soon as possible. Please watch this blog for information as it becomes available.

[30 October 09  12:30am] The Support Portal is currently experiencing issues with both Live Chat and the ticketing system. We are working to resolve this as soon as possible. Please watch this blog for information as it becomes available.

[6:00AM PDT] The various issues affecting residents this morning has been resolved.

[4:25AM PDT] We have made progress however this issue isn’t quite resolved.  Those who were unable to connect to Second Life should now be able to login.

[3:05AM PDT] Logins have been temporarily disabled.  Please check back shortly for further updates.

Some residents will have difficulty logging in while other residents will have some issues inworld with transactions, teleporting, and rezzing failures. Please refrain from rezzing or making any inworld transactions until we have resolved this issue.

Please check back soon for further information. Thank you for your patience.

[RESOLVED 28 Oct, 1:36pm PDT] The Xstreet SL site issue has been resolved and the site is once again available.

[POSTED 28 Oct, 12:33pm PDT] The XStreet SL site is currently unavailable. This is being worked on and will be resolved as soon as possible. Please watch this post for updates.

[Updated @ 9:21am PDT] This issue has been resolved and the affected regions should have come back online.

Once again, thank you for your patience while we resolved this matter.

There are a few regions failing to come online after a restart. We are aware of the issue and working to resolve this as quickly as possible.

Thank you for your patience. Please check back soon for an update.

[05:45 Pacific]  The Support Portal is now back online.

[03:00 Pacific]  The Second Life Support Portal is offline from now, Tue 27 Oct 03:00 Pacific, until Tue 27 Oct 06:00 Pacific.

Ticket submissions and Live Chat will not be available but Telephone support remains unaffected during this maintenance period.

We will be deploying server version 1.32 to the main grid during the weeks of October 26th and 30th. The rolling restart is scheduled as follows:

• 2009-10-28, Wednesday, 8am - Deploy to centrals
• 2009-10-29, Thursday, 8am - Deploy to pilot sims
• 2009-11-04, Wednesday, 7am - Deploy to even numbered sim hosts
• 2009-11-05, Thursday, 7am - Deploy to odd numbered sim hosts

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Please refer to the forums for the most up to date information, including release notes, deploy updates, and comments:
[forums.secondlife.com]

[RESOLVED 26 Oct, 5:07pm PDT] The Login, Teleport and Asset issues, as related to this blog post, have all been resolved and should work as normal now.

[POSTED 26 Oct, 4:23pm PDT] We are currently experiencing some isolated issues with Teleporting and Logins. There may be related asset issues and as such, we advise taking care with any transactions or rezzing of Inventory items. We are working to resolve this as soon as possible and will update this post with any further information as it becomes available.

The JIRA and Wiki will undergo maintenance starting at 11pm tonight (24 Oct), to last approximately 10 minutes. Please check this blog for any updates.

[RESOLVED 21 Oct, 10:57am PDT] Both JIRA and wiki.secondlife.com are back online and available for use. Some functions may be hampered temporarily as the work is finished; however, full functionality should be restored shortly, if not already.

[UPDATED 21 Oct, 10:03am PDT] We have identified some storage issues that are affecting both JIRA and the Wiki. A solution is being worked on. Please continue to check back for more info.

[21 Oct, 9:03am PDT] JIRA and wiki.secondlife.com are currently unavailable. Please check back with this blog for updates.

[19th October 09 1.19AM] This has now been resolved. Logins are working correctly.

[19 October 09 11:20PM]  We are currently experiencing some issues with our logins. We are working to resolve this as soon as possible and will update with any further information.

We recently made several changes to the Second Life servers to address a potential security issue. To take advantage of the recent server upgrades and ensure that Second Life functions as you expect, we have released a new version of the Second Life Viewer.

Residents using the Official and Release Candidate versions of the 1.23.4 Viewer are required to update to version 1.23.5. Those of you still using version 1.22.11 will be offered version 1.23.5 as an optional update. If you choose to remain on 1.22.11, you may notice changes in the way notecard editing behaves. If you use a viewer not created by Linden Lab, you may also see changes in the way notecard editing behaves. To avoid any potential confusion caused by these behavioral changes, we strongly recommend that you immediately update to the new 1.23.5 Second Life Viewer.

We apologize for any inconvenience this update may cause.

[3:30 AM PDT] Service to phone support has been restored.

[12:20 AM PDT] We have temporarily lost phone support via our service provider. We have taken steps to contact them and hope to have phone support back up as soon as possible. We will keep you posted and let you know as soon as phone support has came back.

[COMPLETED - 14 Oct, 11:47am PDT] The second half of the Rolling Restarts have been completed. This concludes the Rolling Restarts for the 1.30.2 server deploy.

[October 14th 7:00am PDT] The second half of the rolling restarts for the 1.30.2 server deploy have begun.

[RESOLVED Oct-13 11:35am PDT] The first half of the rolling restarts have completed for today.

[POSTED Oct-13 7:00am PDT] The first half of the rolling restarts for the 1.30.2 server deploy have begun.

[Oct-13 7:10AM PDT] secondlife.com is once again available.

We are aware of current problems with the Second Life homepage. We are working to fix the problem and hope to have it resolved soon. We will update this post as soon as we have more information as to when we can expect this problem to be fixed.

[Oct 12, 7:16pm PST] Phone Support is back online.

We are experiencing an outage of our Phone Support capabilities currently and are working to resolve the issue as soon as possible. Please watch this blog for further updates.

[RESOLVED Oct 14, 08:45am PDT] The maintenance work has been completed.

[POSTED - Oct 12, 2:07pm PDT] Wednesday, October 14, there will be minor server maintenance which will result in a subset of users that will be logged out of Second Life or unable to login during this time. The maintenance is set to run from 6:30am (PDT) to 7:00am (PDT). This work has now started.

Server 1.30.2 is scheduled for deployment this week to address an important security issue. The schedule is as follows:
* 2009-10-13, Tuesday, 7:00am - Even sims (50% of grid) - [Complete]
* 2009-10-14, Wednesday, 7:00am - Odd sims (50% of grid) - [Complete]

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Release notes for the 1.30 server versions are available here: [https:]

Please leave all comments on the forum: [forums.secondlife.com]

[11 October 2009, 9:00pm] This issue has now been resolved. All transactions should be working as normal

[11 October 2009, 7:40pm] We are currently having problems with our Credit Card service provider. We are investigating the issue. This can cause problems with L$ purchases. If you are attempting to update your credit card details and it fails, please try to enter once more. If after trying twice and still experiencing problems, please wait until the issue has been resolved.

[7:45 AM PDT] This issue has been resolved, you should now be able to teleport and connect to Second Life.

[6.40 AM PDT] We are currently experiencing some issues with our logins and teleports and other services may also be affected. We are working to resolve this as soon as possible and will update with any further information.

[RESOLVED] Maintenance to wiki and Public Issue Tracker has been completed.

[Oct 7th, 8:10pm] The network on which the JIRA and wiki are located will undergo planned maintenance from 11pm to 1am PT tonight. Please check back with this blog for updates.

[RESOLVED - Oct 7th, 5:23pm PDT] We have fixed our Xstreet SL PayPal deposit problem, although we did have some PayPal payments to Xstreet SL fail during this time. If you made a PayPal payment to Xsteet SL today, October 7th 2009, between 10:30am to 5:00pm PST, that did not show up in your Xstreet SL account, we will be refunding these back to you as quickly as possible. If we have any further announcements on this issue, we will update this post. [UPDATED - Oct 7th, 3:10pm PDT] We are continuing to investigate issues with making PayPal payments for Xstreet SL purchases.

[POSTED - Oct 7th, 1:20pm PDT] We are experiencing some XstreetSL issues after a backend release today and are addressing any problems presently. Please watch this post for further updates.

[Resolved 07:31am PST] Logins via OpenID are fully functional again.

***

Logins to Second Life via openid are runing at reduced capacity at present.  Logins might be delayed or fail at times.

However, normal login methods to Second Life are working correctly.

We are aware of this issue and it is being worked on.  We will update you here with further information.

[RESOVLED - Oct 4th, 9:55pm PDT] At this time all regions should be functioning normally.

[POSTED - Oct 4th, 7:30pm PDT] We are experiencing issues with a number of regions that respond “failed to grant capabilities ” upon trying to enter. We are currently aware of the issue and working to correct it as soon as possible. Please watch this post for updates.

[Update 08 October, 1:11pm PST] The rolling restart is complete for 1.30.1

[Update 08 October, 07.08am PST] The roll for odd numbered regions is now under way.

[Update 07 October, 11:41am PST] The roll for even numbered regions has been completed.

[Update 07 October, 07.00am PST] The roll for even numbered regions is now under way.

[Update 06 October, 11:50am PST] The pilot roll has been completed.

[Update 06 October, 08.00am PST] The pilot roll is now under way.

Rolling Restart for 1.30.1
The schedule is as follows:

* Tuesday, Oct. 6th: Pilot regions - [Complete]
* Wednesday, Oct. 7th: even sims (~50% of the grid) - [Complete]
* Thursday, Oct. 8th: odd sims (the other 50%)

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Full release notes are pending. The Beta release notes are here:
[wiki.secondlife.com]

Please post all comments on the forum:
[forums.secondlife.com]

[8:10pm Pacific] wiki.secondlife.com will be down for approximately 15 minutes, starting at 1:00am Pacific on Friday October 2.

We are currently experiencing problems billing to Visa gift cards and are investigating.  If the card you were using previously no longer works, you may want to have a look at your gift card FAQ’s or contact the issuer of the card. In the meantime, we recommend updating to a different payment method.

[11:06 am PDT] {RESOLVED] Support portal is now accessible.

[10:45am PDT] We are currently investigating reports of inaccesability to our support portal. Please check back here for further updates.

[RESOLVED 27 Sept 09 - 1:40AM] The problem with L$ Transactions has been resolved, if you are still experiencing balance issues, please relog and the correct amount wil be shown.

[27 Sept 09 - 1:20AM] Currently there are problems with L$ transactions. The problem is being investigated at the moment. Please watch this blog for any further updates.

[RESOLVED - 27 Sept 09 - 12AM PDT] The Support Portal and Service Desk are now accessible again

[26 Sept 09  - 10PM PDT] The Support Portal and Service Desk is currently down from 10pm-12am Pacific Time on September 26th.  During this time, live chat support and ticket submission will be unavailable for a software update.  We will make notification once this has been completed and service is restored.

[Spe 23, 5:35pm PDT] Currently we are seeing transaction history issues on the Second Life website where if you are using Internet Explorer you will be able to view the transaction history but not download them; other browsers do not seem to be affected. Please check back with this blog for further updates.

[Sep 23, 05:17am PDT] [RESOLVED]  Ticket submissions seem to be back to normal now.

[Sep 22, 11:05pm PDT] The Support ticket system is currently experiencing minor issues. Ticket submission is taking longer to confirm on the Support page. The ticket will eventually post. We are working to correct the issue. Please refer back to this blog for further updates.

[RESOLVED 3:26pm PDT] The Support Portal is now fully functional again. If you see any errors or issues with connecting to a Live Chat session, please try again and it should perform as normal.

[POSTED 3:00pm PDT] The Support Portal is currently inaccessible for starting Live Chat sessions, as well as New Ticket submissions. We are working to resolve this as soon as possible. Please watch this blog for information as it becomes available.

[Resolved 10:01am Pacific] The asset server issues have been resolved and all services have been restored to normal.

[9:30am Pacific] We’re currently experiencing some asset server issues. You may notice inworld issues such as failure to rez, and objects failing to return to inventory. Please refrain from rezzing no-copy objects or making important transactions until we post an all-clear.

[10:43pm Pacific] This issue appears to have subsided.

[12:30pm Pacific] We’re experiencing intermittent connectivity issues with live chat support. If you’re attempting to connect and not getting a response, please wait and try again in a few minutes. We’re working to resolve this issue and will post here when we have more information.

[Resolved @ 9:00 am PDT] The rolling restarts have now concluded. Once again, thank you for your patience.

[Updated @ 8:02am PDT, September 18] The rolling restart has commenced. Any residents currently logged into a region that is being restarted will be automatically logged out.

The rolling restart should finish at 11 am PDT.

Thank you for your patience.

We are having a small additional pilot rolling restart which will follow this schedule:

»Fri. 09/18, 8-11AM PDT : a pilot roll of ~1600 regions

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions will be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

[Sep 17 4:26pm PDT] Our engineers have located an issue, and a fix has been deployed to the grid. We’ve seen a sginificant reduction in region crashes. Please contact support if you are having further crash problems.

[Sep 17 2:16pm PDT] We are currently experiencing a higher-than-normal crash rate after our rolling restart. We are aware of the issue and are working to resolve it. Please check back frequently with the blog for further updates.

[Sep 15th, 9:55pm PDT] We are currently experiencing isolated inventory issues. Please do not rez “no copy” objects or make important transactions for the time being. We are working quickly to resolve the issue. Please check back with the blog for updates.

[RESOLVED 4:14pm PDT] The Support Portal and Knowledge Base are both back up and accessible.

[POSTED 3:40pm PDT] The Support Portal is currently inaccessible (this includes the Knowledge Base). We are working to resolve this as soon as possible. Please watch this blog for information as it becomes available.

[7:15am PDT] The support portal is now available again for tickets and live chat.

[12:00am PDT 15 September 09 ] Support Portal is currently unavailable from midnight until 5am PDT due to upgrades to the system. This will include tickets and Live Chat. Please watch this blog for any updates.

[RESOLVED 7:30 PM PDT] Our Live Chat system problems have been resolved and is accessible again

[POSTED 3:20pm PDT] Our Live Chat system is currently unavailable. We are working to correct the problem with our service provider and are working to have the problem fixed as soon as possible. Until then, please use the Submit a Ticket option or wait until the system is back up and running normally. We will update this post as soon as we have more information or the problem has been resolved.

[RESOLVED 8PM PDT 10 September 09] Problems have now been resolved and payments that were failing have transferred.

[6:50PM PDT 10 September 09] Currently there are problems with L$ transfers inworld from XStreet, all outgoing payments have been suspended until the issue is resolved. If you have withdrawn L$ from XStreet, this will be queued until fixed.

[RESOLVED 12:52pm PDT] The issues with XStreet deposits have been resolved and account balances should all show properly. If you have any further issues, please contact Support as needed.

[POSTED 11:27am PDT] There is an issue with deposits made to XStreet via payments to Exchange Street or when using a remote deposit box. Any deposits made during this time will not show as having been credited; however, they have been recorded and will be updated correctly once this issue is resolved. This started at 8am PDT this morning and is expected to be resolved in 3 hours.

[RESOLVED 12:46pm PDT] All stipends have been paid out at this time. If you believe you did not receive yours, please contact Support.

[POSTED 5:52am PDT] The process that runs our stipends has been delayed and will finish later today.

[COMPLETED 6:21PM PDT] Minor maintenence that has occured is now complete however the issue is not resolved so we will be doing more maintenance at some point and we will update this blog as more information is available

[UPDATE 4:13pm PDT] Minor maintenance has occurred throughout the day; however, the main window for this maintenance will be for 5pm PDT, tonight. We do not have an estimated time for when the maintenance will be completed, but we will update this blog as more information is available.

[Posted 8:04am PDT] We are performing some more unscheduled maintenance work. You might see failures to login, rezzing, teleports and transactions. Please do not rez any no-copy items or make large transactions at this time until we sound the all clear here on the blog.

[Resolved 019:00pm PDT] The maintenance work has finished, and all service have been restored.

[Sep 3rd, 5:49pm PDT] We are currently performing some unscheduled maintenance that may cause isolated failures in logins, rezzing, transactions, or Teleports. Please procede cautiously inworld and check back with this blog.

[8:31pm PDT] All isolated asset issues inworld have been resolved. Thank you for your patience.

[6:40pm PDT] We aware that some areas of Second Life have been experiencing some asset server issues and that they are still present. We are working to resolve these issues as soon as possible. You may notice inworld issues such as failure to rez, and objects failing to return to inventory. Please refrain from rezzing no-copy objects or making important transactions until we post an all-clear.

[RESOLVED 09:43am PST] Logins are open again, the Landstore and XstreetSL have been re-enabled, and all regions previously down are back up.

If you experience a slow log in, please try to ride one attempt through all the way rather than aborting and starting a new one as that will help keep load on the login servers down.

[UPDATE 09:15am PST] In a further attempt to help our database recover, the Landstore and XstreetSL have been disabled.

[UPDATE 09:06am PST] We have temporarily closed logins to alleviate load on our database. This problem also affects our asset servers, please do not rez any no-copy items at this time, or make any transactions as they may fail.

[Wednesday, September 02, 09:01am PST] Due to problems in one of our colocations, many regions are currently offline. Our engineers are already at work to fix this as soon as possible. We will update any news here as they become available.

[Resolved @ 2:00pm DPT, September 19] The rolling restarts have completed. 1.30 is now deployed.

[Updated @ 3:48 pm PDT, September 18] The rolling restarts that were scheduled for Weds Sept. 23th and Thurs Sept. 24th have been rescheduled due to an important security fix. Please see the dates and times below:

• Fri, Sept. 18th, 6pm - 9pm: 1.30 deploy to 1/2 of agni [complete]
  
• Sat, Sept. 19th: 5am - 12pm: 1.30 deploy to 1/2 of agni [complete]
  

Thank you for your patience and please check back for further updates.

[Updated @ 7:12 am PDT, September 17] Today’s rolling restart has commenced. Any residents that happen to be in a region that is under going a restart will automatically be logged out. The rolling restart is expected to be completed by 11 am PDT. Thank you for your patience while we under go this deploy.

[Updated @ 11:33 am PDT] During testing, some regressions were discovered that need to be fixed and re-tested.  Consequently, the 1.30 deployment has been pushed back by a week.  The new times are reflected below.

We will be deploying server version 1.30.0 to Second Life during the weeks of September 14 th and September 21st . The rolling restart is scheduled to look like the following:

• Thurs Sept. 17th: 1.30 deploy to pilot regions
• Weds Sept. 23th: 1.30 deploy to 1/2 of agni
• Thurs Sept. 24th: 1.30 deploy to 1/2 of agni

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

[RESOLVED 5:25pm PDT] - The asset server issues have been resolved and all services have been restored to normal. Please feel free to return to your Second Life as usual.

[4:25pm Pacific] We’re currently experiencing some asset server issues. You may notice inworld issues such as failure to rez, and objects failing to return to inventory. Please refrain from rezzing no-copy objects or making important transactions until we post an all-clear.

[12:53am PDT] RESOLVED - All voice should be functioning correctly now.

[12.39 am Pacific] Regions are experiencing issues with voice currently. If you can not get voice chat to work please turn off voice and then turn it back on. We will update with more information as it is received

[9:56am PDT] RESOLVED - All in-world activities should be functioning normally.

Some residents may be experiencing difficulties in-world at the moment.  These may include teleporting, uploading images or textures, saving scripts, rezzing objects.

Our engineers are working on this issue to resolve it as quickly as possible.

[Resolved] Service to the Lindex Exchange has been restored.  If you find that you’re still experiencing errors, please reload the website. Thank you for your patience while we resolved this issue.

[Posted @ 5:15 am PDT] Some residents are unable to purchase L$ from the LindeX™ Exchange: Buy L$ page.  There are two alternative methods to purchase L$.  You may visit [www.secondlife.com] and click Buy L$ under the Linden Exchange section or click the L$ balance from within the Second Life Viewer.

[Resolved] The issue of login, teleporting, and transaction failures has been resolved. Thank you for your patience.

[Posted @ 1:52 PDT] Some residents will experience instances of login, teleporting and transaction failures. Please refrain from making any transactions for the time being. We appreciate your patience while we investigate this matter. Check back soon for more information.

[5:15PM Pacific] The earlier paypal issues have been resolved.

[7.49 AM Pacific] We are experiencing issues when residents are trying to upload new paypal details. These details are not being saved on the system. Those with current correct paypal details are not experiencing any issues. We will update as we know any further information.

[Resolved] [08:45 am PST]  All regions are now back online.

[08:19 am PST] Approximately 130 regions are currently offline.  Our technicians are working to get them back online ASAP.  We will update here when we have more info.

[Resolved 08:47am PST] The maintenance work has finished, and all service have been restored.

On the morning of August 25, starting at 4am PDT, there will be a short maintenance for voice chat. We expect this to last no longer than 1 hour, but may be shorter. Existing voice chats should see no effect, but there may be difficulties in starting new chats during this period.

[Resolved 01:20am PST] Upgrades have been completed and all logins are working

We’re experiencing some intermittent login issues due to a third party vendor network upgrade. If you receive an error message while attempting to log in, please wait a few minutes and then try again. We will post more details as they become available; check back for updates.

[16:00 PDT] The projected Support Portal maintenance for Aug 25, 2009 has been canceled.  We will post here  again when we have a new date for this maintenance.

On August 25th at 5:00AM PDT our Support Portal will be down while scheduled maintenance is performed.  During this time Ticket and Live Chat support will not be available for hopefully no more than one hour.  Please note that Billing and Concierge phone support will still be available.

This issue has been resolve. We appreciate your patience while we resolved this issue.

[Resolved @ 12:52 PDT]There have been some reports of US dollar transaction failures. We are currently investigating this matter and appreciate your patience. Please check back soon for further updates.

[RESOLVED] [August 20, 2009 9:00PM PDT] Second Life’s Public Issue Tracker is now available again

[August 20, 2009 5:00AM PDT] Second Life’s Public Issue Tracker is temporarily unavailable.  Any updates to this issue will be announced here as they become available.

[RESOLVED 00:08]  There was a brief technical issue which caused some inworld service disruptions, mainly failed teleporting attempts.  The issue has been corrected and inworld services have returned to normal.

[Resolved 17:54 PDT] Billing and Outworld Live chat are now working.

[17:27 PDT] Billing Live chat and Outworld Live Chat are currently not working. We are working on the problem. Please check back here for updates.

29 August 09 10:15pm [UPDATE] Some regions are still experiencing issues with traffic updating, this problem should be fixed with the next traffic update.

We are aware that some regions are showing 0 for traffic in Search > Places while not showing as 0 in About Land.  This should be resolved as soon as traffic calculations for the day have been completed.

[RESOLVED] [1:22am PDT] Aditi, our beta grid is now back online and open. Problems earlier have now been fixed.

[7:50pm PDT] Aditi, our beta grid, is currently offline as a security precaution. We will post an update here when it is re-opened.

[RESOLVED 15:53]  This issue has been resolved.  Regions should now come up.

[14:48]  We are experiencing an issue related to the rolling restart which is affecting the land store.  Land store orders will complete, but the new regions will not come up until after the rolling restart is completed.  If your land store order has not come online after the rolling restart is complete please contact Support.

[Completed  16:00] The rolling restart is completed, thank you for your patience.  We are finishing making sure all regions are back online and accessible.

[Updated 14:55]  We are aware that some regions are not coming back up as the rolling restart hits them.  They will come back up at the conclusion of the rolling restart, which is estimated to be another 30-60 minutes.  Please monitor this post and if your region has not come back up within 10 minutes of the conclusion of the rolling restart please contact support.

[Updated 13:09] The Rolling Restart is commencing.

[Updated @ 12:00 pm PDT] Rolling restarts are still on hold as we continue to bring the remaining offline regions back online.

[Updated @ 11:00 am PDT] We are still working to fix the regions that have been affected by this issue.  Please note that rolling restarts are still in progress.

[Updated @ 9:00 am PDT] The rolling restart has been halted for the time being as there are a number of regions that have failed to restart properly. The engineers are currently investigating this issue. Any regions that may currently be offline should be brought back online once the situation has been resolved.

[Posted August 7, 2009 @ 7:29 am PDT] We’ve completed the rolling restarts for 1/2 of the Second Life grid.

A rolling restart for 1/2 of the Second Life grid will deploy at 7:15 am PDT. We apologize for the delay and any inconvenience caused.

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods.  Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade.  If your region stays down for more than 30 minutes, please contact support.  Each region will receive warnings starting 5 minutes before that region is restarted.

Please check back for further updates.

[Aug 5th, 11:05pmPDT] Voice is now back online. The issue has been resolved.

[Aug 5th, 9:18pmPDT] There will be emergency voice maintenance at 3am PDT. We are currently experiencing possible issues with voice at the moment. Please check with the blog for updates.

[POSTPONED 6:07pm PDT] - The restart of group chat processes has been postponed. We do not currently have a set date or time for when it will occur. We will update this blog when more information is available.

[4:57pm PDT] - We will be restarting the group chat processes between 5pm and 6pm (PDT) this evening in order to activate some changes that will improve our group chat data collection. Unfortunately, this will close all active group chat sessions. To rejoin a group chat, you’ll first need to close the group chat window. Thank you for your understanding.

We are currently experiencing a processing issue causing the delay of US$ payouts to residents.   Due to the backlog, payout turnaround is five to ten days, rather than the usual five.  The situation will return to normal as soon as possible.

[RESOLVED 2:38pm PDT] - The PayPal issues were resolved and should no longer be an issue. If you find you have any remaining issues, please contact Billing.

[11:32am PDT] - There are currently issues with any processes which require PayPal usage. These include buying L$ as well as credit processing/account payments. PayPal is working to correct these issues as soon as possible. Payment notifications may also be delayed or fail to show the name of the payer. Please watch this blog for further information as it becomes available.

[01:14am PDT] All login issues have now be resolved.

[10:55pm PDT] We’re experiencing some intermittent login issues due to a third party vendor network upgrade. If you receive an error message while attempting to log in, please wait a few minutes and then try again. We will post more details as they become available; check back for updates.

[UPDATE 1:30 pm PST] The registration process has been repaired. Activation emails contain working links.

[July 29th 11:56 AM PST] At this time, activation emails are linking to a page which is currently blank. We are working to correct this issue, please watch this post for updates.

[RESOLVED 1:53pm PDT] The process for stipends has completed and all eligible accounts should have received their payment by this time. Please contact Support if you have any remaining issues with your stipend payment.

[7:30AM PST] The process that runs the weekly stipend payout has been delayed. We are working to find the problem and finish running the payout for all premium accounts. They should finish later today. We will resolve this post when the process is complete or add updates if we run into further problems.

On Monday, 3 August, we’ll be turning off some old IP addresses on our login servers.

These IP addresses have not been in active rotation for over a month, meaning that Residents who use “normal” login procedures won’t notice their loss.

However, we’ve identified a small number of Residents who, at their own initiative or as an attempt to debug various Support requests, have customized their logins with the –LoginURI command line option to point to one of the decommissioned addresses.

It is also possible that some Residents will see their logins failing due to stale DNS caches, either local to their machines or (less likely) maintained by their respective ISPs.

We’ve emailed those Residents who have been employing direct logins.

In the event you find your own login disrupted next Monday, we suggest you first refresh your local DNS cache.

Windows users

• go to Start -> Run, type in: cmd and press Enter.
• At the prompt, type:  ipconfig /flushdns and press Enter.  (Please note the space between the first word and the slash.)
• When the prompt returns in the window, close it with the widget or type: exit and press Enter.

OS/X users

• Start an instance of Terminal.app (if you haven’t docked the app, it will usually be located in the /Applications/Utilities path on your boot drive.)
• At the prompt, type:  dscacheutil -flushcache and hit Return (please note the space between the first word and the hyphen).
• Close the window using the widget or by typing: exit and hitting Return.

Linux users will need to research the documents for their distro. Some distros use some variation of: /etc/init.d/nscd restart in a terminal window. If you try to correct persistent DNS errors and these steps are unsuccessful, the next suggested step is to file a service ticket with your ISP, requesting them to refresh their DNS pointers to secondlife.com. (Again, it’s unlikely that this step will be necessary, but it could happen.) Please file a support ticket if these steps don’t resolve DNS errors when attempting to log in.

[Resolved @ 9:00 am PDT] Live chat support has been fully restored. Thank you again for your patience.

[Updated @ 7:45 am PDT] The engineers are continuing to investigate the technical difficulties with live chat support. Your patience is greatly appreciated while we try to resolve this issue.

[Posted @ 6:55 am PDT] We are aware that there is an issue with live chat support. We are investigating the matter and hope to resolve this issue shortly. Thank you for your patience. Please check back here for further updates.

[JULY 26, 2009, 10:30AM PDT] Solution Provider Directory submissions and login problems have been resolved.

[July 26, 2009, 5:00AM PDT] Solution Provider Directory submissions and logins have been temporarily disabled.  We will let you know as soon as there is anything new to report!

[RESOLVED 4:33pm PDT] - The XStreetSL site is now accepting image uploads again. Some items went unavailable during the problem period because the servers couldn’t see their images. IF you experience this issue, please reset your magic box(es). Otherwise they should come back automatically over the next 2 days. More information is available on the XStreetSL forum: [https:]

[2:00PM PDT] XStreet is currently experiencing a problem accepting image uploads for listings on the site. Engineers are working on resolving this issue. We will post updates as they become available.

[RESOLVED July 24, 2009, 4:45AM PDT] Service to these websites has been restored.

[July 24, 2009, 5:00AM PDT] We are aware that XStreetSL.com and OnRez.com are both temporarily unavailable.  Our web team is investigating and we will restore service to these sites as quickly as possible.

[RESOLVED 04:00am PST] The power has been restored, and XStreetSL.com is back up and ready for business.

[UPDATE 03:07am PST] Unfortunately, that did not work out as planned, and we have lost power again. A technician is on their way to replace the faulty part, and we will let you know how we progress from there on.

[UPDATE 02:30am PST] It appears a power outage has knocked over some loadbalancers. We have restored power to the affected facilities and are now working to get all servers back online.

[July 23, 01:15am PST] Both XStreetSL.com and Onrez.com are currently not available, our web team is on the case to fix as quickly as possible. We will update you with any news as we get them.

[RESOLVED 03:26am PST] The wiki has been behaving well over the last couple of hours, and is fully functional.

[UPDATE July 23, 00:40am PST] We have done some testing, and the wiki appears to be back to normal functionality; we will keep monitoring it for a while longer, though.

[July 22, 2009 10:38pm PST] The Second Life Wiki  http://wiki.secondlife.com/ is currently experiencing intermittent issues. You may see errors as you try to access it. Technicians are working on the issue and we will have it back up as quickly as possible.

[Resolved July 30, 11:18 AM PDT] The rolling restarts have completed. Thank you for your patience.

[Updated July 30, 7:15AM PST] Today’s rolling restarts are about to begin.  We’ll be restarting the other half of the regions.

[Updated July 29th, 10:44AM PST] Today’s rolling restarts have been completed.

[Updated July 29th, 7:15AM PST] Today’s rolling restarts are about to begin.  We’ll be restarting half the regions.

[Updated July 27th, 7:36PM PST] The rolling restarts scheduled for tomorrow (July 28th) morning has been canceled.

We will be deploying server version 1.27.1 to Second Life during the week of July 27th.  The rolling restart is scheduled to look like the following:

* Tue. 07/28 7am - 11am : a pilot roll of ~500 simhosts (canceled)
* Wed. 07/29, 7am -12 pm : half of Second Life will be restarted
* Thu. 07/30, 7am -12 pm : the remainder of Second Life will be restarted

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods.  Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade.  If your region stays down for more than 30 minutes, please contact support.  Each region will receive warnings starting 5 minutes before that region is restarted.

Please direct comments or questions about the rolling restart to the "Rolling Restart to deploy 1.27.1" thread in the forums .

The web Support Portal will be down from midnight to approximately 7am Pacific on Monday morning for an upgrade. During this time frame, the Knowledge Base, Live Chat and ticketed Support systems will be unavailable.

Portions of the Knowledge Base are available on the wiki, which will be unaffected by this downtime.

We apologize for the inconvenience.

[9:30pm PDT] We are aware of potential problems with Bulk Permissions as documented in https://jira.secondlife.com/browse/SVC-4444. We are investigating. We recommend that creators and merchants avoid using the Bulk Permissions feature until further notice. More details will be available when we have them.

This issue has been resolved. Our voice vendor experienced a database failure. Voice should be working as normal.

[Posted @ 2:05 pm PDT] Some residents are reporting voice issues in certain regions. We are investigating the situation and working towards a resolution. Thank you for patience. Please check back for further updates.

Payment processing will be undergoing scheduled maintenance Today from 5pm to 7pm PDT.  During this time credit/debit transactions will be unavailable, including registration and update of card information as well as card-funded LindeX transactions.  Please note that PayPal will still be available for transactions.

[Resolved @ 8:29sm PDT] Logins have been restored to full working order and Xstreet is now accessible. Thank you for your patience.

[Posted @ 8:09am PDT, July 18, 2009] There have been some reports of login failures. We are investigating this issue and should expect the situation to be resolved within 30 minutes.

Additionally, Xstreet is experiencing some access difficulties.

Thank you for your patience while we resolve these issues. Please check back shortly for further updates.

[Resolved 6:13pm PST] The issue was identified and corrected.  Teleports are working.

We are aware some teleports are failing and are looking into the issue.  We will update as needed.

Second Life Blogs (blogs.secondlife.com) will be down tonight, July 14th between 6p and 7p PST for a scheduled maintenance on servers.  We will update once completed.

[RESOLVED 07:50am PDT] - The Land Store is operating normally again.

Residents may experience a slight delay in receiving their purchases form the Land Store at the moment.  If this is the case please try again in 15 minutes or contact the Support Team.

[4:19pm Pacific] Live chat has performed properly during the past half hour of load testing. Thank you for your patience this afternoon.

[3:48pm Pacific] Live chat is back online, but we’re still testing and monitoring for further problems.

[3:15pm Pacific] Live Chat continues to be offline.  More information will be posted to this blog as it becomes available.

[2:10pm Pacific] Live Chat continues to be offline. We are working to have this issue resolved as soon as possible.

[1:16pm Pacific] Second Life Live Chat is offline. Our providers are continuing to correct the issues with live chat as quickly as possible.

[1:00pm Pacific] Second Life Live Chat is back online. Chat delays may occur as engineers continue to do further repairs to bring all functionality online.

[12:45pm Pacific] Second Life support Live Chat continues to be offline as our providers engineers work on the problem.

[11:50AM Pacific] Second Life support Live Chat continues to be offline as our providers engineers work on the problem.

[10:30AM Pacific] Second Life Support Live Chat will be offline temporarily as we address a problem with the system. Please keep checking here for further updates.

[Sun July 12, 04:56am PDT] Some regions haven’t been included in our nightly traffic calculations and might show traffic as ‘0′. We are aware of this, and have taken measures to ensure that with our next run of traffic calculations those regions will be updated.

[Resolved 0151 July 12, 2009] Group search is now working

We are aware that group search is failing and working on a resolution.

[Resolved 04:34am PDT] The problem has been fixed. If you tried to change your payment information during the outage, you might have to re-enter your information.

We are currently having a reoccurance of problems connecting to Global Collect. This can cause updates to credit card info to not go through and some payments to fail. We will update as soon as we have more information or the problem has been rectified.

[RESOLVED 08 July 2009 7:45pm PDT] The group chat issue has been resolved and all group chats should work as intended now.  Some individuals may yet experience some similar issues, but they should be completely resolved or at the very least, greatly reduced. We will be keeping an eye on group chat functionality to make sure it is truly resolved. If the issue does return, please see this blog for more information.

[7 July 2009 11:03 PM PDT] We are currently investigating issues with long delays on group chats, silent failures in lieu of popup errors, and other error messages. This may continue overnight as the engineering team gathers data and runs tests; we will keep this blog updated tomorrow as more information becomes available.

We are aware that certain residents and groups may be experiencing silent failures of chat (no error message is given of the failure). We are looking into this problem.

[2.40 AM Pacific] We are experiencing errors with our Global Collect System. This will mean that some credit cards will fail to update and some purchases will not go through. We are working to rectify this as soon as possible and will keep you updated on the situation.

[11:32 pm Pacific] Our provider has cleared the issue. Cards which have been stuck in pending status will clear as the backlog is processed.

[9:37 pm Pacific] Some Residents attempting to update their payment information at the Website are stalling in “Pending” status when supplying credit or debit cards.  Ordinarily, a valid card update is activated within a few minutes of supplying the information.

We’re investigating as quickly as possible.

[RESOLVED 2009-07-17 5:00am PDT]: We’ve completed the rolling restarts.

[UPDATE 2009-07-16 10:20pm PDT]: Full grid rolling restart currently in progress.

[UPDATE 2009-07-16 7:30pm PDT]: Full grid rolling restart commencing at 8pm PDT tonight.

[UPDATE 2009-07-15, 10:37am PDT]:The first half of Second Life regions have now been restarted.

[UPDATE - 2009-07-15, 06:00am PDT]:  The first half of Second Life regions are now being restarted.

[UPDATE - 2009-07-14, 08:53am PDT]:  The Pilot roll has now been completed.

[UPDATE - 2009-07-14, 07:30am PDT]:  The Pilot roll is recommencing now.

[UPDATE 2009-07-14, 00:43am PDT]:  Due to some bugs that showed up in the pilot regions and the desire for additional testing, we’ll be re-doing the pilot roll on Tuesday. If that goes well, we’re planning on doing the full roll on Wednesday and Thursday.

[11:25am PDT, July 9 2009] The Pilot roll is complete.

[7:29am PDT, July 9 2009] The pilot roll has begun.

[11:38pm PDT, July 8 2009] We will be deploying server version 1.27.0 to Second Life during the weeks of July 6th and July 13th. The rolling restart is scheduled to look like the following:

* Thu. 07/09 7-10am : a pilot roll of ~500 simhosts - Complete
* Tue. 07/14, 7-12AM : half of Second Life will be restarted
* Wed. 07/15, 7-12AM : the remainder of Second Life will be restarted

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods. Most regions should be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade. If your region stays down for more than 30 minutes, please contact support. Each region will receive warnings starting 5 minutes before that region is restarted.

Here are the “1.27 Release Notes”

Questions and discussion about the rolling restart may be directed to the “Rolling Restart to deploy 1.27.0″ thread in the forums.

[18:30pm PDT RESOLVED] The login issues have been resolved.

[17:48pm PDT] Please be advised that we are currently experiencing database issues which may cause issues with logins. We will update this blog with more information as it becomes available, and our engineers are working on the problem now. Please consider waiting on major purchases, land transactions, or rezzing no-copy objects until this issue has passed.

[5:59 AM Pacific] All is back to normal, credit card processing has been fixed.

[4:14 AM Pacific] Our apologies. This issue is not as resolved as we had hoped. We are continuing to work with Global Collect to find the source of the problem and repair it.

[3.39 AM Pacific] This issue is now resolved.

[3.05 AM Pacific] We are experiencing errors with our Global Collect System. This will mean that some credit cards will fail to update and some purchases will not go through. We are working to rectify this as soon as possible and will keep you updated on the situation.

[Resolved @ 12:33 pm PDT, June 30, 2009 ] The inventory server rotation has completed. Residents on that particular server should be able to login, although, it may be slow at first. Thank you for your patience.

[Posted @ 11:49 am PDT, June 30, 2009 ] We will commence an inventory server rotation shortly. A small subset of residents will be affected. Those on that server will be automatically logged out and will not be able to login temporarily until the work is complete. Those who are trying to login will not be able to login until the work is complete. Thank you for your patience. Please check back for further updates.

[UPDATE 3:33pm Pacific] The wiki is experiencing intermittent downtime. We are working to restore it fully.

[UPDATE 12:07pm Pacific] Our hosting service has applied another fix and our testing shows no current problems.

[UPDATE 11:17am Pacific] We’re seeing more sporadic behavior from the public wiki.  We’re investigating.

[UPDATE 5:22am Pacific] The wiki appears to be fully accessible again.

[UPDATE 4:49am Pacific]  We are still pursuing a resolution to this issue

[UPDATE 3:49am Pacific]  We are continuing to monitor this issue.

[2:49am Pacific]  Our Second Life wiki is intermittently unavailable at this time and we are working to have this restored to continued operation.

[1st July 2009 - 2:37am Pacific]  Traffic should now be updating again.

[8:50pm PDT] The traffic issues remain, and it will not update tonight.  The estimated time of resolution is not known at this time.  Updates will be posted here as more info becomes available.

[7:25pm PDT] Our traffic statistics are calculating slowly, and are not expected to finish today. Traffic statistics should resume normal updates tomorrow.

[RESOLVED 3:51 pm Pacific] The Public Issue Tracker has been restored to service.

The  Second Life Issue Tracker is currently still not accessible, our hosting company is currently working on the problem.  Please keep monitoring  this blog post for further updates.

[13:25 PDT]  We are aware that the Second Life Issue Tracker is currently not accessible and we are working to rectify the problem. Please monitor this blog for updates.

[June 27, 2009 4:40AM Pacific] Traffic has been recalculated.  If your traffic still shows as 0, please try clearing Second Life’s cache then relog.

[June 26, 2009 7:35AM Pacific] Our traffic statistics are still in the process of calculating. If your parcel or region was showing 0 traffic yesterday, it may still reflect that today until the process completes. Will we update this blog when the traffic updating is complete.

[RESOLVED 11:54am PDT] - The credit card issues have been resolved and you may conduct transactions or update information as normal.

[POSTED 11:22am PDT] - There are issues currently which may present failed transactions and/or information updates for credit card usage. This is being addressed and will be resolved as soon as possible, in the meantime you may wish to avoid any credit card usage. Please watch this blog for updates.

[RESOLVED Jun 29th 4:00pm PDT] The Land Store is now open and operating normally.

[UPDATE 7:00pm PDT] As previously reported, our Land Store at http://land.secondlife.com is continuing to have some issues with the automated delivery of regions. We do not expect this to be resolved before Monday.   As a result, regions are being delivered manually by our staff. If you’ve purchased a new region through the Land Portal and it has not been delivered within 4-8 hours, please contact our Support team at http://secondlife.com/support for assistance. Thank you for your patience.

[UPDATE 3:30pm PDT] - We apologize for the length of time this issue has been outstanding, as it is still unresolved. Work continues to bring back normal operations as soon as possible. Please watch this blog for further updates.

[UPDATE 12:54pm PDT] - While there is no time frame for a resolution, we are continuing to address this issue. Please check back for further updates.

[POSTED 9:48am PDT] - There have been reports from some Residents that the Land Store is experiencing some delays with region creation after purchase. We are aware of the issue and working towards a resolution. Please check back soon.

[RESOLVED 6:30PM PDT] Our technicians have confirmed that there was an issue processing traffic statistics for some regions during the nightly calculations. Traffic status for those regions will return to normal tomorrow after the update. We apologize for the inconvenience.

[5:11PM PDT] There is currently an issue with traffic updates, affecting approximately 3000 regions. Traffic is showing as 0 at those locations or showing up in Search - Places. Our technicians are aware of the issue and working on a resolution. Please stay posted for updates.

[RESOLVED 10:40] We have identified and corrected the problem interfering with inworld services.  You are all clear to resume normal rezzing and transactions.

[10:15]  We are currently investigating an issue interfering with a number of inworld services, including inventory, rezzing and avatar appearance. Please refrain from making any transactions or rezzing valuable or no-copy items until the all clear is given.  Please monitor this post for further information and updates.

[RESOLVED 25 June 10:45] We have corrected the issue.  Thank you for your patience.

[UPDATE 24 June 3:08 pm Pacific] We’re still tracing our way through this issue. We’ll update when we have some news.

[23 June 5:10 pm Pacific]  We’re seeing some Resident comments take entirely too long to appear on main blog articles; hours in some cases. This isn’t the result of moderation.  We’re tracking this down as quickly as we can, but it may be tomorrow before we have some answers.

[2:00am Pacific - RESOLVED] The maintenance should be complete now.

[1:00am Pacific - UPDATE] The maintenance should be under way now.

[12:20am Pacific]  The Second Life Wiki (wiki.secondlife.com) will be offline for approximately 2 hours at 1:00am Pacific on Wednesday 24th June 2009 for software upgrades.  We will update information should this change in any way.

[RESOLVED 10:03am PDT] - The current issues have been resolved. However, the longer term group chat issues are not included in this resolution. For more information on those issues please see the Jira, Wiki and/or Blog posts about the matter:

[jira.secondlife.com]
http://wiki.secondlife.com/w/index.php?title=Group_Chat_Failure_Messages
https://blogs.secondlife.com/community/technology/blog/2009/06/15/improving-the-quality-of-group-chat

[08:05] We are investigating an issue interfering with group chats which may cause them to fail or require multiple send attempts. Please monitor this post for updates and further information as they become available.

[RESOLVED] Scheduled wiki and Public Issue Tracker maintenance have been completed.

wiki.secondlife.com and jira.secondlife.com (the Issue Tracker) will be undergoing maintenance at 3:30 pm Pacific on monday 22th June for approximately 15 minutes.

There may be some intermittent downtime during this period and the both sites will be read-only. Please refrain from creating or editing articles or issues during this time frame.

[RESOLVED 16:19 PDT] Logins are returning to normal.  Please be patient logging in until the queue clears, and try riding a single login all the way through to connection.

[16:03 PDT] We’re currently investigating an issue that is causing some residents to not be able to log in. We will post here when more information becomes available.

[RESOLVED 2:56PM Pacific] Our voice vendor has restarted their servers.

[June 21st 12:03PM PACIFIC] We have been made aware that some residents are experiencing issues with voice.   We are currently working to have this resolved as soon as possible.

[RESOLVED June 21, 2009, 8:34 PM Pacific]  All regions have been returned to service.  The land store is open again.

[UPDATE June 21, 2009, 6:40 PM Pacific] We’re down to under one hundred regions waiting for reactivation.  Hopefully this will be over very shortly.

[UPDATE June 21, 2009 6:22 PM Pacific] We’re still working to bring a few hundred regions online. We’ve found ourselves short on simulator hosts today due to a power issue in one of our colocation centers. As the colocation center gets that issue rectified, we’ll use the available hosts to return existing regions to service first, and then reopen the land store.

[UPDATE June 21, 2009 4:30pm Pacific] The Land Store will be offline while we work to restore the remaining regions.

[UPDATE June 21, 2009 2:25pm Pacific] Our technicians are working quickly to restore remaining regions. Please refrain from restarting regions as this could cause a longer downtime for that region during this maintenance issue.

[UPDATED June 21, 2009 1:43pm Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATED June 21, 2009 12:56pm Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATED June 21, 2009 11:35AM Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATE June 21, 2009 10:42AM Pacific] We are continuing to work on restoring regions at this time.

[UPDATE June 21, 2009 9:38AM Pacific] Multiple regions are still offline at this time, we will continue to enable all remaining regions as soon as possible.

[UPDATE June 21, 2009 8:04AM Pacific] Our technicians are continuing to work to restore service to all remaining regions.

[UPDATE June 21, 2009 6:45AM Pacific] Our technicians are working to restore service to 70 remaining regions.

[UPDATE June 21, 2009 5:00AM Pacific] The number of regions that are offline is down to 250.  Our technicians are working to restore service to these.

[UPDATE June 21, 2009 3:00AM Pacific] We’re working to restore service to approximately 450 regions.

[UPDATE 12:30AM PDT] Maintenance is ongoing, we are slowly bringing services back online. We had some secondary hardware failures, we hope to be back to full speed soon.

[UPDATE 22:10 PDT] The emergency maintenance has begun.

[UPDATE 21:50 PDT] The emergency maintenance has been moved forward, and will be starting at 22:10 (10:10pm). We will post here as further updates are available.

[June 20 18:37pm PDT] - At approximately 23:00 (11:00pm) PDT tonight, one of Second Life’s primary datacenters will be undergoing emergency maintenance. For an orderly maintenance cycle we will be bringing approximately 5,000 regions offline during the course of this maintenance starting at 22:30 (10:30pm) PDT. We will keep this blog updated with more information as it becomes available. Thank you.

[RESOLVED 12:40AM PDT] Regions currently offline will be fixed with Datacenter Maintenance - [status.secondlifegrid.net]

[UPDATE 21:17 PDT] We continue to work at bringing the offline regions back online. There are just under 1400 regions offline now. Thank you for your patience, and please check back for information.

[UPDATE 20:15 PDT] Regions are still coming back online, and there are currently about 1400 regions we are working on bringing back up. Please stay posted for details.

[UPDATE 19:10pm PDT] - The work on the remaining regions is continuing. There are now ~1,500 regions that are still down. More information will be posted to this blog as it becomes available.

[UPDATE 18:00pm PDT] - Logins are now recovered; however, if you find you cannot login, please try logging in directly to a different region as the one you are attempting to log in to may still be down. The engineers are currently working on recovering approximately 3,500 more regions, with many already back up. Work will continue until this has been resolved for all regions, with any updates available here in this blog.

[UPDATE 16:49pm PDT] - Logins should now be recovering. If you cannot login then please wait for a few minutes, then try again. A little over 300 regions should be coming back online shortly, while work continues on bringing back the rest of the regions which remain down. Please watch this blog for further updates.

[UPDATE 15:50] ~4000 regions are currently inaccessible. Logins remain problematic. Please monitor this post for updates.

[UPDATE 15:45] Logins are again being affected by the issue. Some residents may be able to connect to Second Life however the login process may take a while. If you’re unable to connect, please wait a couple minutes before trying again. For those who are connected, some services may not be working at the moment. These include teleports, the world map, display of L$ balances, search, inventory and purchases made inworld. Some additional regions may also be inaccessible or showing as offline.

[UPDATE 3:05pm PDT] We still have about 300 regions to go. If your region is one of those still affected please be patient. We will have it back up as soon as possible.

[UPDATE 1:35pm PDT] Work continues on bringing the remaining regions back up as soon as they can be. Please watch this blog for more information.

[UPDATE 12:11pm PDT] Our engineers are continuing to address the regions that are still down. There are about 500 regions left at this point. Further updates will be posted here as they become available.

[UPDATE 11:02am PDT] The work is continuing on resolving the issue for all affected regions. There are about 600 regions left to bring back. Please watch this blog for further updates.

[UPDATE June 20, 2009 9:11AM Pacific] Our engineers are still working on bringing up around 2000 remaining regions. The region outages were caused by a power outage in one of our colocation facilities.

[UPDATE June 20, 2009 8:11AM Pacific] Logins are once again available however there are still approximately 6,000 regions unavailable at this time. See this post for updates as we resolve the issue.

[UPDATE June 20, 2009 7:05AM Pacific] Approximately 8,000 regions are currently inaccessible. Our technicians are working to restore service to them and other Second Life services.

[UPDATE June 20, 2009 6:10AM Pacific] Some residents may now be able to connect to Second Life however to login process may take a while. If you’re unable to connect, please wait a couple minutes before trying again. For those who are connected, some services may not be working at the moment. These include teleports, the world map, display of L$ balances, search, inventory and purchases made inworld. Some regions may also be inaccessible or showing as offline.

[June 20, 2009 5:30AM Pacific] We’re aware that some residents get an error message instead of being able to connect to Second Life. This is being looked into and we’ll post any updates to this blog post as they become available.

Some regions are experiencing traffic calculation issues.  Traffic may show as 0 at these locations.  Our technicians have already resolved the issue and calculations will update within 48 hours.

[RESOLVED 14:48] The Public Issue Tracker, [https:], is back online and functioning normally again.

[14:37] The Public Issue Tracker, [https:], will shortly be taken down for maintenance to correct the problem with the logins.  The site will be unavailable during this time.  Please monitor this post for updates.

[13:12]  We are currently investigating an issue preventing residents from logging into the Public Issue Tracker, [https:].  The site is otherwise unaffected and browsing is functioning normally, however features requiring a login such as creating or commenting on issues are currently unavailable.  Please monitor this post for updates.

[RESOLVED 10:31am PDT] - Credit card transaction issues have been resolved. If you have attempted to update your information during this outage, you may need to re-enter the information. Please contact support if you need additional assistance.

[9:29am] There have been some reports of credit card transaction failures. We are aware of the problem and working to correct this issue. Please check back for further information.

[RESOLVED 8:46 PM PST] The Land Store has been re-opened and is functioning as normal once again.

Our techs are investigating a possible problem with land store purchases.

We expect to have this resolved soon, but in the meantime if you’ve bought a region and experienced difficulties, please contact Support.

[RESOLVED June 13, 2009 7:35AM PDT] The Second Life Blogs are back online at this time.

[June 13, 2009 5:50AM PDT] A quick update to let you know that we’re still working with our vendor to resolve this issue.

[June 13, 2009 2:30AM PDT] We’re aware that the Second Life Blogs are currently unavailable and are working to bring them back online.  This post will be updated as any updates become available.

[RESOLVED] 11:41pm PST - The regions affected are stable and functioning properly

Some regions are unable to see their neighbors, and Residents are unable to cross some region borders. (Direct TPs still work.)  Also, avatar counts on the map tool are not reporting correctly.  We are currently investigating the issue.

[RESOLVED 6:56PM PST] We have been informed that the issue with the Public Issue Tracker has been corrected and normal functionality restored.

[6:10PM PST] We are looking into reports of problems with the Public Issue Tracker (PJIRA). Please do not attempt to create new issues until we provide an all-clear on this blog.

[RESOLVED 7pm PDT]  The regions are back online now.

[UPDATED 5:35pm PDT] Our technicians are still working to resolve this issue, and we will post an update when it is resolved.

Due to a switch failure approximately 200 regions are offline at the moment. Our technicians are working to fix this asap.

[RESOLVED 12:51 Pacific] ALL CLEAR on asset operations. Please be patient logging in for the next few minutes as we clear the login queue.  Try to keep your place in the queue by riding a single login attempt all the way in, rather than canceling and restarting.

[UPDATE 12:46 pm Pacific] Residents attached to at least one inventory server are currently unable to log in.  We’re correcting the problem as quickly as possible. Please watch for updates.

[12:38 pm Pacific] Residents are reporting that some attempts to rez objects from inventory are silently failing.  We’re checking the processing on all inventory servers. Please refrain from rezzing no-copy objects until we give the all clear.

[Resolved] ~160 regions that were offline should now be back online. Thank you for your patience.

[June 9, 2009 @ 9:07 am PDT) Due to a switch failure approximately 160 regions are offline at the moment. Our technicians are on the move to fix this asap.

[RESOLVED 11:09am PST] Region Visibility and Crossings are back to normal

[REOPENED 10:43am PST] This issue has cropped up again, we are working to fix it as soon as possible and will let you know soon as we have news.

[RESOLVED 12:06 pm Pacific] We needed to restart one server to clear the issue.

*****

Some regions are unable to see their neighbors, and Residents are unable to cross some region borders. (Direct TPs still work.)  Also, avatar counts on the map tool are not reporting correctly.  We’re still assessing the problems.

The issue tracker (jira.secondlife.com ) will be down for 15 minutes on Monday at 1AM Pacific. We are performing some maintenance.

[7:17pm PST June 6 2009] This issue has been resolved, and regions should all be able to see each other now. Thank you for your patience.

[6:41pm PST June 6 2009] Currently some regions are not viewable from the next region inworld. This will affect avatar region crossings, cross-region builds, and communication between regions. This is only a temporary effect and will not cause permanent damage to cross-region builds.

Although the region may appear to be offline from inworld, they will still be accessible by direct teleport. Our technicians are aware of the issue and working to resolve it. Please check back for details.

The auction pages of the Second Life website are currently not responding correctly to some viewers. For example, Safari is not working properly while Firefox is. If you see source code when visiting the auction pages, please use a different browser and you should be able to see them properly.

We expect this issue to be resolved on Monday. If there are any updates before then, they will be added to this blog.

11:30PM PDT - Intermittent problems with regions has now been resolved.

7:20PM PDT - Regions are dropping out of view, they’re not actually going offline (they’re able to be TPed to) and they should restore themselves within 10 minutes or so, they can also reboot the region if its privately owned if they want to speed up the process

The Jira service has now been restored! Thank you for your patience.

[posted @ 2:32 pm PDT] We are aware that Jira has become unavailable and are currently investigating the issue. We will update this post with for information as it becomes available. Please check back for more information.

[RESOLVED 9:36am PST] - The Xstreet SL Marketplace maintenance is completed and the service is available.

The Xstreet SL Marketplace is currently unavailable due to emergency maintenance. We expect it to be available again in approximately 15 minutes. Please watch this blog for any updates.

jira.secondlife.com (the issue tracker) and wiki.secondlife.com will both be down for maintenance 1-2am Pacific on June 4th.

[RESOLVED 10:45am PST] - The Xstreet SL maintenance and upgrade has been completed and all functions have been re-enabled.

Xstreet SL will be undergoing maintenance for 20-25 minutes beginning at 9:00am Pacific, June 3rd. During this time Marketplace purchases and L$ withdrawals will be disabled.

During this outage we’ll deploy an upgrade which will improve the way failed item sales are represented, and it will help us gather information to better troubleshoot item sale and delivery problems when they arise.

Xstreet SL will be undergoing maintenance for 20-25 minutes beginning at 9:00am Pacific, June 3rd. During this time Marketplace purchases and L$ withdrawals will be disabled.

During this outage we’ll deploy an upgrade which will improve the way failed item sales are represented, and it will help us gather information to better troubleshoot item sale and delivery problems when they arise.

[RESOLVED 11:26am PST] The issue has been identified and resolved. All affected regions should now be accessible again.

[UPDATE 10:14am PST] While work continues on resolving this issue in entirety, the affected regions are being restored one at a time. Please watch this blog for any further information.

We are aware that there is a small portion of the grid that is currently inaccessible (can’t log in, can’t tp there, etc). Our engineers are currently working to correct the situation. Please watch this blog for further information.

[Updated, May 29, 2:45pm Pacific] The scheduled maintenance planned for tomorrow night has been canceled.  We will update this blog if it gets rescheduled!

The Support Portal will be undergoing some maintenance on Saturday, May 30th from 9-10pm, Pacific.

During this time period, the KnowledgeBase, Live Chat and Ticket submission will not be available, but our telephone lines will be unaffected.

[RESOLVED 11:29am PST] - The Issue Tracker (Jira) is once again available and can be used as normal.

The Issue Tracker, also known as Jira, is temporarily unavailable. We are currently working to bring the service back up. Please keep an eye on this blog for further updates as they become available.

[RESOLVED 1:26pm PST] Maintenance is complete and the Support Portal is now available.

[UPDATE 12:18pm PST] The maintenance is running a little long, but should be completed shortly. Please watch this blog for further updates.

[Updated 10:17] The maintenance has been extended and the Support Portal will remain offline until 12:00.

[Updated May 27, 7:04am PST] The maintenance has started, and our Support Portal is now offline.

As per our notice a few days ago, the Support Portal will be undergoing some maintenance today, Wednesday 27th May from 7:00am until 10:00am Pacific.

During this time period, Live Chat and Ticket submission will not be available although telephone calls will be unaffected.

[RESOLVED 5:14am Pacific] The Support Portal is back to normal operation.

[UPDATE 4:15am Pacific] We have reports that ticket submissions may also time out. If this happens, please keep trying. Your ticket will go through eventually.

The Live Chat feature of our Support Portal is offline at present.  We are working with our third party provider to resolve this as quickly as possible.

Tickets and Telephone calls remain unaffected meantime.

[12:55am Pacific - RESOLVED]  The planned maintenance work is now complete.

[IN PROGRESS] We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that work behind the scenes onto a new network.

There will be a 6 hour stretch of maintenance work on Tuesday, the 26th of May, between 12am and 6am PST.

Expected impact on your Second Life Experience includes but is not limited to:

• Regions restarted during this time frame won’t restart
• Logins might be affected
• Downloading new viewer software and other website functionalities might be affected
• Crossing between neighbouring regions as well as visibility of neighbouring regions might be impaired
• L$ transactions might not work or not be shown immediately; please do not make any L$ transactions during this time frame
• XML-RPC might be not working
• Estate settings  might appear wrongly or not responsive to changes
• Some Residents might get logged off

We will update this post when the work starts, keep you posted during the process or if we run into any unexpected difficulties, and when we’ll be finished.

We have several rounds of brief maintenance to various Second Life features coming up this week.  

• Today (May 24), 5-7 pm Pacific - Payment Processing:  Some US-based Residents and all non-US Residents may experience intermittent disruptions in their ability to update accounts or to transact on LindeX during this period.  Interruptions are almost certain between 5:20 pm and 5:40 pm Pacific.
• Tuesday (May 26), 3-5 am Pacific - Voice chat:  Residents may experience intermittent disruptions while using voice chat during this time. The Beta Grid will not be affected.
• Wednesday (May 27), 7-10 am Pacific - Support Portal:  Tickets and Live Chat will be unavailable during this time.
  
• Saturday (May 30), 9-10 pm Pacific - Support Portal:  Tickets and Live Chat will be unavailable during this time.

If the scope or schedule of any of the work changes, we’ll update here.

[10:30PM] Emergency maintenance is needed

We are performing unscheduled maintenance to Second Life that may cause a disruption to inworld services such as teleporting.  Keep any eye on this post for any updates as they become available.

[RESOLVED] Scheduled wiki and Public Issue Tracker maintenance has been completed.

wiki.secondlife.com and jira.secondlife.com (the Issue Tracker) will be undergoing maintenance by our third party vendor at 1:00 am Pacific on Wednesday 20th May for approximately one hour.

There may be some intermittent downtime during this period and the both sites will be read-only.  Please refrain from creating or editing articles or issues during this time frame.

[RESOLVED 9:06PM Pacific] The problems have been resolved, and you should no longer encounter any problems in signing up a new account.

[7:26PM Pacific] Due to a technical problem, the new user registration might throw intermittent errors, such as “Registration not available”. We are aware of the problem and working to fix it. In the meantime, if you run into any problems registering a new account, please contact our support team to submit a ticket. Please be sure to use the same email address and give the full name of the account you tried to register.

[RESOLVED] Logins are back to normal and all inworld services appear to be functioning normally.

[UPDATE] We recommend refraining from making inworld transactions and rezzing no copy objects for the time being.

We are currently experiencing some issues with logging into Second Life.  Some residents may have trouble accessing Second Life.  We are currently working on correcting this issue and will update as more information becomes available.

[RESOLVED 3:20AM PDT May 17, 2009] Traffic calculations have been completed, including all parcels that were missed yesterday morning.

[UPDATE] Due to an error last night, traffic calculations did not complete for about 1/3rd of all parcels resulting in their traffic to display as 0. This problem will resolve itself tonight when traffic is recalculated and all parcels are updated as normal.

Residents have been reporting what appears to be a possible reoccurance of problem with traffic where their parcels erroneously report their traffic as 0. We are aware of the problem and expect the traffic numbers to properly update tomorrow with the next set of traffic calculations.

2009-05-21 10:40AM: The 1.26.4 rolling restart is complete.

2009-05-21 08:20AM: Due to an error in the rolling restart, we accidentally started re-deploying 1.26.3. The rolling restart will start over, deploying 1.26.4. This means that regions already restarted will need to be restarted again. We apologize for this inconvenience.

2009-05-21 05:00AM: Rolling restarts will be in progress this morning from 7AM till 11AM.

2009-05-20 09:50AM: The first-half rolling restarts is complete.

2009-05-20 06:52AM: The first-half rolling restarts is beginning.

2009-05-18 10:05AM: The pilot roll is complete.

We will be deploying server version 1.26.4 to Second Life the week of May 18.  The rolling restart will follow this schedule:

• Mon. 05/18, 9-11AM : a pilot roll of ~3000 regions
• Wed. 05/20, 7-11AM : half of Second Life will be restarted
• Thu. 05/21, 7-11AM : the remainder of Second Life will be restarted

As with all of our server deploys, each region will be restarted once during one of the rolling restart periods.  Most regions will be down no more than 5-10 minutes, although some fraction of the regions will take 20-30 minutes to upgrade.  If your region stays down for more than 30 minutes, please contact support.  Each region will receive warnings starting 5 minutes before that region is restarted.

Release notes describing what is new in server 1.26.4 can be found on this page on the Second Life wiki.

Questions and discussion about the rolling restart may be directed to the comment thread of this post on the Second Life Release Team blog.

[RESOLVED 9:32am PDT] All payment method issues have been resolved and you may enter and/or update both Credit Card and PayPal details as normal.

[UPDATE 7:40] We are receiving reports that entering Credit Card details is being affected by this after all. Please refrain from entering/updating payment information until this issue has been resolve. Please monitor this post for updates.

[06:44am PST] We have been made aware that residents creating/updating their payment method through paypal are not completing successfully.

Please refrain from updating your paypal details until the issue has been resolved. This does not effect Credit Card details, only PayPal.

[RESOLVED 9:25]  Access to SecondLife.com is restored.  Google had difficulties, which in turn affected us due to the services Google provides.

[8:46}  We are aware that some residents may be experiencing difficulty reaching SecondLife.com and the associated websites, including the Support Portal.  This appears to be a widespread problem which is affecting other sites as well as ours.  We are investigating and will update you as soon as more information is available.

[UPDATE 2:19 pm Pacific] Database maintenance is complete and XStreetSL is back online. Product pictures and other merchant assets should display correctly without the need to re-upload them.  

*****

The XStreet servers will be offline for the next 1-2 hours while we perform some database tuning.  We’ll notify you here if the scope or schedule of the work changes.

[UPDATE 7:30 pm] The fix this afternoon corrected various cookie-based errors which were randomly striking Residents attempting to log in to XStreetSL.  Residents with a password length of 17 characters or greater will still need to change to a shorter password (16 characters or under) in order to access XStreetSL. 

[UPDATE 3:50pm]  We’ve pushed a fix to this problem.  If you’re still having trouble logging in, please file a Support ticket for XStreet Login problems (see below) and we’ll assist you.

 

Some resident are reporting problems logging into XStreetSl.com.  Passwords longer than 16 characters are currently not accepted on XStreetSL.com, however we are working to correct this as soon as possible.  In the meantime, affected resident who wish to log in immediately may change their Second Life password at http://secondlife.com/account/password.php.

Residents with any other issues with XStreetSL logins may file a support ticket using Special Questions - Basic or Guest with the ticket type “XStreet Login Problems.”

[COMPLETED 9:39am PDT] The maintenance has completed and all support features are available.

Our support portal vendor has announced two rounds of maintenance this coming week:

• Sunday May 10, 9:00 - 9:30 p.m. Pacific (Completed)
• Wednesday, May 13, 7:00 - 9:30 a.m. Pacific

Support LIve Chat and ticketing features will not be available during these periods. If the scope or scheduling changes, we’ll update here.

[RESOLVED 3:00am PDT May 10, 2009] We’ve recalculated traffic for all regions.

[Update 8:00am PDT May 9, 2009] Traffic is now being calculated correctly and will display normally on the next update. Updates will be posted here as more info becomes available.

[5:00pm PDT May 8, 2009] There is currently an issue with traffic updates, affecting approximately 3500 regions. Traffic is showing as 0 at those locations. Our technicians are aware of the issue and working on a resolution. Please stay posted for updates.

[Resolved @ 4:30PM PDT] Textures on the handful of regions mentioned below should be loading textures normally.

Residents have reported a handful of regions that are experiencing difficulties with textures showing as grey. We are aware of the issue and working towards a resolution.

[Resolved 12:12PM PDT] The investigation has been completed. All inworld services should be functioning normally. Thank you for your patience while this situation was resolved.

[UPDATED 11:47AM PDT] Investigation continues into the database issue.  Although some inworld services may function please continue to avoid making transactions or purchases until the issue is resolved.

[10:49AM PDT]  Logins have been disabled while we investigate a database issue affecting a number of inworld services.  Inworld Search, Teleports, and the main Map are not functional.  Please avoid making transactions and purchases, including XStreetSL purchases, until the issue is resolved.

[RESOLVED May 8, 2009 12:00AM PDT] The temporary issues with group chat have been resolved.

We are experiencing issues with group chat. Our team is looking into the issue, and will have the problem resolved as quickly as possible. Please check back with the blog for further updates

[COMPLETE May 8, 2009 4:10AM PDT] Scheduled voice chat maintenance is complete. If voice isn’t working, you may need exit Second Life and log back in.

[IN PROGRESS May 8, 2009 3:00AM PDT] Voice chat maintenance is now in progress.

We’ve discovered that we need about an hour of quality time alone with our in world voice chat system, in order to tune its performance for the coming weekend. Our plan is to take voice chat offline at 3 a.m. Pacific Friday, 8 May, and to have it restored by 4 a.m. Pacific.  If the scope or scheduling change, we’ll post here.

[RESOLVED May 7, 2009 12:45AM PDT] We’ve completed the maintenance to our network!

[UPDATE May 6, 2009 11:00PM PDT] Scheduled maintenance is now underway.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that work behind the scenes onto a new network.

These moves can cause temporary failures in group chat, or inaccuracies in the online statuses in your friends list.

The server changes will start tonight at 11pm Pacific and run until approximately 2am.

[09:21am PST] All earlier issues have been resolved. If you saw a transaction failure earlier, please go ahead and re-enter your payment info to enable a new transaction to be executed.

[02:50am PST] We have received reports of credit card process failures over the last couple of hours. Our engineers as well as our vendor are looking into the problem, and we’ll update you as soon as we have news.

[RESOLVED 01:54AM PST] The scheduled maintenance is now complete.

[UPDATED 12AM PST] The scheduled maintenance is now in progress.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that work behind the scenes onto a new network.

These moves can cause temporary failures in group chat, or inaccuracies in the online statuses in your friends list.

The server changes will start tonight at 11pm Pacific and run until approximately 2am.

Normal 0 MicrosoftInternetExplorer4

[May 2, 2009 5:25pm PDT] RESOLVED Asset server stability has been restored. Thank you for your patience while we worked to resolve this issue.

[May 2, 2009 3:38 PDT] Our engineers are continuing to work in order to restore stability to the asset server. Please check back for further updates.

[May 2, 2009 2:33pm PDT] Our technical teams are still investigating the cause behind the asset server issues. Please do not rez no-copy objects or make transactions for the moment. We appreciate your patience while we work on this issue, and we will post here when the issue is resolved.

[May 2, 2009 1:28pm PDT] We are currently experiencing asset server issues that may result in objects failing to rez. Please refrain from rezzing no-copy objects or making transactions/purchases until further notice. Stay posted for updates.

[May 3, 2009 12:45am]  The traffic update has been completed and traffic should be updated accordingly.

[May 2, 2009 2:20PM PDT] Traffic is now being calculated correctly and will display normally on the next update. Updates will be posted as more info becomes available.

[May 2, 2009 9:15AM PDT] We are aware that some mainland and private regions are not reflecting correct traffic information. At the present time, we are working to update traffic so that it displays properly.  Updates will be posted here as more info becomes available.

[May 2, 2009 7:20AM PDT] Maintenance by our phone support provider is complete!

[May 2, 2009 5:50AM PDT] Our phone support provider is undergoing maintenance that will last approximately 2 hours.  If your call doesn’t go through, please try back sometime after 8:00AM PDT or submit a ticket from our Support Portal.

[RESOLVED May 2, 2009 6:20AM PDT] The database issues have been resolved.

[UPDATE 18:25] The database issues we were experiencing before have stabilized. There should be no noticeable issues, but we are still pursuing a permanent solution. Please monitor this page for further updates.

[UPDATE 14:06] We are continuing to work to resolve the situation.  Please continue to monitor this post for updates.

We are investigating a database issue which may cause some inworld service disruptions including interfering with searching and teleporting.  This issue may also result in an the error “failure to retrieve” when trying to rez or wear some inventory items.  To be safe, please refrain from rezzing no-copy items until the all-clear is announced.  Please monitor this post for updates and further information.

[RESOLVED April 26, 2009 8:15AM PDT] The regions that went offline because of the power outage have been revived.

[UPDATE April 26, 2009 7:40AM PDT] 90 regions remain offline.

[UPDATE April 26, 2009 7:30AM PDT] 160 regions remain offline.

[April 26, 2009 7:00AM PDT] Approximately 250 regions are inaccessible due to a power outage.  This is being looked into and we will let you know as soon as there is anything new to report.  In the meantime, please don’t rez or derez any no-copy assets.

[RESOLVED 28 April 2:18 pm Pacific]   The original URL format has been restored to pages at world.secondlife.com, e.g. ..world.secondlife.com/group/<group-key>. All pages in this collection are now available and updated per usual. From now on, please continue to use this URL format.

For a limited time, a separate version of the pages with /a/ in the URL, e.g. …world.secondlife.com/a/group/<group-key>, will be available in order to give Residents who took corrective action the time to adjust their scripts and links. The content in this collection will not be refreshed. After 1 week, the “/a/” link collection will be deprecated.

[UPDATE 27 April 1:18 pm Pacific] We’ll be reverting all URLs to the published style within the next 24 hours or so. When the work is finished, we’ll update here.

Group and avatar URLs are currently showing an unannounced format.  The published style is:

…world.secondlife.com/group/<group-key>

and the system is currently returning:

…world.secondlife.com/a/group/<group-key>

This is impacting any number of in world tools and documents which make use of fixed group URLs, including notecard-based group invite tasks.

The Operations Team will be working to correct this as quickly as possible.

 

[04:25am PST 22nd April] Due to a rearrangement in the way we process payouts for groups you will notice group payouts happening a few hours later today than what you are used to.

[Resolved 7:33am] The regions have all recovered. Thank you for your patience!

[Updated 6:54am] Better than half of the regions are back online, and we’re hoping to have the majority of them back on in the next hour or so.

As a result of the previous power outage, we still have almost 4000 regions currently offline. They may appear up but are actually offline. Our tech crew is working to bring them back online and running normally as fast as they can and we will have an update as to when they should be finished as soon as we know more.

[RESOLVED] 4:45 AM PDT At at around 4:15 AM PDT a power interruption occured at one of our datacenters causing some of our servers to reboot. Logins have now been re-enabled.

We have experienced a problem with some of our network hardware and have had to temporarily disable logins while we sort out what happened and how to fix it. We will have updates on when logins will be enabled as soon as we have more information.

[RESOLVED - 11:00PM Pacific] The group chat issues have been resolved. If you still experience any issues with logging in to Second Life, please contact support: http://secondlife.com/support/

[10:30PM Pacific] One of our utility servers is having issues which will report group chat problems. We are working on the issue and will report when it is fixed.  This is currently affecting 1/16 of entire Second Life groups.

[April 23, 2009 - UPDATE 00:39AM] Rolling restarts have been completed.

[April 22, 2009 11:00PM PDT] Rolling restarts for this morning are now underway.

[UPDATE 03:50AM PST - April 22nd] The Rolling Restarts have been completed.

[UPDATE 11:00PM PST - April 21st] Rolling restarts scheduled for tonight are now in progress.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network over the next few nights.

The restarts will resume tonight, starting at 11pm Pacific, and running until approximately 4am.

This will require us to restart several hundred regions each night. Unfortunately, every time a region restarts it moves onto the first available hardware of the appropriate class, so we cannot predict which exact regions will be affected. However, there will be in-world warnings in the regions prior to the restarts, as usual.

[Resolved 2:00PM Pacific] Service to our phone support has been restored.

Some phone support calls are being routed by the system to empty desks.  We’re working to clear the problem as quickly as possible.

[RESOLVED 8:34] The issue causing the disruption to Live Chat support has been corrected.  Live Chat support is once again available.  We now return you to your regularly scheduled Monday.

[8:24]  We are investigating an issue disrupting Live Chat support.  Live Chat support is currently not available although support tickets remain unaffected.  Please monitor this post for updates and further information

[10:40pm PST] The phone support issues are resolved, and both Concierge and billing support phone lines are available again.

[10:18pm PST] The issue that is preventing access to the billing line is also affecting the Concierge support phone team. Concierge and Billing phone support is unavailable for the moment. Please check back for updates.

[10:00pm PST] The billing phone line is currently inaccessible. We will post an update here when the issue is resolved.

[RESOLVED 8:07] The hiccup has been corrected and the affected regions are now accessible.

[UPDATE 7:46]  The technical hiccup is taking longer than expected to correct, although work is continuing to correct the situation.  They are currently estimating it will be another 30 minutes to return to normal.  Please monitor this post for further updates and information.

We have had a small technical hiccup that has resulted in disconnecting about 150 regions. They are still up and running, but this will affect logins and teleports to and from those regions. We expect this to only last for about 20 minutes before returning to normal.

[Resolved] Thank you for your patience while we were experiencing temporary phone loss. Phone support has been restored.

Telephone support is currently experiencing difficulties. If you have tried contacting support and have not been able to get through, please try again. We are aware of the issues and working towards a resolution. Please check back here for further updates.

[RESOLVED 16:51] In the process of fixing search we have rolled out a new search backend system, please report any problems via our Public Issue Tracker (http://jira.secondlife.com) with our new system should you encounter them. Note: there have been no significant changes that affect search rankings.

[UPDATE 15:52] We have determined the cause of the problem with All Search, and are working to correct it.  There is no ETA on the fix at this time however we will keep you posted here with any further updates or information.

We are investigating an issue with the All Search feature which is causing all search results to display the error “We’re sorry: This item’s privacy settings prevent us from showing you any further details about it.” when clicked.  Updates and further information will be posted as soon as available.

[April 16, 2009 4:30AM PDT] Scheduled maintenance by some of our network providers has been completed.

One of our network providers will be undergoing some scheduled maintenance.   Residents may experience some in-world functions not available during this time, including teleporting, rezzing objects and possibly transactions.

We will update you when the maintenance work has been completed.

[Completed 9:15pm PST] The XStreet SL maintenance has been completed.

The Xstreet SL Marketplace will be unavailable temporarily for brief maintenance.  This is expected to only last  approximately 15 minutes beginning at 9:00pm PST, tonight, April 15th.

[April 17, 2009 4:00AM PDT] Second Life Server 1.26.2 has been deployed!

[April 16, 2009 11:00PM PDT] Rolling restarts are now in progress.

[03:31am PST - April 16th] Today’s set of Network Rolling Restarts has been completed.

[4:30am PST - April 15th] This series of restarts has now been completed. However 1.26 server rolling restarts will begin at 7am today.

[9:53pm PST - April 14th] Tonights rolling restarts are on schedule to start at 11pm and continue through to 4am.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network over the next few nights.

The restarts will take place Monday, April 13th - Thursday, April 16th, starting at 11pm, and running until approximately 4am.

This will require us to restart several hundred regions each night. Unfortunately, every time a region restarts it moves onto the first available hardware of the appropriate class, so we cannot predict which exact regions will be affected. However, there will be in-world warnings in the regions prior to the restarts, as usual.

We are investigating an issue some Residents are experiencing in 1.26 server regions resulting in duplicate messages when uploading pictures or sounds, or taking snapshots.  Although some Residents may be informed multiple times that they are paying L$10 for the service, they are correctly assessed for a single transaction.

The issue and its effect on the 1.26.2 rollout are currently being assessed.

[RESOLVED 9:38AM] Credit card processing has been restored for all Residents. Please note that if you attempted a transaction during the outage and received a processing error, you will need to re-enter your billing information.  Please wait 15 minutes between re-entering the information and attempting a transaction.

[April 12, 2009 7:10AM] We are aware of an issue with credit card processing which may be causing problems for some residents. This may effect updating your payment information or processing new L$ purchases. Our engineers are looking into this and we will let you know as soon as there is anything to report.

[RESOLVED 03:58AM PST] Rolling restarts have been completed for tonights schedule.

[UPDATE 11:20PM PST - April 13th] Rolling restarts scheduled for tonight are now in progress.

[UPDATE 10:10pm PST - April 13th] Tonight’s rolling restarts are on schedule to start at 11pm.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network over the next few nights.

The restarts will take place Monday, April 6th - Thursday, April 9th, starting at 11pm, and running until approximately 4am.

This will require us to restart several hundred regions each night. Unfortunately, every time a region restarts it moves onto the first available hardware of the appropriate class, so we cannot predict which exact regions will be affected. However, there will be in-world warnings in the regions prior to the restarts, as usual.

[RESOLVED] Rolling restarts have been completed for today. Thank you.

[IN PROGRESS 11:00PM PST] Rolling restarts are now in progress.

[8:08 PM PST] We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network.

This will require us to restart several hundred regions. Unfortunately, since every time a region restarts, it moves onto the first available hardware of the appropriate class, we cannot predict which exact regions will be affected. However, there will be in-world warnings prior to the restarts, as usual.

[UPDATE 7:38 p.m. Pacific]  Live Chat has been restored.

*****

Live Chat has been disrupted for all departments. We’re working to reconnect it as quickly as possible.

The payment processing maintenance has been completed.

[Updated April 7, 2009 @ 6:43 am PST] The maintenance has started. During the maintenance payments will be temporarily unavailable. However, Linden transactions between residents inworld should be unaffected. Thank you for your patience.

[Posted on April 7, 2009 at 4:20 am] We’ve slated an upgrade to part of our payment processing system between 6:30 a.m. and 7:00 a.m. Pacific on Tuesday, April 7. During that time, credit/debit card transactions will be unavailable, including registration and update of card information, as well as card-funded LindeX transactions.

[RESOLVED 6:00AM PST] Phone support maintenance has been completed!

[5:15AM PST] Our phone support is undergoing maintenance that may result in technical difficulties for resident who are trying to contact Billing or Concierge Support. We estimate that maintenance should be completed sometime around 6AM PST.

[Resolved @ 1:53 pm PST] The rotation of the dead Inventory Database has been completed. Any further complications on this matter are not expected.

[Posted @ 1:04 pm PST, April 2, 2009 ] An inventory database is in the process of being rotated. A small percentage of residents that were on the server at the time of death will be logged out and won’t be able to log in until the server has been replaced. Please check back here for updates.

[RESOLVED 03:00am Pacific] All maintenance has now been completed.

One of our network providers will be undergoing some emergency maintenance.   Residents may experience some in-world functions not available during this time, including teleporting, rezzing objects and possibly transactions.

We will update you when the maintenance work has been completed.

[FINISHED 04:00am PST] The Rolling Restarts are done for today.

[IN PROGRESS 10:00PM PST] Rolling restarts are now in progress.

We will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network.

This will require us to restart several hundred regions. Unfortunately, since every time a region restarts, it moves onto the first available hardware of the appropriate class, we cannot predict which exact regions will be affected. However, there will be in-world warnings prior to the restarts, as usual.

[10:40pm PST] The payment processing issues have been resolved. There may be a delay as people update their information - if you experience difficulty, please wait for a bit then try again.

[9:02pm PST March 31] We are currently experiencing issues with our payment processing systems. You may have difficulty making transactions on secondlife.com, or updating your payment information. We’re actively working on this issue, and will post an update here when it is resolved.

[Finished 03:58am PST] The Rolling Restarts are finished for the day.

This evening, much like last night, we will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network.

This will require us to restart several hundred regions. Unfortunately, since every time a region restarts, it moves onto the first available hardware of the appropriate class, we cannot predict which exact regions will be affected. However, there will be in-world warnings prior to the restarts, as usual.

We are aware of an issue where some regions are being billed up to two days early. This is being looked into and as soon as any additional information is available, we will be sure to let you know!

Our service provider be upgrading our public SVN (source code server): svn.secondlife.com to 1.5 on Friday night, between 10pm and 2am PDT.  The downtime should be approximately one hour during that window, and should not affect any other services. This will not affect any running Second Life simulators.

PayPal has announced a maintenance shutdown of their Website this evening, beginning at 11 p.m. Pacific and concluding by 1:00 a.m. tomorrow morning.

It’s possible that Residents who attempt PayPal-backed transactions in Second Life will be unable to complete them during part or all of the maintenance time.  Transactions which flow directly through PayPal Web services, such as creating new agreements, are obviously likely to be impacted; however, LindeX processing for Residents with established agreements may also be unavailable. 

If we obtain additional useful information or notification that the scope or scheduling of their work has changed, we’ll update here.

[RESOLVED 8:16PM PDT] Traffic calculations have been updated properly.

The process for calculating traffic updates is running a bit late today. We expect it to finish later today. Our best estimates at this point are that traffic should be fully updated around 4pm PDT today. This is a rough esitmate only.

[RESOLVED 02:35am PST] The restarts have finished for today.

[UPDATE 12:00AM Pacific] The rolling restarts are now in progress.

This evening, we will be continuing our project to improve both the performance and maintainability of our network, by moving many of the servers that host Second Life’s regions onto a new network.

This will require us to restart several hundred regions. Unfortunately, since every time a region restarts, it moves onto the first available hardware of the appropriate class, we cannot predict which exact regions will be affected. However, there will be in-world warnings prior to the restarts, as usual.

We will update with any further details when the rolling restart begins at approximately 11pm Pacific tonight.

[RESOLVED 07:39AM Pacific] The upgrade is complete. Credit/debit card transactions are back online.

[UPDATED 07:05AM Pacific] We are still in the process of upgrading the payment system, therefore credit/debit card transactions are sill unavailable.   We will continue to keep you updated on this issue.

We’ve slated an upgrade to part of our payment processing system between  6:30 a.m. and  7:00 a.m. Pacific on Tuesday, 31 March.  During that time, credit/debit card transactions will be unavailable, including registration and update of card information, as well as card-funded LindeX transactions.

Our Support Portal vendor will be performing a software hotfix on Wednesday, 25th March, 7:00 am - 8:00 am Pacific.

During this time period our Knowledgebase, Tickets and Live Chat will not be available.  Telephone support will be unaffected by this work.

wiki.secondlife.com and jira.secondlife.com (the Issue Tracker) will be briefly offline at various points during a maintenance period being carried out by our third party vendor.  This will be from 10:00 pm Pacific on Thursday 26th March to 1:00 am Pacific on Friday 27th March.

Please refrain from editing articles or commenting on JIRAs during this time frame.

Thanks!

[Resolved 01:45am PST] All regions are back, and logins are open again.

We are receiving reports of people being disconnected from Second Life and experiencing difficulties in logging back in. We are working to fix this as soon as possible.

A network problem caused some residents to be logged out of Second Life and login failures for a number of residents at or around 3.20am this morning.

The issues was brief and the system logins have returned to normal.

[RESOLVED] We’ve gotten the all clear on the maintenance and do not expect any further problems.

Some of you may have noticed you’ve been disconnected and are having trouble logging back in. Our network provider is currently performing scheduled maintenance to upgrade some systems and this is resulting in dropped connections and periods where you may not be able to log in. We apologize for the late announcement. We don’t expect this to take more than 15-20 minutes.

Last week we posted information on four maintenance sessions that our credit card processor was planning to run (See previous post here).

This is a reminder that the final maintenance session will take place at 3:00am Pacific on Tuesday 17th March and will conclude within sixty minutes.

During each session, debit/credit card processing will be unavailable. This includes registering cards as new funding sources, or making card-backed LindeX purchases at the Website or through the viewer.

Should there be any changes to this we will update you here.

[RESOLVED 21:00 Pacific] The Support Portal is back. Live Chat and Support Tickets are accessible.

[15:01] Due to the increasingly unstable Live Chat support environment we are now taking the Live Chat feature offline pending the maintenance by our service provider scheduled for later this evening.  The support ticket system is largely unaffected by this problem.  Residents should file support tickets for assistance, or wait until the Live Chat maintenance is completed this evening.

Tonight’s service provider maintenance will be conducted between 18:30 and 20:30.  The entire support portal - Live Chat, Support Ticket, and Knowledge Base - will be taken offline for this maintenance.

[UPDATE 14:12] The service provider of our live chat support system will be taking our chat support system temporarily offline this evening, some time between 18:30 and 20:30 to attempt to correct the current connection issues.  We will post more detailed information including an exact time as the maintenance time approaches.

We are currently experiencing connection issues with the Live Chat service. If you are in a Live Chat session and are disconnected, please try opening a new session. The outages generally only last for a minute or two, allowing you to contact us again almost immediately.

We apologize for any inconvenience this may cause. We are working to resolve this issue as soon as possible and will update this blog with further information when it is available.

[RESOLVED] [07:40AM PST] Issues have now been resolved and voice is now available again.

[07:21AM PST] We have be notified that inworld voice is currently unavailable.  The issue is currently being investigated and we hope to have this resolved soon as possible.   We will continue to update you on the progress.

[RESOLVED 8:27pm PST] The asset servers are all running normally, and this issue is resolved.

Two of our asset servers momentarily fell behind.  This has caused a number of Residents to be unable to log in.

One of the two has already recovered. We expect the other to catch up within the next few minutes. 

[RESOLVED, March 17 12:16PM Pacific] Very early this morning, Linden Lab removed several netblocks from the Spamhaus PBL, inclusion in which appeared to be the main cause of the email issues issues. The PBL is supposed to be a list of “machines that shouldn’t send email”. A look at our email infrastructure shortly before 1200 PDT 2009-03-17 seems to indicate that the issue is now resolved on the whole, though it may take up to a day for mail systems across the internet to recognize the PBL updates. It’s not clear when our IPs were added to their list, but that appears to have been the cause of the IM-to-email and object emails failing to reach their intended targets.

[UPDATED 13:32] There is an issue in the Public Issue Tracker regarding this problem.  Residents experiencing this issue may wish to visit [jira.secondlife.com].

[REOPENED 3/16 13:18] We are investigating continued reports of IM to email problems.  We will update this post when we have further information on this issue.

[RESOLVED - 10:40AM Pacific] IMs to Email are now working again. We are still receiving some reports of failed emails, to @hotmail.com, @live.com and @yahoo.com addresses.  If you are still experiencing problems, please contact your email provider, or try an alternative email address.

We are aware of an issue where some IMs are not working properly, or at all. These are currently affecting both Offline IMs and Object IMs.

We are continuing efforts to resolve this issue and will update this blog post when more information is available.

[RESOLVED] 6:00pm PST March 16, 2009 - This issue has been resolved.

[UPDATE] 10:45pm PST  We have identified the issue and the configuration has been corrected.  A small number of residents may still see this as a result of a cached simhost.  If you are still experiencing this issue, you may contact us via the Support Portal.  Sunday, our Operations Team will sweep through the caches to remove any lingering incorrect entries.  This should be completed by Sunday afternoon.

Some residents may find that they are experiencing issues with certain objects not rezzing properly. Symptoms include partial rezzing of objects, no associated error message, and no relation to which region you are in.

We are actively pursuing a resolution and will update this blog post with more information as soon as it is available.

[UPDATE 12:42] Logins have been re-enabled. Teleport issues are now fixed and Support Portal access has been re-enabled.

[UPDATE 10:40] Logins have been disabled as we continue to work to correct this issue.

[UPDATE 10:00] As a precaution, please avoid making transactions and rezzing no-copy items until the issue is resolved.

[UPDATE 9:56] The issue is also being seen to cause problems with teleporting and access to secondlife.com.  Please continue to monitor this post for updates and further information.

[9:24] We are investigating an issue which may affect some residents ability to log in.  Please monitor this post for updates and further information.

[09:00AM PST] This issue has now been resolved.

[07:00AM PST]  This issue is still being looked into and we will let you know as soon as we have something new to report!

[03:19AM PST] Our support portal for submitting tickets to billing is currently experiencing some issues.    We are working to get this resolved as soon as possible and will keep you updated on the progress.

[RESOLVED] 6:09pm PST - The Support Portal is back.  Live Chat and Support Tickets are accessible.

Live Chat & Support Tickets, found at Second Life’s Support Portal, are temporarily unavailable.  This is currently being investigated and any updates will be posted.

[March 12 - 8:45] We require some maintenance to take place at 9:00  AM Pacific this morning at one of our Datacentres.

The work will take place over 2 hours but any disruption in-world should be minor and brief (e.g. Teleport failures).

We will update you here with progress and/or changes to this work.

Maintenance on the Datacentres is completed. Thank you for your patience.

We require some maintenance to take place at 9:00  AM Pacific this morning at one of our Datacentres.

The work will take place over 2 hours but any disruption in-world should be minor and brief (e.g. Teleport failures).

We will update you here with progress and/or changes to this work.

[6:30 pm Pacific]  The next round of migration is complete.

[UPDATE - 5:00 pm Pacific] The second round of the migration is starting at 5:00 PM.

[12:00 pm Pacific]  The first round of migration is complete.  There will be more at 5:00 pm Pacific.

We will be migrating additional VPNs to LLNet. This work is scheduled to begin at 11:00am and end at 12:00pm.

More migratations are expected to take place at 5:00 pm PST and should finish at 6:30 pm PST on March 10, 2009.

[RESOLVED 8:10] The issue affecting access to the websites has been corrected and access has returned to normal.

We are having an intermittent problem with viewing some of our websites from some locations. If you cannot reach one of our sites please be patient and try again in a few minutes. We are working to fix the problem and hope to have it resolved soon.

[Finished 03:30am PST] This maintenance has been successfully completed.

In addition to the planned maintenance for Debit/Credit Card Processing, we will be swapping round one of our inventory databases on Tuesday 10th March at 3:00 AM Pacific.

The work should last around 30 minutes and, for a small number of residents, access to items in your inventory will be temporarily unavailable during this time.

[9 March 09, 3:20 pm Pacific]

Our card processing vendor has announced four planned maintenance sessions over the next several days.

During each session, debit/credit card processing will be unavailable. This includes registering cards as new funding sources, or making card-backed LindeX purchases at the Website or through the viewer.

The maintenance sessions are slated to begin at 3 a.m. Pacific on each scheduled day and to conclude within sixty minutes.  However, our vendor has reserved a fallback time of 7am - 8am Pacific for each day.  We’ll post as soon as we become aware of any rescheduled sessions.

The scheduled days are:

• Tuesday, 10 March
• Thursday, 12 March
• Friday, 13 March
• Tuesday, 17 March

[RESOLVED 9:43] The Concierge and Billing support telephone lines are once again operating normally.

[Updated @ 9:16 am PST] In addition to Concierge services having difficulty, one of the billing phone lines is also experiencing some technical difficulty.

Currently, Linden Lab phone support for Concierge services are experiencing technical difficulities - some calls do not appear to be connecting. We appreciate your understanding while we investigate this issue.

[RESOLVED 1:15pm PST] The LLNet transition rolling restart is complete.

There will be a rolling restart starting at 9am today for LLNet transition. We will update as this commences with any further news.

[RESOLVED 9:16PM Pacific] Ticket issues have now been resolved.

[9:00PM Pacific] The support system is currently experiencing difficulties attaching files to tickets. We are aware of the problem and are working to get this resolved.

[RESOLVED 12:48] The cause of the issue has been discovered and corrected; the affected regions are again working properly.

[UPDATE 12:37] Affected regions may appear offline to residents attempting to reach them.

[STATUS 12:10] ~900 regions are experiencing and issue interfering with a number of inworld services including teleporting, transactions, rezzing, etc.  Until the All-Clear is sounded please refrain from makeing purchase or rezzing no copy items.  We are investigating and will have the situation corrected as soon as possible.  Please monitor this post for updates and additional information.

[UPDATE] 11:00am PST - This has been completed

Between 8:00am and 11:00am PST today, we will be changing the networking of some servers that host regions currently on the grid.

Those regions will have presence issues: Residents will not be able to enter or exit them, and Residents in them will not be able to access inventory. This effect will continue for possibly as long as 30 minutes per region (but probably closer to 5 minutes).

Any time a region restarts, it comes back on the first available host of that hardware class, so it is not possible for us to predict which regions will be affected this far in advance. We will be announcing in each affected region before the outage starts.

If the scope or schedule of the work changes, or if we get more information, we’ll post here.

[Update 06:25am PST March 04] This has now been resolved. If you are experiencing any billing errors currently please contact support.

Our payment processor is currently experiencing technical difficulties, residents may experience problems when attempting to update billing information and other related payment transactions.

We do not have an estimated time for the resolution to this issue, but there will be an update as soon as we have any further information.

[Update 06:24am PST March 04] The rolling restarts for today are completed.

[Update 04:04am PST March 04] The rolling restarts are about to begin.

[Update] 4:00pm PST] The scheduled rolling restarts have been canceled for Tuesday night.  However, we are resuming our LLNet Renumbering project which will result in restarts at 3:00am PST on Wednesday, March 4th.  We will update as necessary.

[Update 05:0am PST] These Rolling Restarts are finished for today.

[Update 03:05am PST March 03] Today’s set of Rolling Restarts for the LLNet transition have started.

[UPDATED 5:45 pm Pacific] In addition to the restarts below, we’ll be restarting an additional 300 or so Regions between 3 am and 8 am Pacific tomorrow as we resume our LLNet Renumbering project. These two restart waves are unavoidably separate, and we apologize for the inconvenience to those impacted by both events.

As reported last week, rolling restarts will continue for the next two nights.

Schedule:
Mon March 2nd 19:00 to Tue March 3rd 10:00
Tue March 3rd 19:00 to Wed March 4th 10:00

Thanks for your patience!

[RESOLVED] 02:55AM PDT: Live Chat issues have now been resolved and the service is now available.

[RE-OPENED] 01:53AM PDT: Live Chat is currently unavailable, we are working to have this resolved as soon as possible.

[RESOLVED] 01:34AM PDT: The support portal issues are now resolved.

[UPDATED] 01:27AM PDT: Our third party provider is actively working to restore the support portal at this time.  We will continue to update you on the progress.

12:10AM PDT: Tickets, Live Chat and Knowledge Base services, found at Second Life’s Support Portal, are temporarily unavailable.  This is being investigated and any updates will be posted to our Status Blog.  Please note that phone support is still available.

[Resolved 6:40pm PST, February 27 2009]

The issue as described below has been resolved. Further rolling restarts have been delayed until such time that we can ensure the issue does not recur.

[Updated @ 4:12pm PST, February 27, 2009]

Please read below for the reasons behind the LLNet transition Rolling Restart Cancellation.

Last night, in our continuing effort to improve both the performance and maintainability of our network, we moved a bunch of hosts in our Phoenix datacenter onto a new network. This morning we noticed that many simulators were showing elevated asset download queues and that teleports were failing more often than usual. The problem took some time to diagnose, as we initially thought that the issue was network-related.

As many of you already know, for each avatar in Second Life, we store what are called “baked” textures, representing what the avatar looks like when it is wearing it’s current outfit. These are stored on the simulator host where the avatar’s appearance was last edited. We found simulators were getting “stuck” downloading baked textures from the *old* IP address of the simulators that were renumbered. As those IP addresses no longer existed, it was taking three minutes for these downloads to time out causing gray avatars and objects, delays in rezzing objects, a larger number of failed teleport attempts than usual, and other issues related to the simulator’s or viewer’s downloading of assets.

Once the root cause was discovered, it was a matter of deleting the entries from the databases that pointed to the old IP addresses. Though it took about an hour and a half because we wanted to avoid impacting other services in the process.

Once the entries were deleted, which completed at about 3:50 PM PST, things started returning to normal right away.

[Posted @ 1:21PM PST, February 27, 2009] The Rolling Restarts for LLNet transition that were previously scheduled for Saturday, February 28th and Sunday, March 1st have been canceled for the foreseeable future. Please check back for any further updates that may be posted on this issue.

The Rolling Restarts for LLNet transition that were previously scheduled for Saturday, February 28th and Sunday, March 1st have been canceled for the foreseeable future. Please check back for any further updates that may be posted on this issue.

[RESOLVED 9:20 3 March] The Rolling Restarts ending March 2nd and March 3rd have been completed.

[Update 7:10pm PST March 1] Rolling restarts are beginning and will continue through to 10:00am PST March 2nd.

[Updated @ 9:06am, February 27, 2009] Rolling Restarts are finished for today and will commence again at 3:00am PST on Saturday, February 28, 2009.

[ February 27, 2009 @ 2:43am] In addition to the ongoing rolling restarts we’ll be seeing a number of rolling restarts to further the LLNet transition that began happening in January.

These will be commencing:

- Friday, 02/27, 03:00 - 09:00 am PST

- Saturday, 02/28, 03:00 - 09:00 am PST

- Sunday, 03/01, 03:00 - 09:00 am PST

Because this transition is taking place at the same time as another Rolling Restart process there is a likelihood that some regions will restart more than once.

[Resolved] [5:32 pm Pacific] The Land store is back online.

[5:14 pm Pacific] The Land Store is being taken out of service for a very update.  It should be back online within about half an hour.

[RESOLVED]  wiki.secondlife.com and jira.secondlife.com (the Issue Tracker) will be periodically offline during a maintenance period from 11pm PST on February 26th to 2am PST on February 27th. Please try to refrain from editing articles or commenting on JIRAs during this time frame.

Thanks!

The eleventh iteration of the 1.22 Viewer, Release Candidate (RC) 10, has been released!

If you’ve been using RC9, you will be required to update to RC10, which may be installed along side the official 1.21 viewer. Please keep in mind that participation in the viewer Release Candidate cycle is entirely optional.

RC10 includes just a few fixes - a couple of voice fixes and a texture save fix (come on… we know you’ve missed texture fixes!).  As always, complete release notes can be found at:

[wiki.secondlife.com]

We’re so close to the official 1.22 viewer that we can practically taste it, but we still need your help to find crash cases and provide essential feedback.  So please download the Release Candidate from the URL below and start experimenting:

[secondlife.com]

You can participate in discussions about this Release Candidate here in the SL Blog.

Don’t forget to report new issues at jira.secondlife.com - and be sure to set the “Affects Version/s” to “1.22 Release Candidate”. Before you log new issues, please review the Release Notes to see if there are already open issues to which you can add comments. Remember… we’re only able to investigate bugs that are reported in JIRA.

[9:40am PST] RESOLVED Rolling restarts have completed for the day.

[8:08pm PST] Tonights rolling restarts have begun, and are scheduled to continue through to 10:00am PST Thursday morning.

[Resolved] Everything should be back to normal now.

[Update 10:20 PST] We have found that the issues are related to the recent rolling restarts and we are working to correct this.

We are currently investigating problems with region being down longer than normal and other performance issues. We hope to have the problem resolved soon. We will update when the issue has been resolved or we have further information.

[Resolved 12.58am PST] These issues have now been resolved. If you are still experiencing issues please contact support.

We are seeing some intermittent problems with assets - things like notecards not loading or saving, attachments not being able to load, things appearing grey for a long time after login - which we are investigating. Please refrain from making inworld transactions and rezzing no copy objects for the time being.

You may also have difficulty logging in or teleporting.

We will update this post when further information is available.

[Resolved 05:24am PST] The issues with our webhost have been resolved.

Our website hosting vendor is currently experiencing an issue and as a result you may not be able to access secondlife.com, lindenlab.com and teen.secondlife.com. The issue is being worked on and will be resolved as soon as possible.

[RESOLVED 26 Feb 6:08 pm PST] Nightly traffic calculations reported no errors overnight.

[Update 25 feb 02:33am PST] The traffic update is in progress. Some parcels will have been updated by now, but the traffic calculations are still in progress, so if you don’t see you figures updated yet, they will be in one of the next runs of our nightly scripts that handle traffic calculations.

[10:00PM Pacific] Residents should be aware that traffic figures for parcels are not up to date until tomorrow. We are currently working on this and hope to have this resolved as soon as possible. Updates will be posted here as more info becomes avaialble.

[1:40pm] UPDATE: We have resolved the asset issues. If you are still experiencing problems, please contact support.

[1:25pm] UPDATE: We have identified and fixed what we believe to have been the cause of the problem. Things appear to be improving.

[12:54pm] UPDATE: We’re still investigating the issues.

We are seeing some intermittent problems with assets - things like notecards not loading or saving, attachments not being able to load, things appearing grey for a long time after login - which we are investigating. Please refrain from making inworld transactions and rezzing no copy objects for the time being.

We will update this post when further information is available.

[Resolved 11:27am PST] Everybody is fine and back to work.

Due to a fire alarm in one of our worksites, the billing and outworld tech support is not available at the moment. We will let you know when they’re safely back at their desks.

[RESOLVED 2:55AM Pacific] This issue has been resolved. If you continue to have problems, please try clearing Second Life’s cache and relog.

Technicians are currently investigating an issue which may interfere with inworld functionality.
Search and teleports may be slow. Please refrain from making inworld transactions and rezzing
no copy objects for the time being. Please monitor this post for updates and further information.

[5:00pm PST] A recent search update introduced a small bug on approximately 2/17/2009, that resulted in the omission of indexing newly created groups.  Searching for groups created in the past three days will result in zero results.

This was discovered and fixed promptly. A new index is being created now but will not be completed until 10:00pm PST. At that time all group search results should be up to date.

[Completed 4:00pm PST] The maintenance has completed successfully. Please contact support if you believe an affected region has not returned to normal.

At 3:30pm PST, approximately 100 regions in our Phoenix datacenter will be restarted due to priority maintenance in one of our server racks. The regions will be shutdown, then restarted on another server on the Second Life Grid. Each region should be affected for no more than 5 - 10 minutes.

We will update this blog once the maintenance has completed.

[RESOLVED 12:35] The issues with the Land Auction site have been resolved and the auction site is now accessible.  DUE TO TODAY’S ISSUES ALL AUCTIONS ENDING TODAY HAVE BEEN CANCELED AND WILL BE RE-LISTED LATER TODAY.

We are aware of issues with the land auction site which is preventing some users from being able to log in and to bid. We are working on this and will be looking to resolve as soon as possible.

[Completed 11:10am PST] The Wednesday rolling restarts have completed. Please watch this blog for updates and information on the next batch, which will commence on Sunday night at 8pm PST and run through Monday 10am PST.

[Update 7:53pm PST] The next batch of rolling restarts will start now and will run until 10AM.

[Update 1:55pm PST] The Region restarts are complete.

[Update 1:15pm PST] The Region restarts are still in progress, we are at about the halfway mark. Please watch this post for updates.

[Update 10:15 PST] A number of Regions are due to restart to alleviate the connection issue. Please watch this post for updates.

We are investigating reports of connection issues with a small number of regions. Residents in these regions will not be able to see or cross into neighboring regions, may experience difficulty teleporting out of the region and transactions in affected regions may fail. Please be aware of your surroundings, and if in an affected region refrain from making purchases or transactions.

[Resolved 4:15AM Pacific]  This issue has been resolved, in order to get voice working again you may need to exit Second Life and log back in.

Voice is currently unavailable for many residents, this problem is being worked on.  Any updates will be posted here as they become available.

It has come to our attention that some groups did not receive payouts from group land sales/etc., in the last 24 hours. This will be resolved during the next 24 hours, when last nights payouts and tonights will both be paid at the same time. We apologize for the delay.

[RESOLVED 9:27 pm Pacific] Account registration pages are functioning properly on all Webservers. 

[UPDATE 7:41 pm] This error is still hitting about 20% of registration attempts.  We’ll keep you posted.

6:15 pm Pacific - We’ve confirmed reports of proxy errors from some of our Web servers when attempting to create new accounts. We’ve isolated the bug and will have it repaired ASAP.  

When the proxy error hits, the avatar name hasn’t been reserved yet. It should generally be available for a retry once the bug is closed.

[RESOLVED 1:00AM Pacific] This issue has been resolved.  Residents should now be able to update their payment information and access our Land Store.

Some residents may experience problems with updating payment information and accessing the Land Store.  We are currently investigating these issues and will update as necessary.

The tenth iteration of the 1.22 Viewer, Release Candidate (RC) 9, has been released!

If you’ve been using RC8, you will be required to update to RC9, which may be installed along side the official 1.21 viewer. Please keep in mind that participation in the viewer Release Candidate cycle is entirely optional.

RC9 includes some voice-related fixes, a fix for slow sculpt loading, and a new raycast fix. As always, complete release notes can be found at:

[wiki.secondlife.com]

Though we feel we’re very close to the official 1.22 viewer (r-e-a-l-l-y soon!), we still need your help to find crash cases and provide essential feedback. In particular, we’d appreciate some hardcore voice testing of this RC, as there are a number of known issues described in the Release Notes. So, please download the Release Candidate from the URL below and start experimenting:

[secondlife.com]

You can participate in discussions about this Release Candidate here in the SL Forums.

Don’t forget to report new issues at jira.secondlife.com - and be sure to set the “Affects Version/s” to “1.22 Release Candidate”. Before you log new issues, please review the Release Notes to see if there are already open issues to which you can add comments. Remember… we’re only able to investigate bugs that are reported in JIRA.

[RESOLVED] February 12, 2009 4:51pm PST - All inworld services appear to be functioning normally.  XStreet is also online.

[UPDATE] February 12, 2009  2:33pm PST - We are continuing to investigate this issue and at this time XStreet is offline until the all clear is given.  We will continue to update.

[UPDATE] February 12, 2009 1:48pm PST - We are still investigating, but things are appearing to clear up.  We will update this post as necessary.

Technicians are currently investigating a issue which may interfere with inworld functionality.  Please refrain from making inworld transactions and rezzing no copy objects for the time being.  Please monitor this post for updates and further information.

[RESOLVED 5:25pm PST] - The issue has been resolved and logins have returned to normal. If you still experience any issues with logging in to Second Life, please contact support: http://secondlife.com/support/

There is currently an issue preventing a number of residents from logging in. (If you see the message which includes, “Despite our best efforts, something unexpected has gone wrong.”, then you are among this group.)

We are working to resolve this issue as soon as possible. Please watch this blog for updates as they become available.

[RESOLVED]  The network difficulties have been resolved.

Second Life is having some network related technical difficulties that is impacting roughly 300 regions, this should be resolved shortly.

[03:05 am PST] The faulty switch was found and fixed, and all systems are back to working order.

[02:20 am PST] Engineers are on route to the site to work on the issue. We will update you as soon as we get more details from them.

[1:00am PST] We are currently experiencing network issues, this will have an effect on 150 regions.  Residents will not be able to teleport, rezz objects or make inworld transactions at this time.   We are working on the problem and will update you as soon as we have the issue resolved.

On Wednesday morning, February 11th, at approximately 3am PST, we will be changing the networking of some servers that host approximately 500 regions currently on the grid.

Those regions will have presence issues, Residents will not be able to enter or exit them, and Residents in them will not be able to access inventory, for possibly as long as 30 minutes (but probably closer to 5 minutes).

Any time a region restarts, it comes back on the first available host of that hardware class, so it is not possible for us to predict which regions will be affected this far in advance.  We will be announcing in each affected region before the outage starts.

If the scope or schedule of the work changes, or if we get more information, we’ll post here.

[RESOLVED 12 Feb 11:53 pm PST] Billing support has been restored.

[UPDATED 12 Feb 11:30 pm PST] We’re estimating  that we’re 20 minutes away (worst case) from restarting phone and chat support.  We’ll keep you posted. 

[UPDATED 12 Feb 10:20 pm PST]  We’re running a little late closing this maintenance event. We’ll update you with details shortly.

[UPDATED Feb 12 18:22pm PST] The maintenance work has begun. Live chat and phone support for billing inquiries will be unavailable for 2-4 hours. We will update this post when maintenance is complete.

[UPDATED 11 Feb 09, 3:30 Pacific] This work has been rescheduled to begin Thursday evening, 12 Feb, at 6pm Pacific.

As originally announced below, phone and live chat support for billing-related inquiries will be unavailable for two to four hours.

If the scope  or the schedule of the maintenance changes, we’ll update you here.

(Feb 10, 8:50am) This maintenance has been canceled - it may be rescheduled, at which point we will blog anew.

Phone and Live Chat support will be unavailable for Billing inquiries beginning 2:30 PM Pacific Tuesday, 10 Feb.  We expect the maintenance to last between 2 and 4 hours.  We’ll keep you updated.

[RESOLVED 4:35pm PST] - Both telephone and Live Chat services are once again available, for both Billing and Outworld services.

Telephone support for Billing is temporarily unavailable. Live Chat is also unavailable for both Billing and Outworld issues. We expect these services to be down for at least an hour. Please watch this blog for further updates as they become available.

[10:55 am PST] RESOLVED Time outs while trying to update credit cards has been resolved.

[10:27 am PST]  We are experiencing time outs when trying to update credit cards on the website. This difficulty is currently being investigated. Please check back here for updates.

[21:10pm PST] RESOLVED Inworld voice has been restored. If you’re not able to activate voice, relogging should fix the issue.

[20:20 pm PST] REOPENED. Voice is currently working intermittently inworld again.  Our technicians continue to investigate. We will update this post as more information becomes available.

[19:34 pm PST] RESOLVED Inworld voice has been restored. If you’re not able to activate voice, relogging should fix the issue.

[19:31 pm PST] Issues persist with inworld voice. Our technical teams are investigating.

[18:10 pm PST] We’re currently looking into an issue with inworld voice. Voice may be sporadic across the grid. We are investigating the issue, and will post an update as soon as we have more information.

[08:05 am PST] The support portal is now back online.

Our Support Portal will be inaccessible on Wednesday, February 11, from 7 - 8 am PST for scheduled maintenance.

[10:40PM Pacific]  Maintenance is complete!

[10:10PM Pacific]  Our Support Portal is temporarily down for maintenance.  Tickets and Live Chat are unavailable for the time being.

[RESOLVED 8:22] Voice functionality has been restored.

[7:00AM PST] We are awaiting an update from the engineering team from the third party vendor for the voice system. As soon as we have an update we will update this blog post.

[12:00AM PST] This issue as been escalated and we continue to work on a solution.  Unfortunately, a resolution may not come until morning.

[9:05PM PST] We are getting reports of voice not working in several regions. Our technicians have already contacted our third party vendor for the voice system, who has confirmed a problem with their main server. This is being worked on as we speak and is exptected to be resolved soon.

The ninth iteration of the 1.22 Viewer, Release Candidate 8, has been released.  If you have been using Release Candidate 7, you will be required to update to RC8.  As is the case with all 1.22 Release Candidate viewers, RC8 may be installed along side the official 1.21 viewer.

RC8 includes a number of voice improvements, as well as support for OpenAL.  Complete release notes can be found at:

[wiki.secondlife.com]

While we’ve made many improvements, we still need your help in identifying new bugs, finding crash cases, and providing final feedback as we prepare to release the official 1.22 Viewer - very soon!

Please keep in mind that participation in the viewer Release Candidate cycle is entirely optional.  If you wish to take part, just download the Release Candidate from the following web page and start experimenting:

[secondlife.com]

You can participate in discussions about this Release Candidate here in the SL Forums.

Don’t forget… if you find new issues, please report them at jira.secondlife.com.  When submitting new issues, be sure to set the “Affects Version/s” to “1.22 Release Candidate” and follow the instructions to paste your system information into the “Environment” field.  We’re only able to investigate bugs that are reported in this way.

The Public Jira will be down for approximately 10 minutes starting at 1am PST on February 6th.  We will update as necessary if any plans change.

[RESOLVED] February 7th 2:00am - Maintenance by one of our Internet Service Providers has been completed.

[UPDATE] February 6th 11:35pm - The maintenance is being conducted at this time and may affect some logins.  We will update while monitoring the maintenance with our Internet Service Provider.

One of our Internet Service Providers will be performing maintenance on their core routing infrastructure on Saturday, February 7th, between 9pm Friday - 2am Saturday PST. We are working with them to avoid any downtime.

We will update this blog post when the maintenance begins and when it has been completed.

[18:31pm PST] [RESOLVED] The network issues have been resolved and the regions are now back online.

 

[18:00pm PST] Teleport issues, some regions offline

 

We are currently experiencing network difficulties. As a result, some regions are showing as offline or unreachable. There may also be problems with teleports and region crossings. Our technicians are working to resolve the issue. Check back for updates.

[RESOLVED 18:25pm PST] Inworld issues have been resolved.

[UPDATED 14:35] The issue appears to be network related.  As a precaution, please refrain from making purchases or rezzing no copy items until the issue is resolved.

We are investigating reports of failed teleports and Searches.  Please monitor this post for updates and further information.

[RESOLVED 04:13 am PST] The maintenance has concluded, and voice services are back.

Our voice service vendor is performing a scheduled maintenance today between 2 and 4 am PST. We will let you know as soon as they’re finished.

[Resolved 08:35 am PST] The inventory server is back up running.

One of our inventory servers is unavailable at the moment. This will affect a small number of Residents, who will see failure to teleport, to rez objects and to log in.

We are investigating and will update you as soon as more information becomes available.

[3:14pm PST] [RESOLVED] The support portal is accessible again.

Our support portal at secondlife.com/support is currently inaccessible. We’re investigating the issue and will have things up and running as soon as possible. Please check back for further updates.

[Resolved 3:52pm PST] All issues seem to have been resolved: We are currently experiencing some issues in relation to teleportation and logins. We are tracking down the cause and will update the blog with more information as it becomes available.

[Resolved 2:03pm] The issues discussed here have now been resolved: We are currently experiencing network issues. Some regions may appear offline, and unreachable. We are working to resolve this as soon as possible and will update the blog with any further information.

[RESOLVED - 12:32pm PST] All issues have been resolved. Should you experience any related issues, please contact us through normal support channels ([secondlife.com]).

[UPDATED 9:58am PST] - While we have not had confirmation from our network providers, the issue appears to be resolved at this point. We will update this blog when we receive confirmation that this has been resolved.

[07:13AM] We are still working on a resolution to the ongoing issues with our network, we will continue to update you on the situation which we hope to resolve as soon as possible.

[04:13AM] The problems in-world being experienced by residents are related to difficulties at our network provider. We are liaising with them to resolve these as quickly as possible.

[03:50AM] RE-OPENED - We are still getting reports that some residents are still unable to teleport. We will update you as soon as the issue has been resolved.

[02:52AM] [RESOLVED] All issues have now been resolved.

[1:15AM] This is still being looked into. For now, there is nothing new to report.

[11:35PM] You may be unable to view L$ balances, teleport, search, cross regions or connect to Second Life. If you’re inworld, please refrain from rezzing or derezzing no-copy objects.

[11:20PM Pacific] We are aware of an issue with our network that may affect inworld services. Please keep an eye on this post for additional information as it becomes available.

[Resolved 10:08am] This issue has been resolved. You may now resume your activities.

[09:41 am PST]  We’re investigating a server issue which may cause some Residents to experience problems with some Inworld services.  Affected services include logins, teleporting and rezzing.  Please refrain from rezzing no-copy items and making transactions until the all-clear is given.  Our engineers are already working to resolve the issue.

[RESOLVED 04:03 am PST] The maintenance was concluded successfully.

[03:14 am PST] As part of scheduled maintenance, we are rotating two clusters of agent databases.

For a small group of our Residents that will cause inability to log in, or if they are already inworld, they will be logged out.

This work is about to begin now, and we will let you know here as soon as it’s finished.

[RESOLVED 13:30] The database issue has been corrected.  Affected inworld services have returned to normal functionality.

We are investigating a database issue affecting some inworld services.  Affected services include Searches, Mapping and Teleporting.  Please monitor this post for updates and further information.

[Postponed] Today’s scheduled work on the Support Portal has been postponed.  We’ll post a new announcement for any future downtime.

—–

Today, on the 28. January 2009, our Support Portal will not be available from 7 - 8 am PST due to scheduled maintenance.

We will notify you here when the work will be finished.

[4:35pm Pacific] Residents who were charged twice for parcel directory listings or group liabilities will be reimbursed.  If you have been charged twice but do not see a refund after 24 hours, please contact support.

Residents who were paid double stipend this week will not receive a stipend next week. All affected residents will be emailed.

[03:29am Pacific] We have found the cause of the duplicate fees and we are now in the process of identifying the affected residents to resolve the issue.

[02:25AM Pacific] We’re receiving some reports of residents being charged parcel directory fees and potentially group liabilities twice.   We are also receiving reports that stipend’s are being paid twice.

We’re looking into it the issue and will update you as and when we have more information.

[RESOLVED 1:10pm PST] - The issues have been resolved and all payment methods and updates are in working order again. If you find you have any continuing experiences with this issue, please contact support.

[RE-OPENED 11:20am PST] - Some Residents may find that they have issues with entering payment information again. We are working to resolve this as soon as possible. Please watch this blog for further information as it becomes available.

[RESOLVED 8:20] Residents are again able to enter their information into the system. Please refresh the page if you do not see the fields to enter your payment information.

[REOPENED 7:38] We are investigating reports from some residents that they are unable to enter their information into the system. Please monitor this post for further information and updates.

[RESOLVED 6:45am Pacific] You may now be able to purchase L$ from the Second Life website.

[UPDATE 5:10am Pacific] We have been notified that this outage is estimated to be resolved sometime around 6:00am Pacific.

[UPDATE 4:30am Pacific] Our payment service provider has informed us they will need to extend their estimated maintenance time by another 30 minutes. We will let you know when they are finished.

Our service provider for payments will be doing some scheduled maintenance from 1:30AM Pacific. During this time residents will not be able to put credit card information on file or make payments to credit cards. However, Paypal will still be available. We will update you when the maintenance has been completed.

Our service provider has restarted the pipeline for accepting payments from Residents who use credit or debit cards as funding sources. For most affected Residents; relief should be immediate. If you are still unable to transact, and you have changed your card information within the past hour, please give the process an additional hour to clear the backlog.

Both we and the service provider will be monitoring the pipeline today for recurring problems. If you are still having problems, please contact us via the Support Portal.

*****

Some Residents are suffering transaction failures on their account funding sources. This is an issue with our service provider, which we are currently investigating. Changing or re-entering account payment information will not help resolve the situation.  Please watch this blog for further information as it becomes available.

[RESOLVED 5:17 pm Pacific] The fix has been deployed to all Resident accounts.

*****

[UPDATE 3:04 pm Pacific] Engineering has prepared a fix for the scrambled Unicode characters. The fix will deploy beginning at roughly 4pm Pacific, and will take a couple of hours to conclude. Residents who have corrected their own texts will have their repairs preserved; otherwise, the fix will repair the existing text.

*****

We are investigating an issue which may cause unicode (ie non-ASCII) characters to display incorrectly in profiles, group titles, etc.  Until the issue is fixed, affected Residents may correct individual problems themselves by retyping the incorrectly-formatted character in their profile, group information, etc.

[05:46am Pacific] The maintenance is now complete and logins have been re-enabled.

[05:08am Pacific]  As per FJ’s post, residents who are already in world will not be bumped offline.  However, there will be a degradation in performance, as access to the database will be blocked, so transactions, teleporting, asset management and actions requiring a database call will not be available. Movement within a region, chat, and voice will all remain available during the database migration.

[05:04am Pacific]  All new logins to Second Life have been blocked in advance of the maintenance.

[04:47am Pacific]  AS FJ blogged yesterday, we are planning to close access to Second Life at 5.30 a.m. Pacific today, Thursday 22nd January 2009 for approximately 60 minutes.

We’ll keep you posted throughout the downtime of progress, and of any changes to the schedule.

[04:28 am PST] The VPN rack is back, and logins have been reopened. Should you experience a slow login, please try to stick with it rather than trying to reconnect.

[04:16 am PST] Our engineers are now onsite and working on the problem.

[03:38 am PST] The root of the problem appears to be one rack of VPN hosts in one of our colocations. Technicians are on their way to fix it.

In the meantime, we have closed logins temporarily. Please refrain from rezzing no-copy objects and making large transactions.

Due to a VPN problem, you will find that regions inworld might not be visible from their neighboring regions at this time, or that region crossings won’t work.

You might also experience wait time or failure to log in.

[RESOLVED 7:12PM PST] All login issues have been resolved and functionality has been restored.

[2:22PM PST] Logins continue to be intermittent.  We will publish info when the issue has stabilized.  Thank you for your continued patience.

[12:14pm PST] - Logins have been re-enabled and all in-world issues have been resolved; however,  login issues may come and go over the course of the next 2 hours. We suggest being wary about rezzing no-copy items and making important in-world transactions during this time.

[Posted 11:08am PST] - Logins have been temporarily blocked due to a database issue. You may experience effects of this issue in-world, such as problems with Teleporting or rezzing objects. We suggest refraining from rezzing no-copy objects or making any important transactions until this is resolved.

[7:30pm PST] [RESOLVED] ALL CLEAR: Inworld functions have been restored.

[6:17pm PST] Inworld services continue to be functioning intermittently. Please refrain from $L transactions or rezzing no copy objects. Our technicians are working to correct the problem.

Most In world services are at reduced functionality at the moment. Please avoid L$ transactions or handling valuable (no-copy) assets until we post an ALL-CLEAR. Regettably, our ability to broadcast a warning in world is also disabled. Please let your friends know if you’re logged in.

Logins continue to be intermittent.  We will publish info when the issue has stabilized.  Thank you for your continued patience.

[RESOLVED 8:01 pm PST] All login issues have been resolved and functionality has been restored.

[UPDATE 7:36pm PST] Logins are open but functioning intermittantly. Our technicians are working to resolve the issue.

[REOPENED 3:09 PM PST] Logins have been restricted again. Please watch this post for continuing updates.

[RESOLVED 2:26 PM PST]  Logins have been re-enabled.

[REOPENED 2:01 PM PST] Logins have been partially restricted again. Please watch this space for continuing updates.

[RESOLVED 1:46 PM PST]  Logins have been re-enabled. LindeX should also now be available.

[1:27 PST] The Linden Exchange is temporarily disabled.  You will not be able to purchase or sell Linden Dollars from in-world or the website.

[REOPENED 1:08 PST] We are continuing to have problems with the database. Logins have been partially restricted again.

[RESOLVED 11:26 AM PST]  Logins have been re-enabled.

[11:18 AM PST] Unfortunately we are having continued problems with the database. Logins have been partially restricted again whilst we clear a momentary database bottleneck. We expect to re-enable them very shortly, and In-World will not be affected.

[9:47 AM PST] Logins have been restored and all inworld services appear to be functioning normally.  Please feel free to resume your regular activities, including purchases and rezzing.

[9:39 AM PST] Technicians are currently investigating an issue which may interfere with inworld functionality. Logins are blocked. Please refrain from making inworld transactions and rezzing no copy objects for the time being. Please monitor this post for updates and further information.

Maintenance will be conducted on the Second Life Wiki Sunday, January 18th, beginning at 1AM PST. The maintenance is expected to last approximately 2 hours. No outage of the wiki is expected, however residents may experience brief hiccups in service when accessing the wiki during this maintenance.

[RESOLVED 18:48] All services have returned to normal and the database issues have been cleared up.

[UPDATE 17:23] We are still working on resolving this issue, and in the meantime would like to advise estate owners and managers not to restart their regions until this is resolved.

[UPDATE 15:30] Technicians continue to investigate the database issue affecting inworld services.  We still recommend refraining from rezzing no copy items or making purchases until the situation is resolved. Please monitor this post for further information and updates.

Technicians are investigating a database issue which may be interfering with several inworld services including teleporting. As a precaution, we recommend refraining from rezzing no copy items or making purchases until the situation is resolved. Please monitor this post for further information and updates.

[RESOLVED 5:15PM PST] Database issues have now been resolved.

[Reopened 4:40PM PST] Unfortunately we are having continued problems with the database. Once again, please refrain from rezzing no copy items or making purchases until we can resolve this issue.

[Resolved 4:15PM PST] We have resolved this issue, things should be functioning normally now.

Technicians are investigating a database issue which is interfering with several inworld services, such as the Map, Teleporting, and some Searches.  As a precaution, please refrain from rezzing no copy items or making purchases until the situation is resolved.  Please monitor this post for further information and updates.

[1:46 PM PST] Logins have been re-enabled. However, there are still some issues inworld that are being monitored such as stale transactions, failure to teleport and failure to rez. Please refrain from rezzing no copy items and conducting transactions for now.

[12:51 PM PST] Technicians are currently investigating an issue which may interfere with inworld functionality. Logins are blocked. Please refrain from making inworld transactions and rezzing no copy objects for the time being. Please monitor this post for updates and further information.

Technicians are currently investigating an issue which may interfere with inworld functionality. Logins are blocked. Please refrain from making inworld transactions and rezzing no copy objects for the time being. Please monitor this post for updates and further information.

Pjira is scheduled for maintenance from 1:00-1:30am PST on Thursday, January 15, 2009.

[RESOLVED 3:05 PM PST] Logins have been restored and all inworld services appear to be functioning normally.  Please feel free to resume your regular activities, including purchases and rezzing.

[2:21 pm. PST] Logins have been partially restricted again whilst we clear a momentary database bottleneck. We expect to re-enable them very shortly.

[2:00pm PST] - Logins have been re-enabled.

[1:52 PM PST] Technicians are currently investigating an issue which may interfere with inworld functionality. Logins are blocked and as a precaution we are recommending residents refrain from making inworld transactions and rezzing no copy objects for the time being. Please monitor this post for updates and further information.

[Resolved] The update work is finished

Due to unforeseen complications with an update to our back-end servers, Residents might occasionally experience logins being rerouted not to their home or last location, but random areas nearby.

The deploy should be finished within the hour, we will let you know when it’s done.

We are seeing some issues with our support portal, that might prevent premium and concierge customers from seeing all support options available to them. This only will only effect their live chat option, you can submit tickets as usual.

[07:49 am PST] The maintenance work has been completed, logins are open again.

On Thursday morning, January 15th at 7:00am PST we will be performing a database upgrade.  Registration and logins will be disabled for approximately 15 minutes.  More information will be posted as necessary.

We were seeing an overload on XML-RPC service requests from particular parties, which had been disabled for a short time to allow us to figure out what was causing this.

Availability has been restored.

[RESOLVED 2:08 pm PST] Logins have been re-enabled, and all inworld services have been restored.

***

[REOPENED 1:53 pm PST] Logins are once again disabled, we will keep updating the blog as the situation changes.

***

[RESOLVED 1:00 pm PST] Logins have been re-enabled, and all inworld services have been restored.

***

[REOPENED 12:30 pm PST] Logins are once again disabled, we will update the blog as the situation changes.

***

[RESOLVED 12:17 pm PST] Logins have been re-enabled, and all inworld services have been restored.

***

Due to a sudden overload on our database, we’ll shortly restrict logins.

You might experience inworld issues like failure to rez, teleports not working and other inworld services being slow or unresponsive.

Please monitor this blog for updates, and refrain from making huge transactions or rezzing valuable no-copy items at this time.

[RESOLVED 4:53pm PST] Logins have been re-opened and all in-world issues have been resolved.

[RE-OPENED 4:05pm PST] Logins have been closed again. We also suggest refraining from rezzing no-copy items or making important transactions at this time. We will update this blog when more information is available.

[RESOLVED 3:16 pm PST] Logins have been re-enabled and inworld issues are resolved.

[2:37 pm PST] Logins have been disabled while we investigate inworld issues. Please refrain from rezzing no copy objects or making important transactions for the time being. Check back for updates.

[RESOLVED 2:08 pm PST]  Logins are open and all issues are now resolved.

[2:00 pm PST] Logins have been re-enabled.

[1:52pm PST] Logins are currently disabled while our technicians investigate database issues. Please monitor this post for updates and information.

[Completed 3:48pm PST] - The Issue Tracker maintenance has concluded and access is open again.

The Issue Tracker (Jira) will be off line Saturday, January 10, from approximately 1:00am PST until 2:00am PST due to maintenance.

We will update this blog once the maintenance has finished.

[RESOLVED 1:52 pm PST]  Logins have been re-enabled and all issues are now resolved.

[1:34 pm PST] Technicians are currently investigating an issue which may interfere with inworld functionality.  Logins are blocked and we are recommending residents refrain from making inworld transactions and rezzing no copy objects for the time being.  Please monitor this post for updates and further information.

[RESOLVED 2:34 pm PST] The Second Life wiki is up and running.

[10:57 am PST] The Second Life wiki is currently inaccessible.  Technicians are investigating and will have the wiki back online as soon as possible.

[Resolved 2:58pm PST] - The issues have all been resolved and regular activities may resume.

[2:22pm PST] Technicians are investigating issues interfering with some inworld services. Please refrain from making purchases or rezing no copy items until the issue is resolved. Please monitor this post for updates and further information.

[Resolved 3:27pm PST] - The Live Chat link is now accessible from the Support page as normal.

[Posted 3:00pm PST] - Earlier this morning we rolled out a new support portal for premium residents. The Live Chat link for premium account holders is not currently available. This issue is being worked on, until such time please use the following work around to access support.

1. Click the “Ticket History” link
2. Find “Live Chat for Premium Support” (this is in the left hand navigation pane)
3. Click on the first article in the search results (should be “Live Chat for Premium Support”)
4. Click appropriate link to start the chat, as normal

Please watch this post for futher updates on this issue.

The Issue Tracker (Jira) will be restarted on Thursday, January 8. It is expected to be down from approximately 1:00am PST until 1:30am PST.

[03:57pm PST] The scheduled maintenance has concluded.

Tomorrow Tuesday January 13, 2009.  Partial outage on Tuesday morning for upgrade to a group of inventory databases.  Some groups of residents will be affected between 6:30am and 8:00am.  Those groups affected will not be able to log in during this time.

We’re arranging a partial outage on Tuesday morning, January 13, 2009, for an upgrade to a group of inventory databases. There will be groups of residents affected by this between 6:30 and 8:00 a.m. PST. Groups of affected residents will not be able to log in during this time.

[RESOLVED 8:12] The issue has been corrected and credit card Payment Information is again updating normally.

[UPDATE 7:35] PayPal information update is not affected by the issue and may be used normally.

There is currently an issue affecting residents ability to update their payment information on the website. The issues is being worked on and will be corrected as soon as possible.

[RESOLVED 4:10pm PST] - Logins have been re-enabled and all issues are now resolved.

[3:53pm PST] - Technicians are investigating inworld issues. Please refrain from making purchases or rezzing no copy items until the All Clear is sounded. Logins have been disabled.

[Resolved 10:45am PST] - The database has recovered and all issues have been resolved.

[10:30am PST] - We are currently experiencing a problem with the database. There may be difficulties with logging in, rezzing objects, completing transactions, and searching. We are aware of this issue and our ops team are working to resolve it as swiftly as possible. In the mean time, please avoid making important transactions or rezzing no-copy objects.

[RESOLVED 8:56] Technicians have restored communications for the affected regions.

Approximately 400 regions have lost communications. Technicians are working to correct the issue. Please monitor this post for updates.

[RESOLVED 15:08] Logins have been restored and all inworld services appear to be functioning normally.  Please feel free to resume your regular activities, including purchases and rezzing.

[UPDATE 14:39] All logins are now blocked as technicians continue to investigate the cause of the issues.

Technicians are currently investigating an issue which may interfere with inworld functionality. Logins are partially blocked and as a precaution we are recommending residents refrain from making inworld transactions and rezzing no copy objects for the time being. Please monitor this post for updates and further information.

[3.20 AM Pacific - RESOLVED] - Voice maintenance is now complete.

[2.00 AM Pacific] - Voice maintenance has commenced.

[5th Jan 09, 12 PM Pacific]  There is scheduled maintenance for Inworld Voice service beginning approximately 2:00AM PST on January 6th.  The maintenance is anticipated to last about one hour.  Inworld Voice service will be unavailable during this time.  Please monitor this post for updates.

We will begin the rolling restarts for the homestead regions at 8am PST, this should take approximately 5 hours, we will update the post when resolved.

[07:43 PST] The land store will be taken offline at 8am PST for Homestead launch and will be restored at approximately 11am PST

[4:30] Logins have been re-enabled.

[4:24] Logins have been temporarily disabled. Please continue to monitor the blog for updates.

[2:07 PST] Logins have been re-enabled.

[1:59 PST] Logins have been disabled. Please continue to monitor the blog for updates.

[1:10 PST] Inworld services appear to have stabilized. Transactions have been cleared for use. Please continue to monitor the blog for updates.

[12:40 PST] Logins have been re-enabled with inworld services slowly coming back online. Please refrain from transactions until we have given an all-clear.

[12:21 PST] Logins have now been re-enabled.

[12:13 PST] Logins have temporarily been disabled.  We will update you again when problems have been resolved.

[RESOLVED 3:13pm PST] - Logins have been re-enabled. In-world issues have been resolved and you may safely resume normal activities.

[UPDATE 2:48pm PST] - Logins have been temporarily disabled as we work on this issue.

[2:41pm PST] - We are currently experiencing a problem with the database that may result in difficulties logging in, rezzing objects, completing transactions, and searching. We are aware of this issue and our ops team are working to resolve it as swiftly as possible. In the mean time, please avoid making important transactions or rezzing no-copy objects.

[CLOSED 12:55 pm Pacific] Logins are re-enabled.

[12:54 pm Pacific] Logins have been disabled for a couple of minutes while we stabilize in world services (mapping, TP, etc.) We expect to have this cleared very shortly.

[Resolved 9:24PM PST] Maintenance was completed on schedule and the support portal is back up.

Reminder that tonight between 8pm and 9pm PST, the service desk and support portal will be down for maintenance and not be available.  If the maintenance schedule changes, we will post the new schedule.

[RESOLVED 3:42PM Pacific] The database issue affecting inworld services has been resolved; all affected services should now be operating normally.

[3:30PM Pacific] Logins have been re-enabled, but it may take more than one try to get in as the grid re-stabilizes.

[3:20PM Pacific] We are currently experiencing a problem with the database that may result in difficulties logging in, rezzing objects, completing transactions, and searching. We are aware of this issue and our ops team are working to resolve it as swiftly as possible. In the mean time, please avoid making important transactions or rezzing no-copy objects. We have temporarily disabled logins while this issue is investigated.

On Monday, December 29 between 8pm and 9pm PST, the service desk and support portal will be down for maintenance and not be available.  If the maintenance schedule changes, we will post the new schedule.

[RESOLVED 12:11 pm PST] We haven’t seen any further complications, and are closing this issue as resolved.

[UPDATE 10:50 am PST] One of our network providers suffered a signifcant outage, which may have contributed to the earlier reported login issues. Once their services were restored, our login servers dealt with the rush, and logins should now be back to normal.

We’ll leave this blog open until we’re satisfied all issues are taken care of.

***

It appears that some of our login servers are struggling, and some Residents won’t be able to complete the login process. Our Ops team is aware of the situation, and working to fix this as soon as possible.

[FINISHED 08:05 am PST] The maintenance has concluded, the LindeX is open again.

We are temporarily halting Linden Dollar (L$) purchases for maintenance and review purposes.   Please monitor this post for updates and further information.

[UPDATE 26. December '08, 03:04 am PST] The economic statistics have now been updated. We will continue to monitor the page and let you know when we’re satisfied it’s fully fixed.

[UPDATE 22:02] The affected statistics that are not updating are “Islands Added” and “Islands Owned.”  Technicians are continuing their work.

Technical problems are preventing the Economic Statistics page from updating, resulting in incorrect data being presented. Technicians are aware of the issue and are working to correct it. Please monitor this post for updates.

[09:24 am PST] The maintenance work has completed.

***

[08:24 am PST] Approximately 170 regions will experience a 5-10 minute outage today, the work starting at 9 am PST, and ending about an hour later.  We will not be able to publish the affected regions ahead of time.

[RESOLVED 2:35AM Pacific] Everything is back in working order!

[2:10AM Pacific] The Second Life database is currently experiencing issues that may result in inworld difficulties for residents.  We are looking into this and will keep you updated.

[RESOLVED 03:56 am PST] Our voice vendor has given the all clear, and voice services should be working as desired again.

You might need to open your preferences and toggle voice off/on to be reconnected.

***

We are getting reports of voice not working in several regions. Our technicians have already contacted our third party vendor for the voice system, who has confirmed a problem with their main server. This is being worked on as we speak and is exptected to be resolved within an hour.

[UPDATED 23:22] We’ll have a brief network interruption in one of our datacenters on Monday, Dec. 20 at around 5:30AM, which will cause a large percentage of Second Life regions to lose Internet connectivity for 5 to 10 minutes. Some additional maintenance will also be performed which will cause approximately 170 regions to be restarted, immediately after the other network maintenance in order to minimize total impact.

Additional maintenance is scheduled for approximately 9:00AM on Tuesday, Dec. 23. Please monitor this post for updates and additional information.

[RESOLVED 5:05AM Pacific] Everything is back to normal!

There are approximately 600 regions currently offline.  We are looking into this and will let you know as soon as there is anything new to report.

The Public Issue Tracker will be offline for maintenance Friday morning, from 00:00 - 1:00AM.

The Public Issue Tracker is down for maintenance until 2:00AM Thursday.  We apologize for the inconvenience.

Technicians investigated an issue interfering with teleporting.  The issue was brief, and self corrected.  Teleporting has now returned to normal functionality.

[RESOLVED] We’re arranging a partial outage on Thursday morning, December 18, 2008, for an upgrade to a group of inventory databases. There will be groups of residents affected by this between 6:30 and 8:00 a.m. PST. Groups of affected residents will not be able to log in during this time.

Update 12-30 08:55AM : The second-half rolling restart is complete.

Update 12-30 05:17AM : The second-half rolling restart has begun.

Update 12-29 08:28AM : The first-half rolling restart has finished.

Update 12-29 05:31AM : The first-half rolling restart has begun. Even-numbered hosts are being restarted today. Note that due to a small update, the pilot regions will be included in the full roll.

Update 12-22 08:52AM : The pilot rolling restart is complete.

Update 12-22 08:07AM : The pilot rolling restart has begun.

We will be doing a rolling restart to deploy server version 1.24.10 to the production Second Life grid.  This includes the server support code needed for the introduction of Homesteads scheduled for January 5.

As usual with rolling restarts, every region will be down once for a few minutes; if your region stays down for more than 20-30 minutes, please contact support.  There is no way to delay the restart of a region once it has been announced.

The rolling restart will follow this schedule:

• Monday, Dec. 22, 8:00-9:30AM : a pilot roll to ~3000 regions
• Monday, Dec. 29, 5AM-9AM : half of Second Life will receive 1.24.10
• Tuesday, Dec. 30, 5AM-9AM : the remainder of Second Life will receive 1.24.10.

Discussion of the rolling restart will occur in this thread in the forums.

The Support Portal will be undergoing maintenance Wednesday morning, December 17, from 7:00AM to 8:00AM.  All support functions handled through the Support Portal - Tickets, Live Chat, Knowledge Base, etc. - will be unavailable during this brief outage.

On Thursday and Friday, between 9 and 10am PST approximately 170 regions will experience a 5-10 minute outage.  As always, we will not be able to publish the affected regions ahead of time.

[11:33 a.m. Pacific] We’re rebooting the login servers. Attempts to log in may fail once or twice during the reboot, which should take about 10 minutes to complete. Please retry them.

[RESOLVED 1:05] All of the affected regions have now been brought back online.

[UPDATED 00:45] The number of offline regions is now roughly 20. Technicians are continuing their work to bring the remaining regions online.

[00:30] There are roughly 275 regions currently offline. The situation is being worked on and technicians believe they should be able to correct the issue within the next hour. Please monitor this post for updates and further information.

[RESOLVED 20:46]  The land store is now back online.

The land store is temporarily offline. Please monitor this post for further information and updates as available.

[RESOLVED 20:27] The issue causing this has been corrected.  Residents still experiencing this please refer to the previous update for the procedure to log in.

[UPDATED 20:22] Residents experiencing this should be able to login normally again by clearing their cache before logging in. To clear your cache while offline: