Did someone actually break the internet? It sorta seems like it. Users of Twitter, Facebook, Instagram, and YouTube, some of the web’s biggest platforms, reported experiencing major issues on Wednesday, with many losing access to basic features and functions.

Reports first poured in concerning Twitter, where users reported being met with a message telling them they’d reached their “Tweet limit” for the day. Twitter actually does have a tweet limit (it’s 2,400 tweets per day), which the platform says it uses to alleviate strain on its backend. However, most people don’t tweet that much, and many of the people who reported receiving the message said they hadn’t even tweeted yet that day.

[…]

Weirdly enough, an almost identical affliction seemed to descend upon Facebook and Instagram Wednesday, with users reporting that they were unable to post new Insta stories or reach Facebook Messenger. Downdetector, which tracks individual complaints for web platforms, showed a spike in incident reports for both platforms around 4:30 p.m. EST, around the same time that Twitter also began having trouble.

To top it all off, some YouTube users reported being unable to reach the platform’s homepage Wednesday.

[…]

 

Source: Twitter, Facebook, Instagram, YouTube Endure Outages

[…] According to outage tracker DownDetector, reports began coming in of users facing a 500 error and being unable to send, receive or search email through Outlook.com from about 4am UTC, peaking at 8 and 9am as Europeans reached their desks.

Microsoft confirmed the outage on its service health website, saying: “We’re applying targeted mitigations to a subset of affected infrastructure and validating that it has mitigated impact. We’re also making traffic optimization efforts to alleviate user impact and expedite recovery.”

It added that extra “Outlook.com functionality such as Calendar APIs consumed by other services such as Microsoft Teams are also affected.”

At the time of writing, the blackout appears to be ongoing. As for what caused it, the Microsoft 365 Status Twitter account said: “We’ve confirmed that a recent change is contributing to the cause of impact. We’re working on potential solutions to restore availability of the service.”

In plain English, Microsoft tweaked something and the house of cards came tumbling down, so they’ll probably have to revert the change. It offered the reference number EX512238 to track in the admin center and otherwise directed users to watch the service health page for any updates.

[…]

Source: Take the morning off because Outlook has already • The Register

This is why cloud solutions aren’t always the best way to go

[…]

European researchers have successfully tested a system that uses terawatt-level laser pulses to steer lighting toward a 26-foot rod. It’s not limited by its physical height, and can cover much wider areas — in this case, 590 feet — while penetrating clouds and fog.

The design ionizes nitrogen and oxygen molecules, releasing electrons and creating a plasma that conducts electricity. As the laser fires at a very quick 1,000 pulses per second, it’s considerably more likely to intercept lightning as it forms. In the test, conducted between June and September 2021, lightning followed the beam for nearly 197 feet before hitting the rod.

[…]

The University of Glasgow’s Matteo Clerici, who didn’t work on the project, noted to The Journal that the laser in the experiment costs about $2.17 billion dollars. The discoverers also plan to significantly extend the range, to the point where a 33-foot rod would have an effective coverage of 1,640 feet.

[…]

Source: High-powered lasers can be used to steer lightning strikes | Engadget

Microsoft’s flagship cloudy productivity services are down across the Asia-Pacific region.

“Our initial investigation indicates that there our service infrastructure is performing at a sub-optimal level, resulting in impact to general service functionality” states an advisory time-stamped 12:41PM on December 2.

The incident means customers of Exchange Online may not be able to access the service, send email and/or files, or use what Microsoft described as “General functionality”.

The impact on Teams means:

• Users may experience issues scheduling/editing meetings and/or live meetings;
• People Picker/Search function may not work as expected;
• Users may be unable to search Microsoft Teams;
• Users may be unable to load the Assignments tab in Microsoft Teams.

Messaging, chat, channels, and other core Teams services appear to be available.

Microsoft appears not to know what’s wrong.

[…]

Updated at 22:00 UTC, December 2nd The incident has ended! An update to Microsoft’s incident report time-stamped 2314 on December 2 offers the description of the preliminary root cause:

Processing components were not performing within optimal performance thresholds because of a legacy process that required tokens to be processed on specific components. In isolation this process wasn’t problematic, but combined with the large number of requests, this resulted in resource saturation, causing impact across multiple Microsoft 365 apps

Microsoft tested transitioning away from the problematic legacy process and restarting affected infrastructure.

Which worked, so the company did the same thing in its live environment.

The incident ran for nine hours and 59 minutes, from 1355 UTC on December 1st to 0954 UTC on December 2.

[…]

Source: Microsoft mistake took down Exchange Online and Teams • The Register

That floor won’t clean itself… well, quite literally it won’t, especially if the vacuum robot you bought to clean the floor won’t hop off its dock when the servers are down

Users started reporting issues with their Roomba app around midday Friday. The status page for iRobot, the maker of Roomba, identified there were outages with Amazon Web Services. The company said they were working with AWS engineers to get the problem sorted out, though as of reporting this, the issue was still unresolved.

Roomba also tweeted about the issue, saying “some customers may be having issues accessing the iRobot app.”

Server outages happen, and that will of course cause issues with apps that rely on connectivity for most of devices more robust features. The problem is when some users cannot access necessary features at all. One user reported they could no longer stop their Roomba from doing its business as child lock features are only accessible in the app.

In response to Gizmodo’s inquiry, iRobot apologized to the customers for the inconvenience and linked to a video and written instructions about how to manually deactivate child and pet locks.

Other users wrote to Gizmodo that although their Roombas can activate manually by hitting the “Clean” button, their devices are still effectively unusable since they cannot tell the vacuum to only do certain rooms or avoid debris in other parts of the house.

This is just another example of the finicky difficulties employed when electronic devices require an internet connection to access necessary functionality.

[…]

Source: Roomba Users Report App Outages

The incident began at 0627 UTC (2327 Pacific Time) and it took until 0742 UTC (0042 Pacific) before the company managed to bring all its datacenters back online and verify they were working correctly. During this time a variety of sites and services relying on Cloudflare went dark while engineers frantically worked to undo the damage they had wrought short hours previously.

“The outage,” explained Cloudflare, “was caused by a change that was part of a long-running project to increase resilience in our busiest locations.”

Oh, the irony.

What had happened was a change to the company’s prefix advertisement policies, resulting in the withdrawal of a critical subset of prefixes. Cloudflare makes use of BGP (Border Gateway Protocol). As part of this protocol, operators define which policies (adjacent IP addresses) are advertised to or accepted from networks (or peers).

Changing a policy can result in IP addresses no longer being reachable on the Internet. One would therefore hope that extreme caution would be taken before doing a such a thing…

Cloudflare’s mistakes actually began at 0356 UTC (2056 Pacific), when the change was made at the first location. There was no problem – the location used an older architecture rather than Cloudflare’s new “more flexible and resilient” version, known internally as MCP (Multi-Colo Pop.) MCP differed from what had gone before by adding a layer of routing to create a mesh of connections. The theory went that bits and pieces of the internal network could be disabled for maintenance. Cloudflare has already rolled out MCP to 19 of its datacenters.

Moving forward to 0617 UTC (2317 Pacific) and the change was deployed to one of the company’s busiest locations, but not an MCP-enabled one. Things still seemed OK… However, by 0627 UTC (2327 Pacific), the change hit the MCP-enabled locations, rattled through the mesh layer and… took

Five minutes later the company declared a major incident. Within half an hour the root cause had been found and engineers began to revert the change. Slightly worryingly, it took until 0742 UTC (0042 Pacific) before everything was complete. “This was delayed as network engineers walked over each other’s changes, reverting the previous reverts, causing the problem to re-appear sporadically.”

One can imagine the panic at Cloudflare towers, although we cannot imagine a controlled process that resulted in a scenario where “network engineers walked over each other’s changes.”

We’ve asked the company to clarify how this happened, and what testing was done before the configuration change was made, and will update should we receive a response.

Mark Boost CEO of Cloud native outfit Civo (formerly of LCN.com) was scathing regarding the outage: “This morning was a wake-up call for the price we pay for over-reliance on big cloud providers. It is completely unsustainable for an outage with one provider being able to bring vast swathes of the internet offline.

“Users today rely on constant connectivity to access the online services that are part of the fabric of all our lives, making outages hugely damaging…

“We should remember that scale is no guarantee of uptime. Large cloud providers have to manage a vast degree of complexity and moving parts, significantly increasing the risk of an outage.”

Source: Cloudflare explains today’s mega-outage • The Register

Is your Insteon smart home system down? I’m getting reports from dozens of Insteon users that as of Friday their smart home hubs have stopped working. So far, none of them have heard from the company, and Insteon’s Twitter account hasn’t been updated since June 2021. I reached out to Rob Lilleness, the president and chairman of Smartlabs, the company that owns Insteon and have not yet heard back.

However, Lilleness no longer lists Smartlabs/Smarthome/Insteon anywhere on his LinkedIn profile and other members of the Insteon management team have also appeared to decamp Smartlabs based on their LinkedIn profiles. Mike Nunes, the former CIO at Smartlabs lists his role at Insteon/Smartlabs ending in April 2022. Dan Cregg, the chief research officer lists his role at Smartlabs as ending in 2022. Matt Kowalec the president and COO lists his role at Smartlabs as ending in 2020;  and Tom Carter, the CIO doesn’t list his role in the company at all.

Smartlabs is a combination of smart home brands that include Insteon and  Nokia Smart Lighting, which Smartlabs purchased last year. It also owns the smarthome.com web site where consumers can buy Insteon gear. An email to Smartlabs’ corporate office in Irvine, Calif. has not been returned and a call to the listen phone number returns a message saying Verizon could not complete the call and asking me to check the number before trying again. Multiple tries return the same message each time.

[…]

With the current outage, Insteon’s app doesn’t work which means users will be hard pressed to change their device settings and add new gear. I’m hopeful to see if the folks over at Home Assistant or Hubitat can perhaps help stranded Insteon users transfer over to their platforms. It might be possible.

Further reading: With Insteon down, possibly for good, what options do you have for your devices?

Source: Insteon is down and may not be coming back – Stacey on IoT | Internet of Things news and analysis

Apple’s services came back online in the late afternoon. Apple’s system status page shows that all of the services that had previously been listed as “down” are now back in the green. It’s still unclear what happened exactly, and Apple never returned Gizmodo’s email for comment on the situation.

---

Apple is experiencing massive technical difficulties, and widespread reports of outages for its various services are flooding the internet.

The company’s own status page shows that several of its most popular products aren’t working. Multiple reports—including from Down Detector, which tracks website and app outages—have shown that users of iCloud, Apple Music, the App Store, iTunes, Apple TV, iMessage, Mail, Contacts, Find My, Apple Maps, FaceTime, Apple Fitness+, and even our beloved domestic helper Siri all appear to be having major problems. Additionally, Bloomberg reports that Apple’s internal systems, both for its corporate offices and its Apple Store retail locations, are down as well. The company reportedly sent internal messages notifying employees, who had difficulty working from home, that domain name system (DNS) problems led to the outage. The full extent of these outages and the regions they are affecting is unclear.

[…]

Source: Apple Maps, Music, iMessage, App Store, and iCloud Are Down

Edit: Websiteplanet has another tool to detect if a website is down or not

Amazon’s crucial web services business AWS has experienced problems today due to a power outage, affecting services like Slack, Imgur, and the Epic Games store for some users. It’s not looking good if you’re working from home, with some Slack users unable to view or upload images and work management tool Asana also hit by the outages.

The official AWS service health dashboard blamed the issues on power outages in a single data center, affecting one Availability Zone (USE1-AZ4) within the US-EAST-1 Region. At 9:13AM ET, Amazon said it had restored power to the affected servers, and by 12:28PM ET, it had “restored underlying connectivity to the majority of the remaining” systems. However, users may still be experiencing issues as services and servers are relaunched.

[…]

Source: Amazon’s AWS logs third outage this month, affecting Slack, Epic Games Store, Asana and more – The Verge

Amazon Web Services (AWS), the engine that powers many of the internet’s most-trafficked websites and apps, appears to be experiencing a widespread outage that is bringing down several popular services.

Amazon, Disney+, and Venmo are all being affected by the outage, and are showing error messages when users attempt to visit their websites. Amazon appears to be aware of the issue and admitted to seeing “Increased Error Rates” in the AWS Management Console. We reached out to Amazon, and the company pointed us to its AWS Service Health Dashboard. An update posted at 8:26 a.m. PT reads:

“We are experiencing API and console issues in the US-EAST-1 Region. We have identified root cause and we are actively working towards recovery. This issue is affecting the global console landing page, which is also hosted in US-EAST-1.

Amazon further revealed the issue to be caused by an “impairment of several network devices.” In a 2:47 p.m. PT update, the company claims to have “mitigated the underlying issue” that caused network devices to be faulty. Server health is improving, according to Amazon, which is now conducting a service-by-service recovery. The company disabled Event Deliveries for Amazon EventBridge in US-EAST-1 as it works for a full recovery for all affected AWS customers. There is still no timeline on when your favorite sites will be fully operational again.

Source: AWS Outage Takes Down Amazon, Disney+, and Venmo

yay cloud!

Some Tesla drivers who fancied going for a spin on Saturday were unable to do so after an update to the cars’ companion app produced server errors.

Teslas don’t use conventional keys. Instead they require the presence of a fob, key card, or authenticated mobile phone app that links to the electric vehicles over Bluetooth. This is apparently easier and/or more convenient than a key, or something. Heck, everything’s better with Bluetooth, right?

Drivers that use the app to start their cars reported it couldn’t do the job and instead produced an error message.

I’m experiencing 500 server error to connect my @tesla Model 3 on my iOS app in Seoul, S.Korea. Seems like this is a worldwide issue. @elonmusk

— Jaehwan Cho 조재환 (@hohocho) November 19, 2021

Tesla founder and CEO Elon Musk personally replied to the above tweet, with the following information:

Should be coming back online now. Looks like we may have accidentally increased verbosity of network traffic.

Apologies, we will take measures to ensure this doesn’t happen again.

— Elon Musk (@elonmusk) November 20, 2021

Measures like, maybe, letting people open their cars with keys? Just a suggestion.

Tesla appears not to have made any other public statement about the incident. The company put its support forums behind a regwall earlier in 2021 and owning a MuskMobile is a requirement for entry. Your correspondent is therefore unable to explore any official missives. Tesla’s Twitter account is silent on the matter and the electric car biz doesn’t bother with Facebook. The exact nature of the outage is therefore hard to divine.

[…]

Source: Tesla drivers locked out of their cars by server error • The Register

Google Cloud suffered a brief outage, seemingly bringing down or disrupting a whole bunch of websites relying on its systems.

If you’ve had trouble accessing Snapchat, Discord, Spotify, Etsy, retailers like Home Depot, and others today, this is likely why: a fault developed in Google Cloud’s networking infrastructure, resulting in websites throwing up 404 errors. Netizens found themselves unable to log into or use certain services properly.

The good news is that, by now, the IT breakdown has been resolved in that sites using Google’s cloud-based load balancers should work again.

The bad news is that Google’s customers can’t update their load balancing configurations until the web giant gives the word, and when that will be isn’t known.

The outage was acknowledged by Google at 1010 PST, about 35 minutes minutes after websites apparently started going wrong, and a fix was deployed within a few minutes to stop the “page not found” errors. Since that update, though, changes by customers to their external proxy load balancers are being ignored.

[…]

Source: Google Cloud partially fixes load balancer issues • The Register

Facebook offered “sincere apologies” Monday afternoon as a sweeping outage of its site and various other properties, including Instagram, WhatsApp and Messenger, stretched for more than six hours and helped to wipe more than $50 billion off Facebook’s market cap — the stock’s worst day of trading in almost a year.

The issues started around 11:45 a.m. ET, according to DownDetector, and hit users globally, taking out critical communications platforms that billions of people and businesses rely on everyday. Service began to return at around 6 p.m.

While Facebook has yet to identify the root of the issue, cybersecurity experts said it does not appear to be a cyberattack and instead seems to be linked to internal issues with Facebook’s systems.

[…]

As Facebook scrambled to solve the issue, investors ditched the stock, sending almost 5 percent lower to $326.23 per share. It was the stock’s biggest one-day plummet since Nov. 9, 2020.

Facebook founder Mark Zuckerberg’s personal wealth took a more than $6 billion hit on Monday, sending him below Microsoft founder Bill Gates to No. 5 on Bloomberg’s Billionaires Index. Zuckerberg is now worth about $121.6 billion, down from almost $140 billion just a couple weeks ago, according to Bloomberg.

The outage also disrupted internal Facebook systems, including security, a company calendar and scheduling tools, The Times reported, adding that some Facebook employees weren’t even able to enter buildings due to the outage.

[…]

In a curious twist, by early afternoon, the domain name “Facebook.com” was listed for sale by Domain Tools. The organization behind the domain registration was still listed as Facebook, Inc. and it’s unclear why the site’s address would be listed for sale.

[…]

Other popular sites — including Gmail and Microsoft-owned LinkedIn –also began to experience some issues throughout the day, according to DownDetector.

[…]

Oculus, the Facebook-owned virtual reality gaming platform, was having issues, too.

“We’re aware that some people are having trouble accessing our apps and products. We’re working to get things back to normal as quickly as possible, and we apologize for any inconvenience,” Oculus tweeted.

As social media fanatics flocked to Twitter, the Facebook rival joked, “hello literally everyone,” in a tweet that racked up nearly half a million retweets.

But Twitter itself saw some outages Monday afternoon, according to DownDetector, with several thousand people reporting issues on the site.

[…]

The outage comes a day after a Facebook whistleblower who leaked a trove of damning internal documents to the Wall Street Journal came forward and identified herself as Frances Haugen, a former product manager at Facebook.

[…]

Source: Facebook, Instagram, and WhatsApp hit by global outage, stock tanks

July this year, Security Discovery researcher Bob Diachenko came across a plethora of JSON records in an exposed Elasticsearch cluster that piqued his interest.

The 1.9 million-strong recordset contained sensitive information on people, including their names, country citizenship, gender, date of birth, passport details, and no-fly status.

The exposed server was indexed by search engines Censys and ZoomEye, indicating Diachenko may not have been the only person to come across the list:

The researcher told BleepingComputer that given the nature of the exposed fields (e.g. passport details and “no_fly_indicator”) it appeared to be a no-fly or a similar terrorist watchlist.

Additionally, the researcher noticed some elusive fields such as “tag,” “nomination type,” and “selectee indicator,” that weren’t immediately understood by him.

“That was the only valid guess given the nature of data plus there was a specific field named ‘TSC_ID’,” Diachenko told BleepingComputer, which hinted to him the source of the recordset could be the Terrorist Screening Center (TSC).

[…]

Source: Secret terrorist watchlist with 2 million records exposed online

If there are 2 million names on that list, isn’t the definition of ‘terrorist’ maybe a little bit broad?

Microsoft is reporting an outage of Office 365, including Microsoft Teams. On its status page, Microsoft adds: Users may be unable to access multiple Microsoft services. User impact: Users may be unable to access multiple Microsoft 365, Azure, and Dynamics 365 services, including the Service Health Dashboard. More info: Any service that leverages Azure Active Directory (AAD) may be affected. This includes but is not limited to Microsoft Teams, Forms, Exchange Online, Intune and Yammer. Current status: We’ve identified the underlying cause of the problem and are taking steps to mitigate impact. We’ll provide an updated ETA on resolution as soon as one is available. Scope of impact: This issue could affect any user.

Source: Microsoft Office 365 Down For Some Users – Slashdot

Yay cloud

Like it or not, connected cars have become a staple of every day life for millions of Americans. The ability to interact with our cars from afar past the key fob has become something we expect to work, but that all relies on the underpinnings of critical IT infrastructure. And when something isn’t working as expected, a minor inconvenience can translate into a customer nightmare.

Someone over at Kia has been having a very bad week. Since Saturday, Kia’s online and connected services have been down, leaving owners unable to pay their bills, remotely unlock their vehicles, or even warm them up in the middle of one of the harshest winters that parts of the U.S. have seen in quite some time.

via Kia, Twitter

Kia’s hamsters have their work cut out for them.

Owners took to Twitter and various online forums to complain about the unscheduled outage, many confused why they couldn’t view the details of their cars on Kia’s website or various phone apps.

Some owners looking to pay their bills also visited Kia’s finance site where they were unable to login and pay their bills, so they resorted to the phonelines which played a message stating that the self-service options were down for scheduled maintenance. Needless to say, that led to a flurry of people tweeting at Kia because they were unsure of the outcome should they miss a payment due to the outage.

via Kia

Now, it’s not just existing Kia drivers that are affected. New buyers are also stuck, unable to set up accounts with Kia’s online services. We confirmed this by trying to create an account on the Kia owners’ portal, but were greeted with an “Internal Server Error” and couldn’t proceed.

[…]

Source: Kia’s Network Is Down, From Finance Sites to Owner Apps, and Nobody Will Say Why

Google services such as YouTube and Gmail started the week with an almighty bang as the Chocolate Factory’s cloud came crashing to the ground.

Despite an insistence from the company’s various health dashboards that all was fine and dandy, it most definitely was not.

Those seeking distraction in video form were treated to YouTube’s “Something went wrong…” monkey, while others wishing to express their disquiet via Gmail were shown a 502 code or a suggestion to try again in five minutes.

The issue appears to have afflicted vast swathes of the globe, with users in the Philippines and India joining Europeans and US early birds in being unexpectedly ejected from the Chocolate Factory’s services.

Problems seemed to start at around 11:30 GMT. At time of writing YouTube was inaccessible, Gmail was borked, Drive was down, image search failed (unless an error code was what you were looking for), and Docs didn’t seem happy.

Some things still worked – we found links to existing Google Docs were working and the search for which the company is famed appeared to be running. So there was no need to resort to something like Bing.

Google is no stranger to outages. Pretty much everything from GCP to G Suite fell over into a heap back in August.

As for today’s outage, Google’s Workspace dashboard was aglow with green lights, even if the reality was quite different.

[…]

Source: World+dog share in collective panic attack as Google slides off the face of the internet • The Register

Amazon Web Services (AWS), Amazon’s internet infrastructure service that is the backbone of many websites and apps, experienced a multi-hour outage on Wednesday that affected a large portion of the internet. The service has been nearly fully restored as of 4:18AM ET on Thursday morning, according to Amazon.

Source: Prolonged AWS outage takes down a big chunk of the internet – The Verge

The video locker was slow to load videos and balked when asked to upload new content on Wednesday, from just before midnight GMT. While all but night-owl European users mostly missed the mess, North American users woke up without their favourite early morning streams and some Asian users were also deprived of their favourite vids and top notch strategic content like Reg lectures.

In typical Google style, YouTube had very little to say about the incident, other than acknowledging it was aware of the situation and then sounding the all-clear without revealing any details about what had transpired.

…And we’re back – we’re so sorry for the interruption. This is fixed across all devices & YouTube services, thanks for being patient with us ❤️ https://t.co/1s0qbxQqc6

— TeamYouTube (@TeamYouTube) November 12, 2020

Multiple observers have pointed out that YouTube’s travails were matched at Google’s Movie-and-TV-show streaming operations, suggesting a problem on common infrastructure.

Plenty of people make a living on YouTube, so the outage is more than an inconvenience or opportunity to make cheap quips about cat videos.

Source: YouTube to world: Move along, nothing to see here … because we’re having an outage • The Register

Traditionally a night for fireworks, 5 November saw some sort of detonation within the Microsoft 365 infrastructure in the form of a borked update or, as the company delicately put it: “an issue wherein some users may be unable to access their mailboxes through Exchange Online via all connection methods.”

There was good news, however, as just over an hour later Microsoft confessed that a recent service update was “causing impact to mailbox access via Exchange Online” but a fix was being prepared that would sort it all out.

The joy was short-lived. Having realised that a network driver issue was to blame, the company then admitted the fix “was taking longer than anticipated.”

Six hours after its initial notification of trouble in the cloud, Microsoft also began looking at alternative options for “faster relief.”

Finally, nearly eight hours after the bad news first dropped from its social media orifice, the software giant claimed a fix was being rolled out. A further four hours was needed before Microsoft trumpeted that everything was up and running once again.

[…]

A glance at social media shows a substantial number of users struggled with the outage, with some making unkind comparisons with arch-rival Gmail and others expressing bewilderment that a driver could cause such an impact. Surely it was tested before hitting production systems?

Oh you sweet summer child. Have you forgotten Windows 10 1809 so soon? We await anxiously the arrival of the rebrandogun. Microsoft 352, anyone?

Source: Network driver issue shaves 12 more hours off Microsoft’s ‘365’ infrastructure, and yeah, it was Exchange Online again • The Register

Yay, cloud