According to correspondence released by the Scottish Police Authority (SPA) under freedom of information (FOI) rules, Microsoft is unable to guarantee that data uploaded to a key Police Scotland IT system – the Digital Evidence Sharing Capability (DESC) – will remain in the UK as required by law.

While the correspondence has not been released in full, the disclosure reveals that data hosted in Microsoft’s hyperscale public cloud infrastructure is regularly transferred and processed overseas; that the data processing agreement in place for the DESC did not cover UK-specific data protection requirements; and that while the company has the ability to make technical changes to ensure data protection compliance, it is only making these changes for DESC partners and not other policing bodies because “no one else had asked”.

The correspondence also contains acknowledgements from Microsoft that international data transfers are inherent to its public cloud architecture. As a result, the issues identified with the Scottish Police will equally apply to all UK government users, many of whom face similar regulatory limitations on the offshoring of data.

[…]

Nicky Stewart, a former ICT chief at the UK government’s Cabinet Office, said most people with knowledge of how hyperscale public cloud works have known about these data sovereignty issues for years.

“It’s clearly going to be a concern to any police force that’s using Microsoft, but it’s wider than that,” she said, adding that while Part 3 of the Data Protection Act (DPA) 2018 clearly stipulates that law enforcement data needs to be kept in the UK, other kinds of public sector data must also be kept sovereign under the new G-Cloud 14 framework, which has introduced a UK-only data hosting requirement.

[…]

Microsoft’s commitment to not access customer data without permission is further complicated by the terms of service, which make that promise strictly conditional by giving the company the ability to access data without permission if they either have to fulfil a legal burden, such as responding to government requests for data, or to maintain the service.

[…]

He added that given Microsoft’s disclosures to the SPA, “it must now be obvious that M365 and Azure Cloud services do not meet the two key requirements” to be a legal processor or sub-processor of law enforcement data under the DPA 18.

“These are: one, to conduct all processing and support activities 100% from inside the UK; and two, to only make an international transfer if they are specifically instructed to make the particular transfer by the controller,” he said.

“Microsoft have confirmed that they do not and cannot commit to requirement one for their M365 services, or indeed for most of the services they operate and support in Azure. They have also said that they cannot ‘operationalise’ individual requests as required of them under section 59(7) of the act, thus failing to meet requirement two.

“There can be no clearer evidence than Microsoft’s own clarifications that they cannot meet the legal requirements for a processor or sub-processor of law enforcement data.”

Stewart said: “If it’s not possible to understand the simple question, ‘do you know where your data is all the time?’, then you probably shouldn’t be putting your data in that platform.”

[…]

Source: Microsoft admits no guarantee of sovereignty for UK policing data | Computer Weekly

With the EU and also some EU domain name registrars (looking at you, SIDN) working with these crazy cloud providers, it should have been blindingly obvious that putting data in a US cloud provider would open it up for US spying and a complete lack of data ownership. However idiots will be idiots.

More than half a million UniSuper fund members went a week with no access to their superannuation accounts after a “one-of-a-kind” Google Cloud “misconfiguration” led to the financial services provider’s private cloud account being deleted, Google and UniSuper have revealed.

Services began being restored for UniSuper customers on Thursday, more than a week after the system went offline. Investment account balances would reflect last week’s figures and UniSuper said those would be updated as quickly as possible.

The UniSuper CEO, Peter Chun, wrote to the fund’s 620,000 members on Wednesday night, explaining the outage was not the result of a cyber-attack, and no personal data had been exposed as a result of the outage. Chun pinpointed Google’s cloud service as the issue.

In an extraordinary joint statement from Chun and the global CEO for Google Cloud, Thomas Kurian, the pair apologised to members for the outage, and said it had been “extremely frustrating and disappointing”.

They said the outage was caused by a misconfiguration that resulted in UniSuper’s cloud account being deleted, something that had never happened to Google Cloud before.

“Google Cloud CEO, Thomas Kurian has confirmed that the disruption arose from an unprecedented sequence of events whereby an inadvertent misconfiguration during provisioning of UniSuper’s Private Cloud services ultimately resulted in the deletion of UniSuper’s Private Cloud subscription,” the pair said.

“This is an isolated, ‘one-of-a-kind occurrence’ that has never before occurred with any of Google Cloud’s clients globally. This should not have happened. Google Cloud has identified the events that led to this disruption and taken measures to ensure this does not happen again.”

While UniSuper normally has duplication in place in two geographies, to ensure that if one service goes down or is lost then it can be easily restored, because the fund’s cloud subscription was deleted, it caused the deletion across both geographies.

UniSuper was able to eventually restore services because the fund had backups in place with another provider.

“These backups have minimised data loss, and significantly improved the ability of UniSuper and Google Cloud to complete the restoration,” the pair said.

[…]

Source: Google Cloud accidentally deletes UniSuper’s online account due to ‘unprecedented misconfiguration’ | Superannuation | The Guardian

Google is dealing with its second “lost data” fiasco in the past few months. This time, it’s Google Drive, which has been mysteriously losing files for some people. Google acknowledged the issue on November 27, and a week later, it posted what it called a fix.

It doesn’t feel like Google is describing this issue correctly; the company still calls it a “syncing issue” with the Drive desktop app versions 84.0.0.0 through 84.0.4.0. Syncing problems would only mean files don’t make it to or from the cloud, and that doesn’t explain why people are completely losing files. In the most popular issue thread on the Google Drive Community forums, several users describe spreadsheets and documents going missing, which all would have been created and saved in the web interface, not the desktop app, and it’s hard to see how the desktop app could affect that. Many users peg “May 2023” as the time documents stopped saving. Some say they’ve never used the desktop app.

[…]

Google’s recovery instructions outline a few ways to attempt to “recover your files.” One is via a new secret UI in the Google Drive desktop app version 85.0.13.0 or higher. If you hold shift while clicking on the Drive system tray/menu bar icon, you’ll get a special debug UI with an option to “Recover from backups.” Google says, “Once recovery is complete, you’ll see a new folder on your desktop with the unsynced files named Google Drive Recovery.” Google doesn’t explain what this does or how it works.

Option No. 2 is surprising: use of the command line to recover files. The new Drive binary comes with flags for ‘–recover_from_account_backups’ and ‘–recover_from_app_data_path’, which tells us a bit about what is going on. When Google first acknowledged the issue, it warned users not to delete or move Drive’s app data folder. These flags from the recovery process make it sound like Google hopes your missing files will be in the Drive cache somewhere. Google also suggests trying Windows Backup or macOS Time Machine to find your files.

Google locked the issue thread on the Drive Community Forums at 170 replies before it was clear the problem was solved. It’s also marking any additional threads as “duplicates” and locking them.

[…]

Of the few replies before Google locked the thread, most suggested that Google’s fix did not work. One user calls the fix “complete BS,” adding, “The “solution” doesn’t work for most people.” Another says, “Google Drive DELETED my files so they are not available for recovery. This “fix” is not a fix!” There are lots of other reports of the fix not working, and not many that say they got their files back. The idea that Drive would have months-old copies of files in the app data folder is hard to believe.

[…]

Source: Google calls Drive data loss “fixed,” locks forum threads saying otherwise | Ars Technica

Google Drive users are reporting files mysteriously disappearing from the service, with some netizens on the goliath’s support forums claiming six or more months of work have unceremoniously vanished.

The issue has been rumbling for a few days, with one user logging into Google Drive and finding things as they were in May 2023.

According to the poster, almost everything saved since then has gone, and attempts at recovery failed.

Others chimed in with similar experiences, and one claimed that six months of business data had gone AWOL.

There is little information regarding what has happened; some users reported that synchronization had simply stopped working, so the cloud storage was out of date. Others could get some of their information back by fiddling with cached files, although the limited advice on offer for the affected was to leave things well alone until engineers come up with a solution.

A message purporting to be from Google support also advised not to make changes to the root/data folder while engineers investigate the issue.

[…]

a reminder that just because files are being stored in the cloud, there is no guarantee that they are safe. European cloud hosting provider OVH suffered a disastrous fire in 2021 that left some customers scrambling for backups and disaster recovery plans.

[…]

ust because the files have been uploaded one day does not necessarily mean they will still be there – or recoverable – the next.

[…]

MatthewSt reports that he has a fix; obviously this is something worked out by a user rather than official advice, so caution is advised.

Source: The mystery of the disappearing Google Drive files • The Register

Microsoft techies are trying to recover storage nodes for a “small” number of customers following a “power issue” on October 20 that triggered Azure service disruptions and ruined breakfast for those wanting to use hosted virtual machines or SQL DB.

The degradation began at 0731 UTC on Friday when Microsoft spotted the unspecified power problem, which affected infrastructure in one Availability Zone in the West Europe region. As such, businesses using VMs, Storage, App Service, or Cosmos and SQL DB suffered interruptions.

So what caused this unplanned downtime session? Microsoft says in an incident report on its Azure status history page: “Due to an upstream utility disturbance, we moved to generator power for a section of one datacenter at approximately 0731 UTC. A subset of those generators supporting that section failed to take over as expected during the switch over from utility power, resulting in the impact.”

Engineers managed to restore power again at around 0800 UTC and the impacted infrastructure began to clamber back online again. When the networking and storage plumbing recovered, compute scale units were brought into service, and for the “vast majority” the Azure services were accessible again from 0915 UTC.

Yet not everyone was up and running smoothly, Microsoft admitted.

“A small amount of storage nodes needs to be recovered manually, leading to delays in recovery for some services and customers. We are working to recover these nodes and will continue to communicate to these impacted customers directly via the Service Health blade in the Azure Portal.”

Source: Microsoft admits ‘power issue’ downed Azure in West Europe • The Register

Earlier this month, a group known as Anonymous Sudan took credit for a service outage that disrupted access to Outlook, OneDrive and a handful of other Microsoft online services. After initially sharing little information about the incident, the company confirmed late Friday it had been the target of a series of distributed denial-of-service attacks. In a blog post spotted by the Associated Press (via The Verge), Microsoft said the attacks “temporarily impacted” the availability of some services, adding they were primarily designed to generate “publicity” for a threat actor the company has dubbed Storm-1359. Under Microsoft’s threat actor naming convention, Storm is a temporary designator the company employs for groups whose affiliation it hasn’t definitively established yet.

“We have seen no evidence that customer data has been accessed or compromised,” the company said.

[…]

Source: Microsoft confirms June Outlook and OneDrive outages were caused by DDoS attacks | Engadget

[…]

The problem kicked off this morning with Redmond saying it was looking into errors within its caching infrastructure. In an advisory, the Windows goliath wrote “some users may be intermittently unable to view or access web apps in Microsoft 365.”

A range of Microsoft 365 online services are affected, such as Excel, the company wrote, adding “the search bar may not appear in any Office Online service.” Others impacted include Teams admin centers, SharePoint Online (users may not be able to view the settings gear, search bar, and waffle), and Planner.

According to DownDetector, complaints of the outage began to spike before 0900 ET (1300 UTC). There’s no sign of any resumption in services for the time being.

The software giant initially indicated the problem was linked to an “unusually high number of timeout exceptions within our caching and our Azure Active Directory (AAD) infrastructure.” It soon updated that its engineers had narrowed down a cause.

“We determined that a section of caching infrastructure is performing below acceptable performance thresholds, causing calls to gather user licensing information to bypass the cache and go directly to Azure Active Directory infrastructure, resulting in high resource utilization, resulting in throttling and impact,” Redmond wrote in an advisory.

[…]

Microsoft has battled its share of outages in recent months. A code change caused a four-hour outage of Azure Resource Manager in Europe in March and a month earlier Outlook was knocked out for a while.

In January, Microsoft had to roll back a network change in its WAN after it cause problems a range of cloud services, including Exchange Online, Teams, Outlook, and OneDrive for Business.

[…]

Source: Microsoft 365 and Teams hit in global partial outage • The Register

Some users of Microsoft’s free Outlook hosted service are finding they can no longer send or receive emails because of how the Windows giant now calculates the storage of attachments.

Microsoft account holders are allowed to hold up to 15GB in their cloud-hosted email, which until recently included text and attachments, and 5GB in their OneDrive storage. That policy changed February 1. Since then, attachments now count as part of the 5GB OneDrive allowance – and if that amount is exceeded, it throws a wrench into the email service.

It doesn’t change the storage amount available in Outlook.com, but could in OneDrive.

“This update may reduce how much cloud storage you have available to use with your OneDrive,” Microsoft wrote in a support note posted before the change. “If you reach your cloud storage quota, your ability to send and receive emails in Outlook.com will be disrupted.”

Redmond added that the plan was to gradually roll out the cloud storage changes and new quota bar starting February 1 across users’ app and Windows settings and Microsoft accounts. Two months later, that gradual rollout is beginning to hit more and more users.

One reader told The Register that his Outlook recently stopped working and indicated that he had surpassed the 5GB storage limit, reaching 6.1GB. He was unaware of the policy change, so he was confused when he saw that in his email account he had used only 6.8GB of the 15GB allowed.

It was the change in how attachments are added that tripped him up. Microsoft told him about the new policy.

No one deletes attachments every time an email is received. This is like blackmail
“So instantly, I have lost 10GB of email capacity and because my attachments were greater than 5GB that instantly disabled my email and triggered bounce-backs (even sending and receiving with no attachments),” the reader told us.

“No one deletes attachments every time an email is received. This is like blackmail. MS is forcing us to buy a subscription by the back door or to have to delete emails with attachments on a regular basis ad infinitum.”

He isn’t the only one perplexed by the issue.

[…]

One who apparently was unaware that it was the attachments shifting over to OneDrive causing the email problems deleted a lot of emails, only to find it didn’t change the “storage used” amount.

[…]

https://www.theregister.com/2023/04/06/microsoft_outlook_onedrive_storage/

Did someone actually break the internet? It sorta seems like it. Users of Twitter, Facebook, Instagram, and YouTube, some of the web’s biggest platforms, reported experiencing major issues on Wednesday, with many losing access to basic features and functions.

Reports first poured in concerning Twitter, where users reported being met with a message telling them they’d reached their “Tweet limit” for the day. Twitter actually does have a tweet limit (it’s 2,400 tweets per day), which the platform says it uses to alleviate strain on its backend. However, most people don’t tweet that much, and many of the people who reported receiving the message said they hadn’t even tweeted yet that day.

[…]

Weirdly enough, an almost identical affliction seemed to descend upon Facebook and Instagram Wednesday, with users reporting that they were unable to post new Insta stories or reach Facebook Messenger. Downdetector, which tracks individual complaints for web platforms, showed a spike in incident reports for both platforms around 4:30 p.m. EST, around the same time that Twitter also began having trouble.

To top it all off, some YouTube users reported being unable to reach the platform’s homepage Wednesday.

[…]

 

Source: Twitter, Facebook, Instagram, YouTube Endure Outages

[…] According to outage tracker DownDetector, reports began coming in of users facing a 500 error and being unable to send, receive or search email through Outlook.com from about 4am UTC, peaking at 8 and 9am as Europeans reached their desks.

Microsoft confirmed the outage on its service health website, saying: “We’re applying targeted mitigations to a subset of affected infrastructure and validating that it has mitigated impact. We’re also making traffic optimization efforts to alleviate user impact and expedite recovery.”

It added that extra “Outlook.com functionality such as Calendar APIs consumed by other services such as Microsoft Teams are also affected.”

At the time of writing, the blackout appears to be ongoing. As for what caused it, the Microsoft 365 Status Twitter account said: “We’ve confirmed that a recent change is contributing to the cause of impact. We’re working on potential solutions to restore availability of the service.”

In plain English, Microsoft tweaked something and the house of cards came tumbling down, so they’ll probably have to revert the change. It offered the reference number EX512238 to track in the admin center and otherwise directed users to watch the service health page for any updates.

[…]

Source: Take the morning off because Outlook has already • The Register

This is why cloud solutions aren’t always the best way to go

[…]

European researchers have successfully tested a system that uses terawatt-level laser pulses to steer lighting toward a 26-foot rod. It’s not limited by its physical height, and can cover much wider areas — in this case, 590 feet — while penetrating clouds and fog.

The design ionizes nitrogen and oxygen molecules, releasing electrons and creating a plasma that conducts electricity. As the laser fires at a very quick 1,000 pulses per second, it’s considerably more likely to intercept lightning as it forms. In the test, conducted between June and September 2021, lightning followed the beam for nearly 197 feet before hitting the rod.

[…]

The University of Glasgow’s Matteo Clerici, who didn’t work on the project, noted to The Journal that the laser in the experiment costs about $2.17 billion dollars. The discoverers also plan to significantly extend the range, to the point where a 33-foot rod would have an effective coverage of 1,640 feet.

[…]

Source: High-powered lasers can be used to steer lightning strikes | Engadget

Microsoft’s flagship cloudy productivity services are down across the Asia-Pacific region.

“Our initial investigation indicates that there our service infrastructure is performing at a sub-optimal level, resulting in impact to general service functionality” states an advisory time-stamped 12:41PM on December 2.

The incident means customers of Exchange Online may not be able to access the service, send email and/or files, or use what Microsoft described as “General functionality”.

The impact on Teams means:

• Users may experience issues scheduling/editing meetings and/or live meetings;
• People Picker/Search function may not work as expected;
• Users may be unable to search Microsoft Teams;
• Users may be unable to load the Assignments tab in Microsoft Teams.

Messaging, chat, channels, and other core Teams services appear to be available.

Microsoft appears not to know what’s wrong.

[…]

Updated at 22:00 UTC, December 2nd The incident has ended! An update to Microsoft’s incident report time-stamped 2314 on December 2 offers the description of the preliminary root cause:

Processing components were not performing within optimal performance thresholds because of a legacy process that required tokens to be processed on specific components. In isolation this process wasn’t problematic, but combined with the large number of requests, this resulted in resource saturation, causing impact across multiple Microsoft 365 apps

Microsoft tested transitioning away from the problematic legacy process and restarting affected infrastructure.

Which worked, so the company did the same thing in its live environment.

The incident ran for nine hours and 59 minutes, from 1355 UTC on December 1st to 0954 UTC on December 2.

[…]

Source: Microsoft mistake took down Exchange Online and Teams • The Register

That floor won’t clean itself… well, quite literally it won’t, especially if the vacuum robot you bought to clean the floor won’t hop off its dock when the servers are down

Users started reporting issues with their Roomba app around midday Friday. The status page for iRobot, the maker of Roomba, identified there were outages with Amazon Web Services. The company said they were working with AWS engineers to get the problem sorted out, though as of reporting this, the issue was still unresolved.

Roomba also tweeted about the issue, saying “some customers may be having issues accessing the iRobot app.”

Server outages happen, and that will of course cause issues with apps that rely on connectivity for most of devices more robust features. The problem is when some users cannot access necessary features at all. One user reported they could no longer stop their Roomba from doing its business as child lock features are only accessible in the app.

In response to Gizmodo’s inquiry, iRobot apologized to the customers for the inconvenience and linked to a video and written instructions about how to manually deactivate child and pet locks.

Other users wrote to Gizmodo that although their Roombas can activate manually by hitting the “Clean” button, their devices are still effectively unusable since they cannot tell the vacuum to only do certain rooms or avoid debris in other parts of the house.

This is just another example of the finicky difficulties employed when electronic devices require an internet connection to access necessary functionality.

[…]

Source: Roomba Users Report App Outages

The incident began at 0627 UTC (2327 Pacific Time) and it took until 0742 UTC (0042 Pacific) before the company managed to bring all its datacenters back online and verify they were working correctly. During this time a variety of sites and services relying on Cloudflare went dark while engineers frantically worked to undo the damage they had wrought short hours previously.

“The outage,” explained Cloudflare, “was caused by a change that was part of a long-running project to increase resilience in our busiest locations.”

Oh, the irony.

What had happened was a change to the company’s prefix advertisement policies, resulting in the withdrawal of a critical subset of prefixes. Cloudflare makes use of BGP (Border Gateway Protocol). As part of this protocol, operators define which policies (adjacent IP addresses) are advertised to or accepted from networks (or peers).

Changing a policy can result in IP addresses no longer being reachable on the Internet. One would therefore hope that extreme caution would be taken before doing a such a thing…

Cloudflare’s mistakes actually began at 0356 UTC (2056 Pacific), when the change was made at the first location. There was no problem – the location used an older architecture rather than Cloudflare’s new “more flexible and resilient” version, known internally as MCP (Multi-Colo Pop.) MCP differed from what had gone before by adding a layer of routing to create a mesh of connections. The theory went that bits and pieces of the internal network could be disabled for maintenance. Cloudflare has already rolled out MCP to 19 of its datacenters.

Moving forward to 0617 UTC (2317 Pacific) and the change was deployed to one of the company’s busiest locations, but not an MCP-enabled one. Things still seemed OK… However, by 0627 UTC (2327 Pacific), the change hit the MCP-enabled locations, rattled through the mesh layer and… took

Five minutes later the company declared a major incident. Within half an hour the root cause had been found and engineers began to revert the change. Slightly worryingly, it took until 0742 UTC (0042 Pacific) before everything was complete. “This was delayed as network engineers walked over each other’s changes, reverting the previous reverts, causing the problem to re-appear sporadically.”

One can imagine the panic at Cloudflare towers, although we cannot imagine a controlled process that resulted in a scenario where “network engineers walked over each other’s changes.”

We’ve asked the company to clarify how this happened, and what testing was done before the configuration change was made, and will update should we receive a response.

Mark Boost CEO of Cloud native outfit Civo (formerly of LCN.com) was scathing regarding the outage: “This morning was a wake-up call for the price we pay for over-reliance on big cloud providers. It is completely unsustainable for an outage with one provider being able to bring vast swathes of the internet offline.

“Users today rely on constant connectivity to access the online services that are part of the fabric of all our lives, making outages hugely damaging…

“We should remember that scale is no guarantee of uptime. Large cloud providers have to manage a vast degree of complexity and moving parts, significantly increasing the risk of an outage.”

Source: Cloudflare explains today’s mega-outage • The Register