Update (2): T-Mobile has told us that everything is back to normal and this was not an attack.
This was not a cyberattack. This has been resolved – it was an internal technical issue that temporarily impacted some platforms.
Update (1): Everything seems to be getting back to normal. According to a Reddit post, a rogue employee ran a script that brought the system down. This claim hasn't been verified but if that's what happened, it suggests a bad actor was able to access T-Mobile's internal system and make it dance to their tune. Alternatively, it may have been an honest mistake.
A script was executed that deleted every single namespace managed by the Conducktor platform. A namespace is essentially an abstraction of a cluster of EC2 instances that are leased from AWS. Conducktor manages the leasing, organization, networking configuration, and API orchestration to handle deployment and configuration of AWS stuff in general but its primarily EC2 instances, K8S configuration, some Redis, Elsaticache, Routing/Load Balancers, etc. It’s a lot. Too much to list and it gets complicated in a hurry and I don’t know how to succinctly summarize it. Maybe “Giant magical AWS wrapper” ?
But the overall is that this means that every team that owned an application or service, that deployed to AWS via Conducktor, had their stuff nuked. Conducktor is very widely used in Digital for APIs and applications. So most UI applications that are served from a webserver, APIs running on a java server, etc., were impacted as those servers themselves were deployed to EC2 instances managed by Conducktor. This is why this was such a widespread problem across channels (Retail, Care/Telesales, Web and App) as well as across lines of business (Prepaid, Postpaid, Business, Tmobile Money – which nobody knows exists nor should they, etc etc).
Quoting from a guy on the bridge, this was done by “A rogue admin ID” …so …I dunno, that smells really bad to me. Like, someone’s going to jail kinda bad.
Recommended Stories
The original story continues below.
###
T-Mobile is experiencing what appears to be a widespread outage, with affected users flocking tooutage monitoring site Downdetector and social media platforms Reddit and X to report issues.
Initially it was suspected that the company might be under another cyber attack, but it has confirmed to BleepingComputerand CNET that the outage has been caused by an internal technical issue which is currently being worked on.
— Eli Blumenthal (@eliblumenthal) January 11, 2024T-Mobile seems to be having an issue. Website is down, can’t log into account.
T-Mobile tells @CNET, however, that this is not a data breach or hack. pic.twitter.com/uroMDaOxG4
Customers are unable to log into their accounts and the company's app is also not working. Users who try to sign into their accounts get a message that says the website has been "unplugged" and are being redirected to an outage site, per The Mobile Report. The carrier's banking service T-Mobile Money is also down.
More worryingly, somecustomer accounts have been suspended because they were unable to make payments due to the outage.
Oops, somebody unplugged the site. The site is currently unavailable. We're working on it, but in the meantime please give us a call for anything you need."
The breadth of disruption seems wide, with one X user reporting that they saw an error forT-Mobile's Apache Kafka event store and received a warning that the service to route the host addressaccount.t-mobile.com couldn't be found.
P.S. Even assuming that this is caused by an innocent error (which is common), it could still be leveraged by attackers who were waiting for a disruption like this to strike. So regardless of the explanation, monitor and exercise extra care for now if you are a t-mobile user.
— zooko ⓩ (@zooko) January 11, 2024
Apparently, even T-Mobile'sfrontline teams are seeing login and other errors in many applications, hindering their ability to carry out tasks like process activations and account modifications.
If you reach out to retail and customer care representatives, they will likely tell you that they are experiencing "challenges" that have impacted their ability to process transactions.
We're experiencing system challenges impacting our ability to process nearly all transactions."
(T-Mobile) Galaxy S24 Ultra: up to $300 off with a trade-in
Your ultimate Galaxy experience is now available at deeply reduced prices. The Galaxy S24 Ultra sells with a FREE storage upgrade, plus T-Mobile lets you trade-in an eligible device for an extra up to $300 off. The deal applies to users with ONE, Magenta, or Go5G plans.
$420 off (30%)Trade-in
$99999
$141999
Save up to $1,100 on the Z Fold 6 with T-Mobile and trade-in
Trade in an eligible device in good condition and activate a Go5G Plus or Go5G Next plan for your new Z Fold 6, and you can save up to $1,100 on the foldable. The T-Mobile offer is available from the official Samsung store.
$1100 off (58%)Trade-in
$79999
$189999
Galaxy S24+: up to $300 off with T-Mobile at Samsung
If you get the Galaxy S24+ with a T-Mobile plan from the official store, you can get up to $300 off over 24 monthly bill credits. The offer requires you to trade in an eligible device, switch to, have, or activate a ONE, Magenta, or Go5G plan.
$300 off (27%)Trade-in
$81999
$111999
Anam Hamid is a computer scientist turned tech journalist who has a keen interest in the tech world, with a particular focus on smartphones and tablets. She has previously written for Android Headlines and has also been a ghostwriter for several tech and car publications. Anam is not a tech hoarder and believes in using her gadgets for as long as possible. She is concerned about smartphone addiction and its impact on future generations, but she also appreciates the convenience that phones have brought into our lives. Anam is excited about technological advancements like folding screens and under-display sensors, and she often wonders about the future of technology. She values the overall experience of a device more than its individual specs and admires companies that deliver durable, high-quality products. In her free time, Anam enjoys reading, scrolling through Reddit and Instagram, and occasionally refreshing her programming skills through tutorials.
