I accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
I accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Really required a lot of coincidences.
@AndresFreundTec congrats and thank you for the investigation- IMO this is going to go down as the vuln of the decade. What a find.
@AndresFreundTec Congrats on the amazing find fellow MSFTy!
@AndresFreundTec Finding a backdoor because minimal timing errors occurred. That's some Clifford Stoll level work! 😱
@AndresFreundTec Great work! If you’re ever in southernmost Sweden I’ll buy you a beer. 😁
@AndresFreundTec
A pretty important discovery.
There may be a better way to put this, but congrats and thanks!
@AndresFreundTec Respekt! Herzlichen Dank für das Verhindern eines IT-Security-Suoer-GAU. Nicht auszudenken was passiert wäre, wenn du das nicht entdeckt hättest und das in die stable-Zweige der großen Distributionen gekommen wäre! Der Schaden ist ja jetzt schon groß, aber das... Nicht auszudenken! Danke!
@AndresFreundTec thanks.
@AndresFreundTec outstanding work 👏
@AndresFreundTec You are a legend!
@AndresFreundTec Unbelievably great work! We all owe you a huge debt of gratitude!!
@AndresFreundTec
Exhibit A for why I run stable on everything.
@AndresFreundTec
You are awesome.
@AndresFreundTec intercepting RSA_public_decrypt implies it only triggers for RSA-based ssh keys. Glad I switched all my keys to ed25519 already.
@AndresFreundTec does that mean stable versions of Debian and Ubuntu are okay?
@AndresFreundTec what's really remarkable is that you've found the attack not through a functional analysis or formal review but with a sound software engineering approach: the Valgrind effect had been tinkered on - the CPU load impact, however, is a classical side channel usually only used for attack, not for defense. Kudos for making the role of good software engineering obvious for systems security! #security #oss
@AndresFreundTec thank you so much for digging in on this. i hope you never have to pay for your own beers again.
@AndresFreundTec thank you so much!
@AndresFreundTec Wow impressive!!!
Many thanks to you...
@AndresFreundTec thanks!
@AndresFreundTec Thank you for taking the time and finding & publishing this one!
@AndresFreundTec Thank you so much, we all dodged a bullet because of you.
@AndresFreundTec You accidentally hit the find of the year (so far at least). Might even have fucked up a state-sponsored campaign with your find. Kudos for that and also great write-up of the whole issue.
@AndresFreundTec Thank you for your efforts, and the excellent writeup you did when reporting what you uncovered.
It’s already clear that this whole situation will shake things up quite a bit, and rightfully so.
For those interested, see also https://boehs.org/node/everything-i-know-about-the-xz-backdoor on the history behind this…
@AndresFreundTec You literally saved the internet and probably billions of dollars, well done
@AndresFreundTec you're a hero!
@AndresFreundTec fascinating read. Thanks.
@AndresFreundTec Thank you 🤗
@AndresFreundTec
So, is the exploding complexity of software in itself perhaps a security risk? The fact that the existing code is so complex that it is relatively easy to hide a backdoor with nobody noticing is somehow worrying.
@AndresFreundTec How many of your 2407 followers did you gain in the last 48 hours? (One of them being me 😇)
Read about this on Lemmy. Just want to say I appreciate it. Very nice work! I'm not that smart, so the fact a half-second delay set off red flags for you is incredible.
@AndresFreundTec Thank you.
@AndresFreundTec This is wild, your hyper-specific benchmarking accidentally saved the majority of computers worldwide 😭
@AndresFreundTec Mildly curious about something in the report; why were you using Debian Unstable in a production environment? Isn't that generally intended for testing or am I misunderstanding your report here?
@AndresFreundTec You saved millions of Internet users
@AndresFreundTec if we ever meet IRL, I owe you a beer!
@AndresFreundTec kudos!!!
@AndresFreundTec incredible work!
Thanks !
Ja ho deveu saber, que li devem una birra (i més) a @AndresFreundTec, veritat?
@AndresFreundTec Thank you for being lucky and not letting it go 🙏
@AndresFreundTec Thank you for your analysis and time!
@AndresFreundTec thank you, I see the paragraph that begins:
"To reproduce outside of systemd, …"
I have someone claiming:
"The exploit requires systemd. …"
Can both be true?
Postscript: thanks to @vi for helping me to realise my misunderstanding.
Apologies for the noise.
@AndresFreundTec Luckily I read the information early to prepare.
@djchateau This was on a system for testing and benchmarking, not a production system in that sense.
@hyc @AndresFreundTec It triggers for *anyone* trying to log in using RSA-based SSH keys. Even if you've switched your keys over, an attacker can still trigger it by trying to log in using RSA.
@carnildo @AndresFreundTec by switching I meant disabling use of RSA in the sshd config. It's been deprecated in OpenSSH for a while already.