I accidentally found a security issue while benchmarking postgres changes.
If you run debian testing, unstable or some other more "bleeding edge" distribution, I strongly recommend upgrading ASAP.
I was doing some micro-benchmarking at the time, needed to quiesce the system to reduce noise. Saw sshd processes were using a surprising amount of CPU, despite immediately failing because of wrong usernames etc. Profiled sshd, showing lots of cpu time in liblzma, with perf unable to attribute it to a symbol. Got suspicious. Recalled that I had seen an odd valgrind complaint in automated testing of postgres, a few weeks earlier, after package updates.
Really required a lot of coincidences.
@AndresFreundTec Oh, I didn't know you're on the fediverse! I embedded your post in my article @ https://boehs.org/node/everything-i-know-about-the-xz-backdoor
Please let me know if there's anything you'd like me to add to it or clarify!
@AndresFreundTec And a lot of persistence! Reminds me of one of the classics of the industry, Cliff Stoll's Cuckoo's Egg - "Stoll traced the error to an unauthorized user who had apparently used nine seconds of computer time and not paid for it" leading to a german hacker selling content to the KGB - 38 years ago. It is impressive (but uncommon) to see someone paying that level of attention to anomalies these days, with how thick tech stacks have gotten...
@AndresFreundTec well done mate, thanks for flipping the switches what needed flipped
@AndresFreundTec JiaT75: "And I Would Have Gotten Away With It Too, If It Weren't For You Meddling Andres Freund"
@AndresFreundTec Gotta add to the noise, many thanks for digging into this. If it wasn't for you, who knows what could've been done here. You're a hero, and I mean it.
@AndresFreundTec Am I understanding correctly that these were sshd processes fending off the deluge of bogus logins that any publicly accessible sshd has to contend with? If so, good thing you weren’t testing on a more isolated machine.
@AndresFreundTec many thanks... The "coincidences" would not help unless you had abilities and cared.
@AndresFreundTec The open source community really owes you a huge debt of gratitude. Thank you for exposing this security issue and bringing it to the light of day.
Thanks for your work on this. Everyone should follow your example.
One more aspect that I think emphasizes the number of coincidences that had to come together to find this:
I run a number "buildfarm" instances for automatic testing of postgres. Among them with valgrind. For some other test instance I had used -fno-omit-frame-pointer for some reason I do not remember. A year or so ago I moved all the test instances to a common base configuration, instead of duplicate configurations. I chose to make all of them use -fno-omit-frame-pointer.
@AndresFreundTec "... and kids, that's how i saved the world"
I read news that about this attack, thank for your alertness. 👍
Afaict valgrind would not have complained about the payload without -fno-omit-frame-pointer. It was because _get_cpuid() expected the stack frame to look a certain way.
Additionally, I chose to use debian unstable to find possible portability problems earlier. Without that valgrind would have had nothing to complain.
Without having seen the odd complaints in valgrind, I don't think I would have looked deeply enough when seeing the high cpu in sshd below _get_cpuid().
@AndresFreundTec I recognize -fno-omit-frame-pointer as being important for ASAN https://github.com/google/sanitizers/wiki/AddressSanitizerFlags which has some overlap in functionality with valgrind (but obviously a fundamentally different approach).
There are more coincidences that are even less interesting. But even the above should make it clear how unlikely it was that I found this thing.
@AndresFreundTec lmfaooooooo the fact that this was found because of -fno-omit-frame-pointer confirms this is the final FOSS Discourse Event, literally every single thing that people have ever talked about in FOSS for the past year makes an appearance here
@AndresFreundTec now that @fedora, @ubuntu and @archlinux have frame pointers, maybe this #xzorcist issue will encourage #Debian to follow suit
Edited 349d ago
Nonetheless, amazing work. Hope you get a Gold Star at the office...
Edited 350d ago
@AndresFreundTec outstanding...luck , knowledge and experience have met the right person at the right point in time.
Just to be clear: I didn't mean that I didn't do good - I did. I mean that we got unreasonably lucky here, and that we can't just bank on that going forward.
@AndresFreundTec dont for a minute think that you are ever having a holiday again! We need you monitoring those timings! The world needs you! 🏅
@AndresFreundTec but there are thousands of devs running all sorts of tests. The question is not "what were the odds that you missed it", but "what were the odds that everybody missed it"...
@AndresFreundTec That was more than just good - you probably stopped a devastating attack on the whole industry. The open source community owes you a huge debt of gratitude. Had that exploit gotten into the wild it would have been awful. Catching it early was an immense win.
I hope Microsoft recognizes how much of a contribution you made to the entire industry.
@AndresFreundTec Hey dude, you really save Millions, thanks a lot.
(Luck or not you did a fantastic job).
@AndresFreundTec Yep! Important to remember that one of the coincidences in the chain was that it landed in your lap, and you had built a set of skills over your career that were necessary to dig deeper. 🍻
Unlikely they attacked only one out of every possible package that could provide access to a system. How do we find all the other dormant hacks?