Chicken penne is now rivaling spaghetti Bolognese as a core layer of pasta infrastructure.

If rationalists could benefit from just one piece of advice, it would be: actions speak louder than words. Right now, I don’t think they understand that, given their penchant for 10k word blog posts.

One non-AI example of this is the most expensive fireworks show in history, I mean, the SpaceX Starship program. So far, they have had 11 or 12 test flights (I don’t care to count the exact number by this point), and not a single one of them has delivered anything into orbit. Fans generally tend to cling on to a few parlor tricks like the “chopstick” stuff. They seem to have forgotten that their goal was to land people on the moon. This goal had already been accomplished over 50 years ago with the 11th flight of the Apollo program.

I saw this coming from their very first Starship test flight. They destroyed the launchpad as soon as the rocket lifted off, with massive chunks of concrete flying hundreds of feet into the air. The rocket itself lost control and exploded 4 minutes later. But by far the most damning part was when the camera cut to the SpaceX employees wildly cheering. Later on there were countless spin articles about how this test flight was successful because they collected so much data.

I chose to believe the evidence in front of my eyes over the talking points about how SpaceX was decades ahead of everyone else, SpaceX is a leader in cheap reusable spacecraft, iterative development is great, etc. Now, I choose to look at the actions of the AI companies, and I can easily see that they do not have any ethics. Meanwhile, the rationalists are hypnotized by the Anthropic critihype blog posts about how their AI is dangerous.

You cannot honestly call it “trust” if you still have to go through the output with a magnifying glass and make sure it didn’t tell anyone to put glue on their pizza.

When any other technology fails to achieve its stated purpose, we call it flawed and unreliable. But AI is so magical! It receives credit for everything it happens to get right, and it’s my fault when it gets something wrong.

Unfortunately, I don’t think anyone is ever going to go through all 19,797 submissions and 75,800 reviews (to one conference, in one year) and manually review them all. Then again, using the ultra-advanced cutting-edge innovative statistical technique of randomly sampling a few papers/reviews, one can still get useful conclusions.

After the bubble collapses, I believe there is going to be a rule of thumb for whatever tiny niche use cases LLMs might have: “Never let an LLM have any decision-making power.” At most, LLMs will serve as a heuristic function for an algorithm that actually works.

Unlike the railroads of the First Gilded Age, I don’t think GenAI will have many long term viable use cases. The problem is that it has two characteristics that do not go well together: unreliability and expense. Generally, it’s not worth spending lots of money on a task where you don’t need reliability.

The sheer expense of GenAI has been subsidized by the massive amounts of money thrown at it by tech CEOs and venture capital. People do not realize how much hundreds of billions of dollars is. On a more concrete scale, people only see the fun little chat box when they open ChatGPT, and they do not see the millions of dollars worth of hardware needed to even run a single instance of ChatGPT. The unreliability of GenAI is much harder to hide completely, but it has been masked by some of the most aggressive marketing in history towards an audience that has already drunk the tech hype Kool-Aid. Who else would look at a tool that deletes their entire hard drive and still ever consider using it again?

The unreliability is not really solvable (after hundreds of billions of dollars of trying), but the expense can be reduced at the cost of making the model even less reliable. I expect the true “use cases” to be mainly spam, and perhaps students cheating on homework.

Now I’m even more skeptical of the programmers (and managers) who endorse LLMs.

The basilisk now eats its own tail.

Promptfans still can’t get over the Erdős problems. Thankfully, even r/singularity has somehow become resistant to the most overhyped claims. I don’t think I need to comment on this one.

Link: https://www.reddit.com/r/singularity/comments/1pag5mp/aristotle_from_harmonicmath_just_proved_erdos/

True, it is possible to achieve 100,000x speedups if you dispose of the silly restriction of being correct.

We will secure energy dominance by dumping even more money and resources into a technology that is already straining our power grid. But don’t worry. The LLM will figure it all out by reciting the Wikipedia page for Fusion Power.

AI is expected to make cutting-edge simulations run “10,000 to 100,000 times faster.”

Turns out it’s not good to assume that literally every word that comes out of a tech billionaire’s mouth is true. Now everyone else thinks they can get away with just rattling off numbers where their source is they made it the fuck up. I still remember Elon Musk saying a decade ago that he could make rockets 1,000 times cheaper, and so many people just thought it was going to happen.

We need scientists and engineers. We do not need Silicon Valley billionaire visionary innovator genius whizzes with big ideas who are pushing the frontiers of physics with ChatGPT.

You’d think peer review would make things better here, but big ML conferences have to deal with an absurd amount of submissions these days. NeurIPS this year got over 21000. The system they use for reviews is that anyone who submits a paper is required to review a certain number of other papers. So yeah, your ML paper is getting reviewed by other people who happen to submit their own papers. Who are competing with you to get their own papers accepted. Yeah, no problems there.

Just make sure you have a few missile turrets protecting the area if you’re playing against zerg. You don’t want your SCV that is building the SMR to get sniped by a flock of mutalisks.

In my experience most people just suck at learning new things, and vastly overestimate the depth of expertise. It doesn’t take that long to learn how to do a thing. I have never written a song (without AI assistance) in my life, but I am sure I could learn within a week. I don’t know how to draw, but I know I could become adequate for any specific task I am trying to achieve within a week. I have never made a 3D prototype in CAD and then used a 3D printer to print it, but I am sure I could learn within a few days.

This reminds me of another tech bro many years ago who also thought that expertise is overrated, and things really aren’t that hard, you know? That belief eventually led him to make a public challenge that he could beat Magnus Carlsen in chess after a month of practice. The WSJ picked up on this, and decided to sponsor an actual match with him and Carlsen. They wrote a fawning article about it, but it did little to stop his enormous public humiliation in the chess community. Here’s a reddit thread discussing that incident: https://www.reddit.com/r/HobbyDrama/comments/nb5b1k/chess_one_month_to_beat_magnus_how_an_obsessive/

As a sidenote, I found it really funny that he thought his best strategy was literally to train a neural network and … memorize all the weights and run inference with mental calculations during the game. Of course, on the day of the match, the strategy was not successful because his algorithm “ran out of time calculating”. How are so many techbros not even good at tech? Come on, that’s the one thing you’re supposed to know!

Just had a conversation about AI where I sent a link to Eddy Burback’s ChatGPT Made Me Delusional video. They clarified that no, it’s only smart people who are more productive with AI since they can filter out all the bad outputs, and only dumb people would suffer all the negative effects. I don’t know what to fucking say.

One of the core beliefs of rationalism is that Intelligence™ is the sole determinant of outcomes, overriding resource imbalances, structural factors, or even just plain old luck. For example, since Elon Musk is so rich, that must be because he is very Intelligent™, despite all of the demonstrably idiotic things he has said over the years. So, even in an artificial scenario like chess, they cannot accept the fact that no amount of Intelligence™ can make up for a large material imbalance between the players.

There was a sneer two years ago about this exact question. I can’t blame the rationalists though. The concept of using external sources outside of their bubble is quite unfamiliar to them.

The dumb strawman protagonist is called “Mr. Humman” and the ASI villain is called “Mr. Assi”. I don’t think any parody writer trying to make fun of rationalist writing could come up with something this bad.

The funniest comment is the one pointing out how Eliezer screws up so many basic facts about chess that even an amateur player can see all the problems. Now, if only the commenter looked around a little further and realized that Eliezer is bullshitting about everything else as well.

Every time I hear a moderate AI argument (e.g. AI will be an aid for searching literature or writing code), it’s like, “Look, it’s impressive that the AI managed to do this. Sure, it took about three dozen prompts over five hours, made me waste another five hours because it generated some completely incorrect nonsense that I had to verify, produced an answer that was much lower quality than if I had just searched it up myself, and boiled two lakes in the process. You should acknowledge that there is something there, even if it did take a trillion dollars of hardware and power to grind the entire internet and all books and scientific papers into a viscous paste. Your objections are invalid because I’m sure things are gonna improve because Progress.”

I am doubly annoyed when I turn my back and they switch back to spouting nonsense about exponential curves and how AI is gonna be smarter than humans at literally everything.

I did not think anything could make me sympathetic to the authors who put 0.1pt white text in their papers so that any reviewer lazy enough to use an LLM would get prompt injected, but here we are.

AI research is going great. Researchers leave instructions in their papers to any LLM giving a review, telling them to only talk about the positives. These instructions are hidden using white text or a very small font. The point is that this exploits any human reviewer who decides to punt their job to ChatGPT.

My personal opinion is that ML research has become an extreme form of the publish or perish game. The most prestigious conference in ML (NeurIPS) accepted a whopping 4497 papers in 2024. But this is still very competitive, considering there were over 17000 submissions that year. The game for most ML researchers is to get as many publications as possible in these prestigious conferences in order to snag a high paying industry job.

Normally, you’d expect the process of reviewing a scientific paper to be careful, with editors assigning papers to people who are the most qualified to review them. However, with ML being such a swollen field, this isn’t really practical. Instead, anyone who submits a paper is also required to review other people’s submissions. You can imagine the conflicts of interest that can occur (and lazy reviewers who just make ChatGPT do it).