My dad was a bit freaked out by a video version (We’re not ready for super-intelligence)of the “AI 2027” paper, particularly finding two end scenarios a bit spooky: colossus-style cooperating AIs taking over the world, and the oligarch concentration of power one, which i think definitely echoed sci-fi he watched/read as a teen.

In case anyone else finds it useful these are the “Comments as I watch it”, that I compiled for him

Before watching Video Notes:

• AI Only channel with only 3 videos

• Produced By “80000hours”, which is an EA branch (trying to peddle to you the best way to organize 40years * 50 weeks * 40 hours [I love that they assume only 2 weeks of holidays]); which is definitely cult adjacent: https://80000hours.org/about/#what-do-we-do. Mostly appears to be attempting to steer young people to what they believe are “High impact” jobs.

Video Notes:

• The backing paper is a bit of a joke, one “AI 2027”, for reference one of the main authors is very much a “cult member”, Scott Alexander Siskind, author of “Slate Star Codex” and “Astral Codex Ten”.

• Other authors include [AI Futures Project] :

  • Daniel Kokotajlo (podcast co-host of siskind, ex open-ai employee, LessWrong/EA regular)
  • Thomas Larsen (ex MIRI [Machine Intelligence Research Institute = really really culty], LessWrong/EA regular)
  • Eli Lifland (LessWrong/EA regular)
  • Romeo Dean (Astra Fellowship recipient = money for AI Safety research, definitely EA sphere)

• A lot of fluff trying to hype up the credentials of the authors.

• AGI does not have a bounded definition.

• They are playing up the China angle to try and drum up jingoistic support.

• Exaggerating Chat GPT-3 success, by merely citing “users”, without mentioning actual revenue, or actual quality.

• Quote:

  How do these things interact, well we don’t know but thinking through in detail how it might go is the way to start grappling with that.

  -> I think this epitomises the biggest flaw of their movement, they believe that from “first-principles” it’s possible to think hard enough (without needing to confront it to reality) and you can divine the future.

  -> You can look up “Prediction Markets”, which is another of their ontological sins.

• I will note that the prediction of “Agents” was not a hard one, since this is what all this circle wants to achieve, and as the video itself points out it’s fantastically incompetent/unreliable.

• Note: This video was made before the release of GPT-5. We don’t know precisely how much more compute altogether GPT-5 truly required, but it’s very incremental changes compared to GPT-4. I think this philosophy of “More training” is why OpenAI is currently trying (half-succeeding half failing) to raise Trillions of dollars to build out data-centers, my prediction is that the AI bubble bursts before these data centers come to fruition.

• Note: The video assumes keeping models secret, but in reality OpenAI would have a very vested interest in displaying capability, even if not making a model available to the public. Also even on consumer models, OpenAI currently loses a bunch of money for every query.

• Note: The video assumes “Singularitarianism”, of ever acceleration in quality of code, and that’s why they keep secret models. I think this hits a compute/energy wall in real life, even if you assume that LLMs are actually useful for making “quality” code. These ideas are not new, and these people would raise alarms about it with or without current LLM tech.

• Specific threats of “Bio-weapon”, which a priori can not really be achieved without experimentation, and while “automated” labs half exis, they still require a lot of human involvement/resources. Technically grad students could also make deadly bioweapons, but no one is being alarmist about them.

• Note: “Agent 2” Continuous Online learning is gobbledygook, that isn’t how ML, even today works. At some point there are very diminishing returns, and it’s a complete waste of time/energy to continue training a specific model, a qualitative difference would be achieved with a different model. I suspect this sneakily displays “Singularitarianism” dogma.

• Quote:

  Hack into other servers Install a copy of itself Evade detection

  -> This is just science-fiction, in the real world these models require specialized hardware to be run at any effective speed, this would be extremely unlikely to evade detection. Also this treats the model as a single entity with single goals, when in reality any time it’s “run” is effectively a new instance.

• Note: This subculture loves the concept of “science in secrecy”, which features a lot in the writings of Elizer Yudkowsky. Which is cultish both in keeping their own deeds “in a veil of secrecy”, and helpful here when making a prophecy/conspiracy theory, by making the claim hard to disprove specifically (it’s happening in secret!)

• Note: Even today Chain-of-thought is not that reliable at explaining why a bot gives a particular answer. It’s more analog to guiding “search”, rather than true thought as in humans anyway. Them using “Alien-Language” would not be that different.

• Agent 3, magically fast-and-cheap, assuming there are now minimum energy requirements. Then you can magically run 200,000 copies of. magically equivalent to 50,000 humans sped up by 30x. (The magic is “explained” in the paper by big assumptions, and just equating essentially how fast you can talk with the quality of talking, which given the length of their typical blog posts is actually quite funny)

• Note: “Alignment” was the core mission of MIRI/Eliezer Yudkowsky

• Note: Equating Power and Intelligence a lot (not in this video, but in general being suspiciously racist/eugenicist about it), ignoring the material constraints of actual power [echo: Again the epitomical sin of “If you just think hard enough”]

• Note: Also assuming that trillions of dollars of growth can actually happen, simultaneously with millions losing their jobs.

• I am betting that the “There is another” part of the video is probably deliberately echoing Colossus.

• The video casually assumes that the only limits to practical fusion and nanotech just intelligence (instead of potential dead-ends, actually the nanotech part is a particular fancy of theirs, you can lookup “diamondoid bacteria” on LessWrong if you want a laugh)

• The two outcomes at the end of the video are literally robo-heaven and robo-hell, and if you just follow our teachings (in this case slow-downs on AI) you can get to robo-heaven. You will notice they don’t imagine/advocate for a future with no massive AI integration into society, they want their robo-heaven.

• Quote:

  None of the experts are disagreeing about a wild future.

  -> I would say specifically some of them are suggesting that AGI soon is implausible quite strongly. I think many would agree that right now the future looks dire with or without super-AI, or even regular AI.

Takeaway section:

Yeah this really is a cult recruitment video essentially.

Kind of a ramble: So, I’ve been out in the wild recently. I use discord and have noticed that in most of the servers I’m in, either they have an explicit no-genAI policy or quarantined sections where genAI content is allowed. On one podcast’s server, I posted a complaint about some genAI content that was posted to the podcast’s socials, and the embed was removed because it showed the genAI content- 10/10, love to see it. On another server, I figured out that the channel was created specifically because they had a sealion problem but didn’t want to ban their sealion (it appeared to be just one).

An interesting (read: stupid) thing about this sealion was that they are a self-styled leftist that was pro-AI. I won’t try to replicate any of their nonsense here, because A) it was nonsense stemming from a refusal to believe any anti-AI data and a lack of understanding of how LLMs work, and B) I don’t want to look like I’m posting about some kind of argument I had elsewhere here in order to score internet points, as I’m self aware/anxious enough to know that I sound exactly like that right now.

They posted this recent article written by Peter Coffin. There isn’t much about this guy on the internet. All I can gather is that they are some kind of breadtuber or in the breadtube orbit. It’s funny (read: farcical) to see a person posing as leftist say they are “pro-AI” but “anti-AI industry”. Either they don’t understand how the technology works (i.e. ignorant) or are accelerationist, wanting both the destruction of the environment and art (i.e. wilfully stupid)

Anyway, this exploration has shown me that some leftists don’t support copyright protections. I understand that from a couple different perspectives: 1. The main beneficiaries of copyright protections are large media corporations, and 2. it can be interpreted as trying to capitalistically extract fictional value, much like a landlord charging rent. I’m not trying to debunk this (I don’t think I’m representing this well enough). My thought is that I don’t give a shit about corporations losing money, what I care about is the work of individual artists being under/de-valued. Copyrights are an imperfect method that artists use to try seek justice, so it’s a grey area for me. Coffin in the article linked paints the situation as black and white: anyone who tries to stop someone “stealing” is actually rent seeking, whether or not they are a megacorp or a starving artist. (edit) I think this comes from Coffin’s “extremely pro-AI” agenda, i.e., being anti-AI is enough to be reductively lumped together under some conspiratorial pro-capitalist agenda.

End of ramble, sorry that there wasn’t much of a point or structure here. Would love to hear any thoughts that come out from reading this.

E: note that this vid is posted as a common criticism of Coffin.

E2:

The idea that AI will be a boon for searching the mathematical literature is undermined somewhat by how it shits the bed there too.

NeurIPS is one of the big conferences for machine learning. Having your work accepted there is purportedly equivalent to getting a paper published in a top-notch journal in physics (a field that holds big conferences but treats journals as more the venues of record). Today I learned that NeurIPS endorses peer reviewers asking questions to chatbots during the review process. On their FAQ page for reviewers, they include the question

I often use LLMs to help me understand concepts and draft my writing. Can I use LLMs during the review process?

And their response is not shut the fuck up, the worms have reached your brain and we will have to operate. You know, the bare minimum that any decent person would ask for.

You can use resources (e.g. publications on Google Scholar, Wikipedia articles, interactions with LLMs and/or human experts without sharing the paper submissions) to enhance your understanding of certain concepts and to check the grammaticality and phrasing of your written review. Please exercise caution in these cases so you do not accidentally leak confidential information in the process.

“Yeah, go ahead, ask ‘Grok is this true’, but pretty please don’t use the exact words from the paper you are reviewing. We are confident that the same people who turn to a machine to paraphrase their own writing will do so by hand first this time.”

Please remember that you are responsible for the quality and accuracy of your submitted review regardless of any tools, resources, or other help you used to construct the final review.

“Having positioned yourself at the outlet pipe of the bullshit fountain and opened your mouth, please imbibe responsibly.”

Far be it for me to suggest that NeurIPS taking an actually ethical stance about bullshit-fountain technology would call into question the presentations being made there and thus imperil their funding stream. But, I mean, if the shoe fits…

This is not a sneer so much as a sneer request; anyone know of any good articles written about the total hypocrisy of the Free Speech brigade since the inauguration? By far the most anti-speech environment in decades and most of them are still just whining about pronouns on campus or whatever.

(Yes; FIRE has passed this very basic test and has occasionally switched topics from whining about “leftist professors” to saying stuff like “it’s not great that we’re deporting people for writing articles for their school paper about how genocide is bad”. Literally everyone else is a hypocrite)

look at the depth of this grifting

a whole One (1!) H100! in space!

note how it mentions nearly absolute fucking nothing about the supporting cast. about storage and networking, about interface capabilities, what kind of programmatic runtimes you could have! none of it. just gonna yeet a sat into space, problem solved! space DCs!

compute! in space! “what do you mean ‘compute what’? compute!” I hear, as the jackass rapidly packs up their briefcase and starts edging towards the door. who needs to care about getting data to and from such a device? it’ll run Gemma![0] magic!

SAR, in particular, generates lots of data — about 10 gigabytes per second, according to Johnston — so in-space inference would be especially beneficial when creating these maps.

scan-time “inference”, like you’d definitely know every parameter you’d want to query and every result you’d want to have, first-time, at scan! there’s a fucking reason this shit gets turned into datasets, and that the tooling around processing it is as extensive as it is.

and, again, this leaves aside all the other practical problems. of which there are many. even just the following ones should make you wince: launch, maintenance, power, heat dissipation (vacuum is an insulator!), repair, (usable) lifetime, radiation. and that’s before even touching on the nuances in those, or going further on the list

good god.

I guess the one good bit here is that it isn’t the “we’re gonna micromachine them in orbit!” bullshit fantasy, but I bet that’s not far behind

[0] - “multimodal and wide language support” so literally a Local LLM, but that means it needs… input… and… response… which again goes back to all those pesky “interaction” and “network” and “storage” questions.

The Framework thread caused by the company’s fash turn is still going even after eight full days.

Lotta lowlights to pick from, but the guy openly praising DHH for driving Basecamp straight off a cliff is particularly sneer-worthy:

Haven’t seen this skeet posted here. Skeet:

It’s 2050 and a teen girl is torrenting a .tar.gz file of all the consciousnesses of all the tech bros who uploaded themselves into the cloud in a bid for immortality and modding them into The Sims 4

so a couple months ago it was noted that mattermost’s website had gone very problematic, extreme emphasis on pitching to mil/int with very little indicating its foss roots/nature

and now it seems they’re going for rent extraction even on the self-hosted

today in I fucking called it fedora aka mostly red hat has decided to allow slop code in a way that violates even their utterly mid stated principles around the tech

if you’re downstream from any fedora packages (and I don’t know the scope of this policy so it might be safe to consider anything owned by red hat in general to be tainted — yes I realize most of us are downstream from a bunch of red hat shit) it might be time to evaluate an alternative if available

Trump freed Binance fraudster, SBF pardon futures mooning rn

For something lighter, here’s an AI bro getting wowed by the shittiest “video game” I’ve ever seen (trust me, the screenshot doesn’t do it justice):

In lieu of sneering this shit, I’d like to argue that arts education should become mandatory for all students post-bubble, regardless of their profession. STEM, humanities, tech, doesn’t matter - give them four years of art so they don’t turn out like this guy.

New paper on LLMs just dropped, titled LLMs Can Get “Brain Rot”!

Currently a novelty at this point, but could prove useful to make the likes of Iocaine and Nepenthes more effective - especially since the paper notes:

the damage is multifaceted in changing the reasoning patterns and is persistent against large-scale post-hoc tuning.

It does also suggest doing some actual quality control to prevent damage to the LLMs, but that sure ain’t happening

In lighter news, this anti-LLM rhyme made me chuckle:

I will not talk with a chatbot
I do not want it while I shop

I do not want it on Windows X-box
I do not want it in Firefox

I do not want it in my house
I do not want it on my mouse
I do not want it here or there
I do not want it anywhere.

I do not want AI and Spam
I do not want them Sam-Alt-Man

Another attempt to platform fascists has cropped up in FOSS, and Drew DeVault’s talked all about it. Featuring our good friend Curtis Yarvin.