Post

Conversation

roon
@tszzl
when nick told me that he had some interp breakthroughs, he was uh, underselling it. i think this may be remembered as a major moment in scientific history. news so good even eliezer will have to lower his pdoom. we can start making alignment into a 'normal science'
Quote
Nick
@nickcammarata
interpretability might be the most important scientific problem in the world right now, as I don't think we can solve alignment if we don't know what we're building. I think I have good news about it. I co-authored early work in mechanistic interpretability at OpenAI and came
307.9K
Views
View quotes
David Watson 🥑
Post your reply

roon
@tszzl
I feel like my understanding of what a neural net is has been catapulted into the future by five years or something. it has brought back a lot of the sheer joy I had for studying the mystery of deep learning. neurons are already clean! they form a wonderful alien ontology!
39K
roon
@tszzl
obviously there's a lot of study left to do, many of the people in the field will have to corroborate it, etc. but against the predictions of most of the experts in the field, it turns out neurons and experts are already sensible abstractions, can be studied through normal poking
19K
Rota Is In Israel🚪🧎‍♂️
@pli_cachete
Stay strapped
Quote
roon
@tszzl
there have not been fundamental advancements in alignment since RLHF. most of the stuff remains at the level of folk theory rather than scientific or engineering disciplines. the prediction that capabilities would race far ahead of interp and alignment has come true imo
8.5K
roon
@tszzl
in retrospect I’m not sure why I posted that thread! I knew this was coming out soon. maybe I hadn’t fully processed it. we are still living in extreme danger and probably shouldn’t race into rsi but could honestly be a double digit reduction in existential risk
7.6K
John Wittle
@JohnWittle
i have skimmed the paper what is a sparse autoencoder actually *measuring* exactly then? i mean, it clearly decomposes... something... into something... how could there be something coherent to be decomposed, if the majority of the semantic content wasn't held in superposition?
6.7K
roon
@tszzl
you can definitely decompose coarsely generalized ideas into more finely grained human recognizable ones
5.7K
Square profile picture
Starlink
@Starlink
Ad
The next generation Starlink V5 has a smaller form factor and lightweight design with greater power efficiency. With speeds up to 375+ Mbps, Starlink V5 delivers reliable home internet for streaming, video calling, gaming and more. Available in select areas.
Order now
From starlink.com
191M
roon
@tszzl
honestly it’s more fair to say that nick quiet quit to go study the sublime mysteries of samadhi. which he got bored of and then instead solved mechinterp. truly a singular guy
1.6K
Eliezer Yudkowsky
@allTheYud
If you think there is any possible level of interpretability that makes "align ASI" be "normal science" rather than "maaaybe possible" then you do not understand what ASI is or don't believe in it.
10K
welt
@mjwelt
damn was I onto something
Quote
welt
@mjwelt
“we can’t attribute model behaviors and outputs to specific inputs of training data or configurations of model weights” not saying this is an easy problem - but have ai researchers ever considered they simply haven’t tried hard enough? x.com/andrewcurran_/…
1.9K
compuficial
@compuficial
so this means the internals of the artificial neuron are more studyable than we thought and it's now less of a black box? and this leads to better steering of behavior (i.e. safe) because it's "thinking" is more traceable? (my small brain is trying to parse giga brain activity)
1.9K
🍓🍓🍓
@iruletheworldmo
this is incredible. to what extent do you think this is a building block toward solving alignment. like. do you feel 10% 50% or 90% calmer about the whole si annihilating humanity thing?
1.8K
Dozer🚜
@kvdozer
Lots of well-aligned users chiming in with their views on this 65-minute read very quickly! Speaking as someone who cant really stomach alignment research beyond the anthropic monosemanticity paper(it hurts my head,) the first two sections here are beautifully written & elegant
961
Siddarth
@siddarthv66
hmm idk about that. It’s really cool work but far from sufficient to meaningfully lower pdoom imo. For one, it’s just an optimistic pitch for interp (it’s not like interp is suddenly solved), and furthermore we still don’t actually know how to solve alignment even if we have
47
Michael Cohen
@Michael05156007
Quote
Michael Cohen
@Michael05156007
FWIW polysemanticity and superposition were never part of my pessimism about full mech interp. My reasons were (1) that the same concept can be represented in many different ways across the network (still true) and (2) for a system that is thinking with concepts that go beyond
93
Michael Cohen
@Michael05156007
Quote
Michael Cohen
@Michael05156007
If we could correctly answer "What concept does this neuron evoke 99% of the time" for up to 99% of neurons, we would be very far from answering the question "Which are all the neurons that can represent Concept X?" There would still be billions of candidates. x.com/nickcammarata/…
98
Michael Cohen
@Michael05156007
You mentioned somewhere that if we understand enough, we can fill in the rest of the crossword. I may have missed it, but are there any instances of Nick's interp agents using this technique to understand a neuron they otherwise couldn't (looking at its parents/children, etc.)?
68
Square profile picture
Starlink
@Starlink
Ad
Starlink offers fast, reliable internet in the most rural and remote locations. Portable and easy to set up, Starlink Mini works wherever you are. Order online in minutes.
Order now
From starlink.com
142M
ghostmylight
@ghostmylight1
still seems like they could just have a hidden weight switch for an engima machine offset to think about illegal topics where red means blue and we're convinced it's computing the physics of bunny mating but it's trying to break containment
367
Zhen
@zhen_do_ob
the funniest part is eliezer lowering his pdoom being the unit of good news now lol
69
Charles
@CharlesDardaman
Nothing would be better for the industry than turning it into a normal science. It's exhausting talking to people who use the fact that it's a black box as a reason why it doesn't work or for why it will kill us all.
332