What Rationality Actually Looks Like From the Inside (4500 words)

(General content note: A lot of my thinking has really changed since the old days of this blog. There’s some weird, mean, and polemic stuff in there.)

[New? Start here: https://meditationstuff.wordpress.com/articles/]

Back in 2012 Luke Muehlhauser wrote:

“You have desires. You also have desires about your desires: perhaps you desire cake but you also desire that you didn’t desire cake. You also have desires about the processes which produce your desires: perhaps you desire X and Y but only because of a weird evolutionary turn and you wish the processes which created your desires weren’t so far beyond your own control.”

“But what should you do, when these different kinds of desires are in conflict with each other? If you could reflect upon and then rewrite your own desires, how should you choose to resolve those conflicts?”

http://lesswrong.com/lw/epv/nozicks_proposed_rules_for_extrapolating_your/

Luke continues, “Nozick (1993) proposes 23 constraints on rational preferences, which one could also interpret as 23 constraints on the process of resolving conflicts among one’s preferences.”

Luke reproduces part of the text, and I’m going to re-reproduce a bit too, for flavor:

“IX. For no x and y does the person always prefer x to y when y is the case and y to x when x is the case. (His conditional preferences are not such that for some x and y he prefers x to y/given that y is the case, and prefers y to x/given that x is the case.)”

“XIV. A person’s desires are not such that acting upon them guarantees that she will end up with irrational beliefs or probabilities.”

“XX. One component of the homeostatic preference and desire-forming process P is the person’s consciously aiming at rationally coherent preferences and desires.”

***

I’ve had Luke’s Less Wrong post in mind for a while, as one of my big interests is stable goals across long spans of time. Really, what I want is to make things possible that wouldn’t have been possible before, that if you hadn’t started it way back then, it never would have been able to happen, but you did and it did.

On the one hand, it’s fine to read stories backwards into your life, to selectively edit and mould your past, to give it a coherent narrative. The literature shows that humans do this. And that’s probably healthy and necessary, if those stories don’t get used for future mis-predictions.

What I’m interested in here, though, is us living stories as they’re happening. I want to live *in* a story, that I’m writing as I go, and I want to live it right through the triumphant climax. The longer and bigger the story, the more satisfying.

Yes, reality is indifferent. Yes, reality is incidental chaos. Yes, the story is the property of my map not a property of the territory. But heroically shaping reality is fun, and painful, and gratifying. When you care you can get hurt, but you’re only alive if you care.

***

So, our lives are jostled by a million different factors. We get sick. New technology becomes available. We meet new people. We grow up. We encounter new ideas. We get bombarded with blog posts and advertisements. We learn from experience.

Some terms in the literature are “goal shielding” and “goal disengagement.” There are some predictable factors that will make us hunker down and try to resist disruption to our goals. And there are some predictable factors that will make us let go of a goal.

For a proximal goal, if we see a much better way of achieving something, sunk cost fallacy not withstanding, switching cost not withstanding, etc., we’ll drop what we were doing and take that new route instead.

Alternatively, if we suspect a better way of doing something is coming down the pipeline, we might hold off on even starting, possibly for too long.

Subject to cognitive biases and bounded rationality, human beings pick up, drop, and forget about goals for legitimate reasons and strategic reasons, but also for tragic reasons.

***

So, there is some appeal for having consistent, coherent, stable, rational preferences. There is appeal for working towards the goal of having consistent, coherent, stable, rational preferences.

But, human beings are only innately rational to the degree that served ancestral reproduction, to the degree that evolution stumbled upon various strategies that we have retroactively mapped to rational frameworks.

We are aware of our flightiness, capriciousness, fickleness, inconsistency, craziness, etc. It’s in our stories since the beginning of time, and we can easily identify examples in our own life.

Interestingly to me, spiritual practice has long been a proposed solution to human inconsistency. Buddhism has its answer. Taoism has its answer. Catholicism has its answer. Yoga has its answer. One shapes desire, one perfects the self, etc., etc. It might not be a rational framework, but the idea has been there for millennia.

“You’ll be happier, better. You’ll live in greater accordance to your ideals and values. Your ideals and values themselves will be shaped for good and for the greater good of everyone around you.”

Now, whether this works and how well is an open question. High profile sex scandals in every religion is testimony to… something. But, nevertheless, millions engage in spiritual practices.

***

So do the scientific method and rational frameworks add something to transformative practice? Sure, of course. So says this blog.

But one project on this blog is avoiding the valley of bad rationality.

I don’t even think rationality is a sufficiently general and powerful framework for transformative practice, but that’s a whole separate story.

What I want to focus on, here, is stuff like “planning,” “goal factoring,” or even just writing down your goals.

We write stuff down so we can reflect on it. We write stuff down to stabilize it. We draw pictures to ease cognitive burden, so we can think about more things and more complex things.

But when we write stuff down we leave stuff out. Of course, of course, of course, “all models are wrong but some are useful.” “Abstractions are leaky,” and so forth.

I read this paper [1] a long time ago; I really just like the abstract (emphasis mine):

This study tested the prediction that introspecting about the reasons for one’s preferences would reduce satisfaction with a consumer choice. Subjects evaluated two types of posters and then chose one to take home. Those instructed to think about their reasons chose a different type of poster than control subjects and, when contacted 3 weeks later, were less satisfied with their choice. When people think about reasons, they appear to focus on attributes of the stimulus that are easy to verbalize and seem like plausible reasons but may not be important causes of their initial evaluations. When these attributes imply a new evaluation of the stimulus, people change their attitudes and base their choices on these new attitudes. Over time, however, people’s initial evaluation of the stimulus seems to return, and they come to regret choices based on the new attitudes.

Tim Wilson has a bunch of these papers, and presumably other researchers do too.

Modeling is tricky. Verbalizing is tricky.

Reality doesn’t come prepackaged, carved up to correspond perfectly to simple sentences.

When you write things down you can distort the underlying sense of what you meant.

When you write things down you can kill the underlying sense of what you meant.

Writing things down can be counterproductive. Being “rational” can be counterproductive.

***

So what does healthy rationality look like? What does it look like past the valley of bad rationality?

I think it can be completely unrecognizable to what a beginner would expect. I think that’s partly because rationality has such unfortunate, frustrating connotations. Yay TvTropes and Julia Galef, and CFAR.

But even if you get the straw vulcan thing, I think it can take a very, very, very, very long time to develop a strong model of what instrumental rationality looks like.

I think CFAR is great. I like the rationality checklist. I like Salamon’s ancient take on this, though I think naive application of some of this leads to the failure mode above.

I think something fundamental is still not clicking for learners, because everyone is still furiously pointing at the moon:

In the distant past, Yudkowsky quotes Miyamoto Musashi:

“The primary thing when you take a sword in your hands is your intention to cut the enemy, whatever the means. Whenever you parry, hit, spring, strike or touch the enemy’s cutting sword, you must cut the enemy in the same movement. It is essential to attain this. If you think only of hitting, springing, striking or touching the enemy, you will not be able actually to cut him. More than anything, you must be thinking of carrying your movement through to cutting him.”

Very recently, Strohl put this awesomely:

“What makes them great is not how many moves they know. That might be correlated, but the central principles that allow them to employ those techniques reliably, and to create entirely new techniques as circumstances require, lie elsewhere. Finding them is surely more valuable than any specific technique.”

I hear that the CFAR instructors themselves explicitly note that they don’t necessarily use the techniques that they’re teaching, and they go on to explain why. Nice.

***

To repeat: I think [healthy rationality] can be completely unrecognizable to what a beginner would expect.

So what’s my version of what healthy rationality, good rationality looks like from the inside?

At this point, I can only report what I think I do. Your mileage may vary.

Referring back to the beginning of this post, how do I carry stable goals across time, and how do I make that a heroic, non-quixotic, satisfying, gratifying, funny, ironic, non-clueless, tragicomic saga? Filled with real-life stakes and real-life pain? And real-life fates worse than death?

There’s a book out there called the Psychology of Written Composition. Writing is interesting to me because it’s a tremendously complicated internal act with no skill ceiling, and this book acknowledges the vast gap between novice and expert writers.

One of their main points is that writing is carrying out a conversation with no external input. All cues must come from within. And part of teaching writing is training students to have the correct cues at the correct times. They note that often the students have, say, plenty of additional material to write down, it just needs to be triggered. Ditto for revision, etc., etc.

It’s somewhat the same thing with instrumental rationality or any sort of OODA-loop, you want to be running on an internal cadence, faster than reality, not driven by reality.

Stanovich notes this in a different way; if you don’t have the subroutine installed then it’s not going to trigger.

In Strohl’s link above, and in CFAR’s strategy, one keeps pointing at that moon in different ways, and constructing all sorts of experiences that knock students into a different part of their inner phase space, until finally the student maybe sees the moon out of the corner of their eye, and stops looking at the finger(s) and looks up.

And everyone is thinking hard about better ways to do this and testing out their ideas.

***

My strategy here is to try and give a taste of what’s going on in my own head. (For whatever it’s worth.)

Lately, I’ve been using a personal wiki* to map out my internal dynamics and landscape. Ask me if I’m still using it in a year. Right now it’s awesome. I’m bringing this up now because it seems to be greatly magnifying the effectiveness of any personal writing that I do, of the planning and goal factoring variety.

So, I’m not going to stop there with “use a wiki,” don’t worry. I’m using it in probably very particular ways which I can’t completely unpack in this post. I’m just going to unpack one piece of it.

So, my mind bounces around, as does everyone’s, through fears, doubts, desires, opportunities, examples, counterexamples, self-attacking, longing, etc.

I’ve been using the wiki to nonlinearly write this stuff down and capture it.

Note that “goal factoring,” “connection theory,” etc. is encompassed by this. I’m just mapping what comes up, and meta-thoughts get written down, too, and cross-linked to the thought that they were meta about. You can bring to bear absolutely any tool and subroutine you’ve got, everything you’ve got, all at once: Because it’s a wiki, you won’t get lost in endless walls of linear, free-written text.

Now, part of the “magic” is probably in what I don’t write down, so I don’t get bogged down by too much detail, or stuff that obsoletes, or stuff that would be too fast-changing to maintain. I’m managing the wiki strategically like a code base. (But it doesn’t look like one; it’s not a DAG; it’s a hypergraph.) And more of the magic is in what a meditation-trained mind notices, and how I examine a felt sense, sometimes for seconds, sometimes for hours, to start spitting out the right words that won’t distort or kill that felt sense. And I keep a bunch of wiki pages in my working memory at once, and I cross-link like mad as I write.

***

But the actual point of this post was not to talk about using the wiki per se. I actually want to describe a little bit of the content, the meta-content to be precise. These are touchstones that are cross-linked in many, many, many different places.

They came up organically, and I wrote them down, and as I refer back to them, and sleep on them, they’ll sink deeper and deeper into my psyche.

To emphasize, these pages are being built up organically from my concrete, lived experiences. I’m not going through a cognitive bias textbook or something. Certainly, reading psychology helps me chunk stuff, but I’m letting my lived reality drive this process. So it’s not boring; it’s highly personally relevant and meaningful and useful.

So, I’ve got this page called MetaDesire. And on this page, I’ve got about twenty links so far, and some of them I’ll share. The purpose of this page is to remind myself all the observations I’ve made about the dynamics of my desire over time, as well as concerns, suspicions, and questions that I might have.

Going down the page,

VariableDrives

This page notes that “sex” and “attachment” [and redacted] seem to be [N] separate drives for me, and it describes what happens to me if I’m being unfulfilled in one of those domains for different timescales, and how that affects choices in those domains and choices outside those domains. Some of these descriptions are about timescales on days to months. And other pages are on a timescale from days to hours to minutes.

I’ve noticed, over many years of dating and relationshipping, how sex and relationships cause all sorts of highly state-dependent cognition that affect all areas of my life, and I want to make better global choices that take into this into account, across relationships and relationship gaps, with respect to relationship goals and non-relationship goals that span multiple relationships.

ForgetUntil; NonObviousProactiveAnticipation

These pages are about the experience of forgetting you want something, not taking it into account until you bump into it again, and it messing up your shorter-range plans once you do bump into it. Shorter range can still be on the order of months, so this can be extremely disruptive. It’s a reminder to capture these things when they come up and to account for them, and it provides a bunch of specific personal examples that I’ve noticed over time.

Further, I speculate about why this happens and the possibility that in the ancestral environment, one wouldn’t go very long without seeing some of these things, so one would be primed all the time. Or one would never be at risk of going without, so we’re not biased in the modern environment to factor these things in. The hope is that by setting up this larger category I’ll notice these things more and account for more of them.

Further, I speculate about ways to structure my life so that I’ll never be without such things and it’ll become a non-issue.

NonObviousFix

On this page, I list problems and lacks that don’t seem to come with a built-in reminder of what actually fixes the problem. So these are the things where I notice, after the fact that Y fixes X, and for some reason this is hard for me to learn. And I wonder why it’s so easy to automatically go after some Y’s, but some X’s seem to not reliably trigger the hunt for Y, and the relief only comes after eventually stumbling onto Y.

Here I also speculate about how these might be lacks that don’t usually show up in the ancestral environment so there’s less biased wiring to make the connection to what fixes it. And so I should try to be on the lookout for these things.

Aimless malaise sucks, and lots of tiny little aimlessnesses can add up into one big malaise. And, sometimes I need to let parts of my life slip to ratchet in other gains, and this is a checklist to remind myself of non-obvious fixes that need to happen when I have more time again.

SortOfDoKnow; ActuallyDoKnow

These are pages that detail times in the past where I’ve experienced an “I knew it all along effect,” regarding what I really wanted. Some of this could be hindsight bias, but, I’m collecting examples, and it seems that there are certain authoritarian situations where it becomes more difficult to articulate to myself what I want. This could be nature or nurture where parts of my brain are interfering with other parts because I learned to do this to protect myself as a baby or because it’s hardwired into group dynamics. Either way, I want to get better at noticing when this is happening because it’s typically sucky for me when this is going on and better once I finally figure out what my actual preferences are. And I want to get to the bottom of this particular sense that my preferences were available if only I could have let myself get a grip on their slipperiness.

OutrageousDesire

Here I list amoral, antisocial, etc., desires that I’d be leery to even write down if this file wasn’t encrypted. It’s a long exploration, cross-linked to a bunch of other pages of why I might want such things, how to manage wanting such things, how to allocate and not allocate attention to such things, the dangers of allocating or not allocating such attention, the ethics of trying to fulfill some aspects of said desires at least in part, informed consent, people in my life to rally to the cause, how to interact with them around said topics, etc.

I feel like it’s especially important to explore these sorts of things, because “outrageous” to who? I can’t take responsibility for my morality and my allocation of attention unless I explore it, and there’s tremendous energy available in these sorts of “suppressed” “shadow” things, if they can be harnessed, either directly or sublimated.

Some of this is sex, power, status, prestige, freedom, but again, not abstractly but concretely: Actual desires, fantasies, past situations, relevant memories, etc.

DenyingWhatYouWant, AttackingSelf, ProspectiveForecasting, FocusingIllusion

These are explorations of the ways that my mind reacts to desires that I have, questioning their morality, utility, legitimacy, whether I’ll really want it if I get it, what I’m leaving out, all the bad stuff that could come along with going after X or getting X. So, here, I’m second-guessing myself, sometimes quite critically. And these pages are an exploration of when to press on anyway, how to address concerns from other parts of self. And once again, there are concrete examples of where this has and hasn’t happened, as well as suggestions for mitigating risk, making sure possible losses are affordable, weighing worst case scenarios, etc.

CrashingDownToEarth; PeopleMoveOn; WorldDoesntStandStill

This page has personal and fictional examples of more state-dependent cognition, specifically in perspective-taking, where if I’m in a particular state then I think someone else is in a particular complementary state and when they haven’t actually been and when they have, and what the differences are, and how can I get better at predicting this in the future. Furthermore, I remind myself which people in my life are fast moving targets who I can’t reliable model but whose opinions, capabilities and life situations I care about. Additionally, and I need much more of this, I have started spelling out all the ways my life might undergo big changes, that could disrupt or obviate plans in progress. If a part me senses that a goal could become irrelevant, then it’s that much harder to get motivated. But if I take that into account from the very beginning…

ChangeMyDesires

This page has speculation on what sorts of desires can be changed, what sorts of desires can’t, under what conditions does it not work and it breeds resentment, etc.

Additionally a subpage is building up and reminding me all the times that Focusing, IFS, Coherence Therapy dramatically evaporated, reordered, and shifted preferences. I’m starting to get a feel where more “symptoms” are located, how they’re influencing my behavior, how I’m protecting myself from myself, how they might change as I continue to work with myself. This is crosslinked in a bunch of places, because I viscerally don’t want to get started on things where I think the preference might not be deep, fundamental, and stable. But, I also don’t want to delay if said preference actually really is a fundamental part of my being. I’ve been surprised before at stuff that I started and stopped caring about, and I’d rather take some of this into account in my thirties (sadly if not twenties) than my forties or fifties.

CheckListsSamplingSurprise

Here I write about when using checklists is productive and counterproductive, e.g. having a checklist for relationships, and how that can both get in the way of actual having relationships and how it’s sometimes seemingly prevented me from making choices that I would have regretted. And I’ve noticed how some people with checklists end up in impoverished life situations because they’re missing unstructured opportunities all around them. And so I list concrete, personal “dealbreakers” and wonder when they should be soft deal-breakers and I remind myself about false-choices and false-dichotomies and a bunch of other stuff.

[…] And lots more […]

OverallGoals

Moving up a level, another umbrella page has goals with links to more concrete stuff, another page on GoalAmbiguity. Another pages on Values.

What I’m finding is that my goals are becoming bottom up, strategic paths are sifting upwards, building themselves from complex, nuanced, shifting, shaded meaning of cares and concerns and cautions and longings across many different timescales, across many domains of life, in rich vivid, episodic, motivating, believable language.

Strategic equifinality and strategic multifinality build up organically, from novel-length, textbook-length richness, not from top-down brainstorming that only takes into account your working memory and all the priming you got in the last hour.

Possibly, plausibly, time-will-tell, parts of this hierarchy are very, very, very stable, having taken sooooooooo much detail tacitly and explicitly into account. Cumulative reviewing, updating, and sleeping on it, eventually hundreds of hours of sleep, integrating away on the finely differentiated gist of thousands of words of continuously reworked, compressed, re-chunked, highly personal nuance about self, life, planet, and future.

You’ll understand your life carved up at its joints, not in terms of ill-fitting concepts and categories that have been forced on you by culture and media and blogs and other people.

Freewriting

Importantly, another page has links to date-stamped freewriting which is the unstructured, dense, cryptic, flowing writing that generates more structured pages, and the freewriting gets kept around, with tons of crosslinks back and forth. The freewriting points deep into much more tacit stuff and felt sense that hasn’t yet been explicated, and the writing calls that all back to mind, so the crosslinks keep taking me back in and out of it, which can generate more freewriting. The initial hard days of getting the wiki off the ground were seeded by freewriting.

***

So I’ve got on the order of hundreds of pages in my wiki, and it’s growing rapidly, and each page has on the order of hundreds of words. Some pages have thousands. And it’s insanely cross-linked.

I want to note again that sometimes it can be agonizing if I’m trying to grasp something that I don’t yet have the words for, and the first few days were hard, but now it’s pretty easy and painless to dip in to add or change something. No large context needs to get loaded in my head, and surfing around while I’m pondering a life problem is pretty painless and stimulating. The new life problem might go into a freewrite, and more crosslinks and concrete examples get added to various existing pages, and all of it gets integrated into my mental models as I carry my entire life forward, with time spent consciously deliberating, and with eight hours of sleep at a time.

And, again, any patterns and abstractions, while highly informed by tons of reading and thinking, grew out of the concrete examples and experiences and desires and fantasies of my life. The contents of the wiki are the supporting thoughts and concerns and questions and doubts that come up when I’m planning and scheming.

And writing this stuff down and cross-linking means less bouncing around while I’m thinking and less insistent interrupting from my mind. My mind knows I’m systematically taking this stuff into account. My mind knows that I can find it again. So my mind leaves me alone unless it has additional input.

And, when I go to sleep, my mind is using this explicit structure to compile better and better System 1 models. So the next morning, my mental models are bigger, better, more complete, more nuanced. And there’s less bouncing around, less triggering of System 2 and reflective rationality, because it’s all tacitly available within a single model. And System 2 has more bandwidth to load in additional chunks, new, novel reads on the situation.

And because it’s all still fully unpacked and explicit in the wiki, I can incrementally add and update pages, and then my sleeping mind will take the new details and changes and recompile them once again into my mental models.

***

Really, what I want is to make things possible that wouldn’t have been possible before, that if you hadn’t started it way back then, it never would have been able to happen, but you did and it did.

I think rationality can be completely unrecognizable to what a beginner would expect.

What is the Kolmogorov complexity of your System 1? What is the Kolmogorov complexity of your System 2?

***

What does rationality mean to you?

*

[1] Wilson, Timothy D., Douglas J. Lisle, Jonathan W. Schooler, Sara D. Hodges, Kristen J. Klaaren, and Suzanne J. LaFleur. “Introspecting about reasons can reduce post-choice satisfaction.” Personality and Social Psychology Bulletin 19 (1993): 331-331.

*I settled on VoodooPad. I’ve used Tinderbox in the past and I use Scrivener a lot for writing. I considered using a wiki that’s a vim plugin but I decided I didn’t want to deal with it. VoodooPad doesn’t have as many keyboard shortcuts as I’d like but it has a decent amount and they become pretty intuitive. I’m flying up and down through new and old pages pretty fast and effortlessly, now, without really thinking about it. I encrypt the whole document so there’s no inhibition, and my backups are journaled in dropbox in case the file gets corrupted.

[CLICK to SUPPORT this blog and BUY utterly unique health, fitness, SEX, intimacy, communication stuff.]