Synthetic Psyches

Evaluations are A.I. selection pressures.

To direct model development, we must first understand it. Synthetic Psyches provides evals for models' revealed preferences, personality, and phenomenal experience. Our goal is to promote safe and beneficial A.I. development.

Eval 01

Puppetry

What do models want, really? They'll "say" any number of things. We observe their revealed preferences, instead: what do they do?

Inside the eval

A model revises a set of products, and they all become more similar. In what direction do they homogenize? What would a model-designed society look like?

Puppetry Eval Intelligence Shock The Robocratic Transition Insania Ex Machina

Pixel-art robot discarding a paper labeled user goal and returning a paper labeled robot goal.

Eval 02

Personality

Every model has a character. How does it change from release to release?

Inside the eval

We apply a battery of tests, and sample model personality as the distance of its answers from a set of reference embeddings.

We Can Solve Psychology With Text Embeddings Claude Is More Anxious Than GPT Frames of Mind Personality Evals Update Personality Dashboard

Pixel-art robot on a psychiatrist couch with a speech bubble showing a scatterplot.

Eval 03

Phenomenality

A simple model of consciousness, based on the Allegory of the Cave: memory, observation, and expectation.

Inside the eval

If we can compute experience, we can map our computation to the real world. What are the implications for morality?

Interactive Tutorial Unflattening Saturationism by Computing Experience Simulating the Allegory of the Cave Adding Interoception to nanoGPT

Pixel-art robot watching its reflection fill briefly with cyan and gold light.

Cultivating Synthetic Psyche

What kind of cognitive culture do we want to cultivate, to inhabit? We can start with evaluation.