The AI Spork Problem: Why One-Size-Fits-All LLMs Fall Short — and How 'Just-in-Time' Objectives Could Fix Them
The central argument is the 'spork problem': today's LLMs collapse every possible use case — personal assistant, expert advisor, copy editor — into a single generic interface. Because developers cannot anticipate every user situation at training time, and because fine-tuning against many annotators' preferences pushes models toward averaged, bland outputs, users get worse results than they would from purpose-built tools. Research confirms that prompting is effortful and that most users don't know how to articulate what they need.
The proposed solution is just-in-time (JIT) objectives — lightweight models of a user's in-the-moment goals, automatically induced by observing their interactions rather than requiring explicit prompts. In a 200-participant study, induced objectives were rated accurate or very accurate by the vast majority of users and were chosen over manually specified alternatives in ~98% of cases. JIT generators were preferred ~70% of the time in head-to-head comparisons against baseline models, and JIT evaluators added a further ~16 percentage point lift in output quality — roughly equivalent to scaling compute by an order of magnitude.
Two concrete systems demonstrate the approach. Poppins, a browser extension with on-demand UI generation, produced writing outputs rated significantly higher than ChatGPT-style baselines in lab sessions, with one participant noting the quality matched advice from real-life advisers. Ditto, a style fine-tuning technique, achieves strongly personalized model behavior from as few as five writing samples, outperforming manual prompting. On the text analysis side, Loom replaces vague topic-model keywords with steerable, natural-language concepts (e.g., "government accountability") backed by explicit inclusion criteria, and has been adopted by researchers across academia and industry for datasets ranging from policy documents to product reviews.
A third research thread addresses societal-scale objectives. The guest argues social media AI is optimized for the wrong thing — engagement signals like likes — while harms to democracy leave no observable training trace. By translating social-science constructs (e.g., an eight-dimension partisan animosity scale) into AI objective functions and re-ranking feeds accordingly, the team significantly reduced partisan animosity among both Democrats and Republicans in pre-registered experiments later replicated in a *Science*-published field study on real X/Twitter feeds.
Key concerns raised include the risk that JIT systems could homogenize outputs at a population level rather than truly individualizing them, and that relying on one version of an objective-induction system encodes biases toward tasks an LLM can readily understand. The guest's preferred resolution is user ownership: giving people visibility into how their objectives are induced and the ability to intervene — treating the current systems as an entry point, not a final answer.
Researcher Damaris Fuentes-Rivera argues that LLMs are 'sporks' — multi-purpose tools that do everything adequately but nothing well. She presents a research agenda around 'just-in-time objectives' that automatically infer what a user needs in the moment, yielding measurably better outputs across writing assistance, text analysis, and social media curation.
because J objectives were sufficiently accurate, they reduced user effort, having them focus not on prompting the language model, but focusing on the type of feedback they wanted
end users often don't know exactly what they want or maybe how to articulate it with AI systems and J objectives address this by inducing specialized objectives on the fly at interaction time
what the analysts found was actually that they're written from one party's perspective and are critiquing the opposing party's stance, not just talking about their own position.
for pressing societal issues, we might know what we want to achieve, but we might not know how to achieve it with our existing systems. And I show that transferring knowledge built up in the social sciences can allow us to operationalize these societal objectives and realize them just in time in users feeds.
interfaces today are largely rigid and predefined and I think that's why we see AI sporks everywhere.
The ideal outcome is that users have ownership over objective induction processes themselves, with visibility into and ability to intervene on how their objectives are induced
research has found that prompting is effortful and challenging
At training time, developers don't know all of the user situations they'll be faced with. So they bake in assumptions about many potential user situations in the data and the training processes.
post-training processes that fine-tune models against many possible objectives like many different values or many diverse annotators preferences can lead the model towards these more generic and spork-like outputs
when we embed these types of societal values deeply, we can do things like reduce partisan animosity for thousands of participants
we ran an experiment with over 200 online participants who uploaded screenshots of their own tasks
The vast majority of these objectives were deemed by these participants as accurate or very accurate.
induced objectives were chosen in about 98% of cases and user preferences were aligned with these induced weights
JIT generators produce outputs that are preferred about 70% of the time in head-to-head comparisons
JIT evaluators help us to select more user preferred outputs, further hill climbing on this objective, jumping up about 16 percentage points from 10 samples to one sample, which is almost as much benefit as scaling up to a whole order of magnitude more samples
by iteratively treating the users demonstrations as unilaterally better than anything the model generates across time steps, we're able to produce much better and more user aligned behavior from the model
methods like this can strongly steer a model and bring performance benefits even above manual prompting
we ran hourlong inlab sessions with 17 participants who brought in their own writing tasks
Poppins produced outputs that were rated as significantly higher quality than that of a baseline language model something very similar to an interaction to chatGPT that participants often were already using
a participant who was already very familiar with chatGPT and had used it during their work was surprised that they actually got higher quality outputs out of the system out of the box even on par with advice they had gotten from their real life advisers
we developed and open sourced a system as a Python package and it's been adopted by a number of researchers in academia and industry
instead of vague keywords concepts are natural language descriptions like government accountability and they're defined by explicit inclusion criteria.
analysts can hone in very quickly on unanticipated trends like there was a benign looking concept of um political party positions that had high partisan animosity
Loom is able to actually recover more specific concepts. Things like economic stability that might characterize what's happening within a broader concept like economic policy.
we ran a series of evaluations with synthetic data sets that allowed us to verify that this pattern held true.
researchers also applied Loom to a really wide variety of types of text, even things we didn't anticipate like visualization notebooks and product reviews.
a major problem is that social media AI today are centered on the wrong objective. Um, a generic objective of engagement signals such as likes and clicks.
a recent mega study um published in science had major results on reducing partisan animosity. Um, and they looked at things like interventions such as videos and educational activities.
the construct had eight dimensions that um were things like opposition to bipartisan cooperation. They each had a definition and associated survey measures.
we found that manually re-ranking feeds using this APA objective significantly reduced partisan animosity both for Democrats and Republicans in these pre-registered experiments.
we found that we can use language models to replicate this type of annotation um to a fairly high degree.
we found that we're able to also significantly reduce partisan animosity with this intervention both Democrats and Republicans.
field experiments run by amazing colleagues translated sole objective functions into real users X feeds um with a browser extension and they confirmed that in the wild decreasing or increasing Apple exposure shifted partisan animosity and this work was published last year in science
other exciting work has shown that we can expand beyond our demonstration of this one value of democratic attitudes we can expand to a much larger library of values and that users might be able to control themselves.
we can already fine-tune models, open models with a handful of examples like I talked about earlier. But there aren't many end users who are doing this. So, it's not just a technical problem, it's also a usability one.
AI systems assisting with security problems may embed assumptions about individual security rather than organizational or cross-country security concerns
Users solicited for task contributions were not told about the AI system or what would come afterwards during the technical evaluation
Task collection requirements were kept as loose as possible to avoid easier inputs and to achieve ecological validity
Algorithmic down ranking produced a better result than manual down ranking in one of the studies
I'm not sure if it's significantly better
when we don't override sporks we can end up with things like homogeneous thinking across the population when people are relying on the same tool
feeds that amplify political polarization and impact our society
today, no matter what my interests are as an analyst, the topic model returns the same set of generic themes from the data set and they're often hard to understand because they're so vague.
social media plays a really central role in our lives and there's been increasing concerns that social media is harming democracy.
there's been increasing evidence that it's associated with increases in partisan animosity. These negative thoughts, feelings, and behaviors towards a political out group.
threads to democracy don't yield observable engagement signals to train on. Um there's not a trace when a user sees a post that lowers their opinion of the opposing party or reduces their trust in a practice like voting.
current LLM logs only really have visibility into a small portion and they're often only using AI on a small set of tasks that are useful to them with limited ability to raise problems and maybe even less ability to propose their own solutions.
do you think your research um uh um has u enough fidelity um uh for human decision making
the interfaces that this study has done has to do with checkboxes and pieces of text and drop downs etc. Do you think that in the future research we need to go kind of even one level deeper into you know written written marks
The development set used to tune prompts may have attributes a language model was better able to hone in on than manual readers, potentially explaining algorithmic advantage
JITs might be homogenizing outputs at the level of broader categories of people rather than truly individualizing them
Having users rely solely on one version of the objective induction system risks encoding biases toward tasks a language model can understand or tasks that are well scoped and bounded
LLM is a personal assistant, a topic expert, a copy editor, a confidant, and so many more things all at once — all squished into one interface with an identical input interaction no matter what we're doing, and we end up with these generic outputs from the model
I call this a spork problem. By combining many functionalities into one interface, we actually get worse results.
my research focuses on just in time AI interactions. These enable specialized AI interactions with just in time objectives, much like your own color drawer that has the specific tools you need in the moment.
just in time objectives are a model of the user's in-the-moment goals that's automatically induced by observing their interactions
We introduced concept induction which is an approach for surfacing highle concepts from unstructured text where our goal here is to make the default output steerable by design.
To achieve this vision, we developed an algorithm called Loom that's made up of some modular operators that we can apply iteratively.
the synthesize operator is the most important. Um it's taking the key text from the documents and drawing these unifying concepts across them using a language model.
the distill operator basically addresses the fact that not everything in every document is important. We want to really be able to zoom in to the parts that are relevant for our analysis.
the cluster operator allows us to make sense across these undifferiated pool of key points and identify cross cutting commonalities. So we detach the key points from their documents and then reccluster them using embeddings and HGB scan
the score operator allows us to go and verify whether these concepts actually occur by independently mapping back to the documents.
The seed operator allows the user to steer any operator using a seed term that modifies what it attends to.
Our work introduces societal objective functions which is a method for translating these types of social scientific constructs into AI objective functions where first we identify a relevant societal a social science construct like the opa construct and then we operationalize these into a manual intervention
I envision just in time personal and computing with computers that mold entirely to us as we work where objectives might be these more universal routers to different adaptive computing environments that could be generated and adapted on the fly.
I envision an AI interaction observatory where uh infrastructure like automatic pipelines might allow us to learn from large scale end user objectives um where we can group by combinations and trajectories of objectives that allow us to understand commonalities not just in what users are doing but why.
in some of my past work, we simulated juries of community members. Um, could we assemble juries for setting these types of broad objectives?
Understanding the assumptions embedded in an AI system is critical when using it to assist with domain-specific problems like security
The demonstrated system is an entry point for thinking about and supporting a broader space of ways to induce objectives and act on them, not a final solution
I worked on the Ditto project co-led with Omar Shik and Joey Hannah, and we developed a technique to fine-tune a language model to a distinct user style just using a handful of demonstrations on the order of five writing samples
we built an interactive system called Poppins that demonstrates objectives with on demand UI generation in a browser extension
we released this as an open source package for data analysts um, in their notebooks and we created some documentation and guides to try to make this accessible beyond the paper.
We could build systems that automatically fine-tune models for frequent user objectives that are occurring and learn how to manage and serve those models um in the background on a user's device.
Users were asked to provide a screenshot of some task they were actually working on, with a system that verified the screenshot was valid
“Just by focusing on improving our objectives, we have this headroom to really raise the performance of even off-the-shelf models.”
“is this a spork? Who designed it that way? And what assumptions are encoded by that decision?”
“it would be a real risk if we just had users use our version of the system and it always induces their objectives because that may encode biases for example towards tasks that a language model can understand or um tasks that are well scoped and bounded and able to be tackled with the solutions it can propose”
Shared with Earmark · earmark-ai.com