Researcher-in-the-Loop

AI is changing how research is done, and, I would like to believe, for the better. However, I increasingly see an important danger in delegating too much to these systems, and that danger has made me cautious, leading to a particular preferred setup philosophy I’m sharing in this post.

Skill-first versus agent-first

As some of the most popular terminal-based systems, Claude Code offers many features that can help you design tasks. Others, like Codex, Cursor, and Hermes, have their own vocabularies for roughly the same core features, and all of it is evolving rapidly. I spend some time each week trying to find new useful features, having to cut through the hype. Although many could help you structure your setup, two currently form the backbone of most setups:

A skill is a written procedure or protocol. It loads when your request matches and then runs in your main thread, one visible step at a time. You watch it, you can set it up to ask for your input along the way, and you can interrupt it.

An agent is a separate instance with its own context window. It receives a task, disappears, works somewhere you cannot see, and returns only a final result. You can verify only the output; this is closer to having someone else do the task.

Setups tend to either be skill-first or agent-first. Skill-first setups use skills as elaborate processes with phases, audits, loops, specified tasks, goals, and output. Agents exist, but skills summon them for simple execution tasks for which the process can be forgotten. Agent-first setups work a little differently; skills are thin subroutines or reference manuals, and the main work lives in a stable of autonomous (sub)agents.

So, one important question I asked myself at some point: Should I use AI in a skill-first or agent-first fashion? So far, I encountered both. Some peers create lighter skills that an army of autonomous research agents uses to process tasks (one economist shares such an agent-first setup publicly), whereas others, like myself, gravitate mostly toward skills to automate whole workflows.

Anthropic’s documentation offers a criterion I find clarifying: isolate a task in an agent when its intermediate output is disposable, meaning you will never look at the reasoning again, only the answer. Keep it in a skill when you want the procedure to play out where you can see and steer it. That seems to suggest that, as a researcher, I prefer a skill-first setup because I care more about the process than the result in most of what I do.

Why research leans skill-first

In research, a finding alone is not sufficient. Research requires both the finding plus a defensible and verifiable account of how it was obtained. A skill is a commitment device pointed at the model: you write the steps (i.e., the methodology) before the work runs. The model focuses on the minor, disposable tasks, while you focus on tracing the process and providing human judgment in a predetermined way.

An agent, by construction, makes those judgments for you and shows you only the destination. For example, under an agent-first setup, you are much more likely to produce a fabricated citation without realizing. Similarly, a robustness check that was quietly skipped looks exactly like one that passed. In agent-first setups, these mistakes are less likely to announce themselves. They surface later, in a discussion with co-authors if you are lucky or in a referee report if you are not.

What we are actually protecting

A result nobody can reproduce is not just a weaker result; it is not useful. Reproducibility is far from dead, even in accounting. Accounting now has its own open science journal, Accounting Open, and the Journal of Accounting Research recently decided that all submissions with human data will require preregistration after January 1, 2027.

Can an agent-first setup do reproducible research? I believe this is challenging. You can delegate an analysis to an agent. The agent will return a table, but how did it produce the table? You can ask an agent to merge and clean your data, but how did it precisely end up with the final dataset? You won’t be able to answer these questions, because the reasoning happened in a context window that no longer exists. There was simply never a moment at which the processes were documented and passed through your hands. You have delegated yourself out of the one position that made you accountable for the result, and out of the entire reason you’re a researcher.

I am not arguing against automation. A skill-first setup can automate a great deal of boring and manual labor, and researchers who refuse to automate will be less productive than those who do. My objection is to delegation, which is a different beast and only sometimes travels with automation. A skill is automation that leaves the right traces when designed right. An agent is usually a form of delegation that runs the risk of consuming too much of the research process by putting it in a black box.

Example: revising a working paper

Researchers must often revise working papers, and the process tends to be similar each time. Skills can help automate some of the boring steps, which saves a lot of time and automatically opens up more room for me to focus on the important parts of the process that require human input.

A skill I use frequently is called academic-paper-reviser. It takes one of my manuscripts and a feedback source, such as a discussant’s notes or a co-author team’s revision list, as input and its first move is not to automously edit anything. Instead, it parses the comments into clear and explicit tasks, which are then summarized in an ordered roadmap for me to go through, with preliminary suggestions for the edits. My job is to walk through the tasks and suggestions, together with the input the model has prepared. Any revisions and additions I submit are subsequently processed by subagents, whose only task is to execute the final plan. Skills like these work even better if you provide them with more context examples of how tasks are ideally executed.

<skill>/
  SKILL.md       # the entry point: what the skill does and when to use it
  references/    # deeper task details and agent specifications
  scripts/       # helper scripts that the skill runs
  assets/        # files a skill stages into a project
  templates/     # templates the skill uses
  examples/      # real past artifacts used as style exemples
  evals.json     # example scenarios the skill is expected to handle, including boundary cases it should decline

At the end, I have an updated manuscript that I would have written myself, even without the help of this skill. The two purposes of the skill is to guide the process and automate a few of the boring tasks, such as some of the actual editing and file management on my computer. It also reasoned to give me a starting point for handling the comments. However, these starting point are far from random because you can feed your system and skills context and examples of how you usually process paper revisions. Skills like these become very valuable when you repeat certain work processes regularly (e.g., update last year’s slidedeck, create a conference presentation of this paper, etc.). This is where where much of our time usually went, and automating the boring mindless stuff, as part of a traceable repeatable process, also steers me more strongly toward those activities where I can add the most value. This two-sided leaver which is the result of automating tasks is the potential productivity gain of AI for my job right now, and it is far from being a delegation and replacement story.

In contrast to a skill design for this work process, an autonomous paper-reviser agents will also return a revised manuscript, perhaps even a decent one by pre-AI standards. However, there are no predetermined and verifiable stages. When you disagree with something, it will be harder to locate the moment it was decided, and you are reduced to arguing with the final result, which is to say reviewing your own changes as a stranger. The revision arrives as a fact rather than as a sequence of choices that should have been yours. I personally believe this work setup is also much more prone to the slippery slope of overreliance and ultimately mistakes and errors.

A workflow skill does the mechanical work, keeps the order straight for me, and together with me, it hands back something legible. There is always a researcher-in-the-loop: I give permissions, provide judgment when it is needed, audit intermediate output, and must always read anything that is produced properly at the end. Some of what it produces I regularly discard or correct, which also can give input for skill improvement over time. That last step is the reason the setup is safer to use, and a setup that makes that step impossible has not saved me labor.

Where this breaks

Every argument has a failure case that proves the point. Mine sits at the beginning of a research project. When I am generating research ideas or turning a hunch into a proposal, a heavy skill is a less effective instrument, and sometimes an actively harmful one.

A skill encodes a fixed procedure that should work every time. But early-stage research is the phase in which fixed procedures can become a straitjacket. A well-built skill might take a half-baked research idea and march it confidently down one path, and I will never see the four other papers I might have written. It will suppress exactly the variance I need.

Therefore, some parts of our work do benefit from an agent-first setup, but most of our work does not. As soon as your research ideas begin to take shape and you’ve gathered input and context, a well-designed skill becomes the dominant instrument. For instance, I have a skill called academic-paper-proposal, which walks me through the process and produces an initial document, containing the research idea in the format I like (e.g., a summary, Kinney paragraphs, Libby boxes, and hypotheses).

Should all professions use a skill-first setup?

Research is an unusual profession: it values reproducibility and the process itself just as much as the outcome. Although some other jobs share this focus, not all do. For instance, you will see many agent-first setups in the entrepreneurial and founder community, and they are not making a mistake. For someone building a company, the method is not the claim. Nobody audits how you found your customers or came up with your idea when you end up being successful. The market is a fast and free verifier, and the tasks rarely repeat. Encoding a procedure for something you will do once is inefficient. So, entrepreneurs typically rely on many autonomous agents that focus on idea generation and running other start-up processes.

Keep the researcher-in-the-loop

If you want the model to do the boring, mechanical work while you do the research, you need a setup in which the mechanical work leaves a trace you can inspect, and the judgment calls arrive at your desk one at a time. That is a skill-first setup. If you delegate to autonomous agents, you will get output, but you will have no trace of the process, and you will slowly stop being the person who made it.