Researcher-in-the-Loop
In recent months, I have been experimenting intensively with CLI agents, like Claude Code. My current setup automates many of the more boring and time-consuming tasks I usually had to do by hand. I have reached a point where most of my daily research activities now happen in the terminal, or in some cases in an IDE, like VS Code. Generative AI is changing how research is done, and, I would like to believe, for the better. However, I increasingly see an important danger in delegating too much to agents and other AI tools. That danger has made me cautious, leading to a particular preferred setup philosophy I’m sharing in this post.
Skill-first versus agent-first
CLI agents and other agentic systems have accumulated a lot of surface area. Claude Code alone offers many features that can help you design tasks. Other CLI agents, like Codex and Hermes, have their own vocabularies for roughly the same core features, and all of it is evolving rapidly. I spend some time each week cutting through the hype, trying to find new useful features. Although many could help you set up your CLI agent, two currently form the backbone of most setups:
A skill is a written procedure or protocol. It loads when your request matches and then runs in your main thread, one visible step at a time. You watch it, you can set it up to ask for your input along the way, and you can interrupt it.
An agent is a separate instance with its own context window. It receives a task, disappears, works somewhere you cannot see, and returns only a final result. You can verify only the output; this is closer to having someone else do the task.
Most setups I have seen are either skill-first or agent-first. Skill-first setups use skills as elaborate processes, with phases, audits, loops, including a specified tasks, goals, and output. Agents exist, but skills summon them for limited tasks for which the process can be forgotten. Agent-first setups work a little differently. In such setups, skills are thin subroutines or reference manuals, and the main work lives in a stable of autonomous agents.
So, one important question I asked myself a while back: Should I use CLI agents in a skill-first or agent-first fashion? I have seen peers use both. Some create lighter skills that an army of autonomous research agents uses to process tasks (one economist shares such an agent-first setup publicly), whereas others rely primarily on skills to automate whole workflows.
Anthropic’s documentation offers a criterion I find clarifying: isolate a task in an agent when its intermediate output is disposable, meaning you will never look at the reasoning again, only the answer. Keep it in a skill when you want the procedure to play out where you can see and steer it. That seems to suggest that, as a researcher, I prefer a skill-first setup because I care more about the process than the result in most of what I do.
Why research leans skill-first
In research, a finding alone is not a sufficient result. A sufficient result is the finding plus a defensible and verifiable account of how it was obtained (this is the well-known reproducibility principle). A skill is a commitment device pointed at the model: you write the steps (i.e., the methodology) before the work runs. The model focuses on the minor, disposable tasks, while you focus on tracing the process and providing human judgment in a predetermined way.
An agent, by construction, makes those judgments for you and shows you only the destination. Under an agent-first setup, it is much more likely to produce a fabricated citation without realizing. Also, in an agent-first setup, a robustness check that was quietly skipped looks exactly like one that passed. None of this announces itself. It surfaces later, in a discussion with co-authors if you are lucky or in a referee report if you are not.
What we are actually protecting
A result nobody can reconstruct is not just a weaker result; it is not useful to us. Reproducibility is far from dead, even in accounting. Accounting now has its own open science journal, Accounting Open, and the Journal of Accounting Research recently decided that all submissions with human data will require preregistration after January 1, 2027.
Can an agent-first setup do reproducible research? You can delegate an analysis to an agent. The agent will return a table, but how did it produce the table? You can ask an agent to merge and clean your data, but how did it precisely end up with the input dataset? You won’t be able to answer these questions, because the reasoning happened in a context window that no longer exists. There was simply never a moment at which the processes were documented and passed through your hands. You have delegated yourself out of the one position that made you accountable for the result, and out of the entire reason you’re a researcher.
I am not arguing against automation. A skill-first setup can automate a great deal of boring and manual processes, and researchers who refuse to automate will be less productive than those who do. My objection is to delegation, which is a different beast and only sometimes travels with automation. A skill is automation that can leave a trace when designed right. An agent is usually a form of delegation that runs the risk of consuming too much of the research process by putting it in a black box.
Example: revising a working paper
I revise many working papers, and the process I go through is similar every time. Skills help me automate some of the boring steps, saving a lot of time, and structuring a procedure that I would always like to follow.
A skill I use frequently called academic-paper-reviser. It takes one of my manuscripts and any feedback source, such as a discussant’s notes or a co-author team’s revision list, and its first move is not to edit anything. Instead, it parses the comments into tasks. It then produces an ordered roadmap for me to follow, with preliminary suggestions for the edits. It enables me to walk through the tasks, letting me tackle each in turn, together with the input the model has prepared. Skills like these work even better if you provide them with more context examples of how the task is ideally executed.
At the end, I have an updated manuscript that I would have written myself, even without the help of this skill. The skill automated only a few boring tasks, such as some of the editing, file management, and documentation. It also reasoned to give me a starting point for handling the comments. All of this sounds trivial perhaps, but that’s where most of our time usually goes. Automating these steps, as part of a traceable process, steers me toward the activities where I can add the most value.
A autonomous paper-reviser agents will return a revised manuscript, perhaps even a decent one by pre-AI standards. But there are no predetermined and verifiable stages. When you disagree with something, it will be harder to locate the moment it was decided, and you are reduced to arguing with the final result, which is to say reviewing your own changes as a stranger. The revision arrives as a fact rather than as a sequence of choices that should have been yours.
A workflow skill does the mechanical work, keeps the order straight for me, and together with me, it hands back something legible. There is always a researcher-in-the-loop: I audit it, read it properly, give my input when prompted, and take what I need. Some of what it produces I discard or correct. That last step is the reason the tool is safer to use, and a setup that makes that step impossible has not saved me labor. It has quietly moved up my productivity line.
Where this breaks
Every argument has a failure case that proves the point. Mine sits at the beginning of a research project. When I am generating research ideas or turning a hunch into a proposal, a heavy skill is a less effective instrument, and sometimes an actively harmful one.
A skill encodes a fixed procedure that should work every time. But early-stage research is the phase in which fixed procedures can become a straitjacket. A well-built skill might take a half-baked research idea and march it confidently down one path, and I will never see the four other papers I might have written. It will suppress exactly the variance I need.
Therefore, some parts of our work do benefit from an agent-first setup, but most of our work does not. As soon as your research ideas begin to take shape and you’ve gathered input and context, a well-designed skill becomes the dominant instrument. For instance, I have a skill called academic-paper-proposal, which walks me through the process and produces an initial document, containing the research idea in the format I like (e.g., a summary, Kinney paragraphs, Libby boxes, and hypotheses).
Should all professions use a skill-first setup?
Research is an unusual profession: it values reproducibility and the process itself just as much as the outcome. Although some other jobs share this focus, not all do. For instance, you will see many agent-first setups in the entrepreneurial and founder community, and they are not making a mistake. For someone building a company, the method is not the claim. Nobody audits how you found your customers or came up with your idea when you end up being successful. The market is a fast and free verifier, and the tasks rarely repeat. Encoding a procedure for something you will do once is inefficient. So, entrepreneurs typically rely on many autonomous agents that focus on idea generation and running other start-up processes.
Keep the researcher-in-the-loop
If you want the model to do the boring, mechanical work while you do the research, you need a setup in which the mechanical work leaves a trace you can inspect, and the judgment calls arrive at your desk one at a time. That is a skill-first setup. If you delegate to autonomous agents, you will get output, but you will have no trace of the process, and you will slowly stop being the person who made it.
