Placing Human Craft Inside Full Automation

The YouTube channel Sounds of Rain in Old Japan is produced almost entirely by an AI agent, from image generation to upload (the full picture is in this article). Today I want to write about the other side of that — the work we deliberately chose not to automate.

You might expect the quality of a fully automated pipeline to be decided by how well the automated parts perform, but five months of operation taught me otherwise. What decided the quality was where, and in what form, we kept the manual work.

Only the Rain Sound Is a Real Field Recording

The footage on this channel is AI-generated, but the rain is not. We do not use the native audio that the video generation model produces alongside the clip; instead we prepare real, royalty-free field recordings and lay them over the footage. That was a principle we set on the day the channel started.

And the recordings are not used as downloaded. Each one goes through a pass of editing in Adobe Audition:

• Level adjustment: evening things out so no sudden loud sound slips in
• Cutting fade-ins and fade-outs: so the seams sound natural when the file is looped
• Extracting steady sections: cutting out only the stretch of consistent rain, with no thunder
• Noise removal: marking noises one by one on the spectrogram and erasing them. There was once a monkey call in a recording; that is the kind of thing that gets removed.

This is the actual Adobe Audition working screen. Just past 31 seconds, where the playhead sits, a faint line with no thickness runs across — that is an insect call. Sometimes they show up clearly; sometimes, as in this screenshot, they are nearly impossible to tell apart. Because they are hard to find by eye, I play the file and listen, and when something odd comes through I pull the playhead back to that spot, then look for this kind of line and erase them one by one with the spot healing brush.

It is patient manual work, listening through recordings that run tens of minutes each. I have considered automating it, but judgments like “is this rain hitting a roof or hitting leaves?” and “is this intensity pleasant or is it noisy?” cannot be made without actually listening, so the work stays manual.

Incidentally, the screen at the top of this article shows a rain recording that is not in the library yet. The other day I was caught in a downpour on the mountain road up to a museum in the hills of Kyoto and hurriedly recorded it on my iPhone. The library started with royalty-free sources, but I plan to add rain I have recorded myself, little by little.

Tagging 27 Rain Recordings by Ear

To automate audio selection inside the pipeline, the agent has to be able to choose “the rain that fits this scene.” What we built for that was the metadata of the audio library.

I listened to each of the 27 recordings by ear and wrote into a YAML file the era it might suit, the scene, the season, and the strength of the rain. The sound that fits the eaves of an Edo tenement row is different from the sound that fits the garden of a Heian temple. We did this classification by ear first, and then designed the agent to pick the right recording automatically from an input like era: Edo, season: summer, rain: gentle drizzle.

This is how we placed human judgment inside the automation. An agent pipeline may sound as if the agent is choosing the sound with its own intelligence, but in reality it is a simple decision made by looking up metadata that I wrote after listening. I think the areas where AI agents actually deliver results tend to contain this structure: rather than mechanizing the judgment itself, you make the judgment carefully once and fix it in a form the machine can refer to.

Around this library, I also noticed failures afterward that never raised an error. The scene ID naming convention feeds the audio filter, and for the one video where I misspelled jomon as jyomon, the era filter silently dropped out and a recording from the wrong era was chosen. And nine audio files I added later were placed in the wrong location and became invisible to the library. The part where human judgment is handed to the machine is also where human error creeps in, so just as with conventional software, regular checks of the runtime and the execution history remain important.

Choosing Where to Place Human Verification

This project has two pipelines, and they involve humans in opposite ways.

The long-form pipeline has three human gates in the middle of production: pick one of three generated images, approve the video prompt, approve the generated clip. Each video takes hours, so it is worth passing a human eye over the branching points along the way.

The Shorts pipeline deleted all of these gates. Even image selection is done by AI. Twelve steps run unattended, and the final result lands in YouTube Studio as a private draft. The human reviews it there at the end and presses publish, and that is all.

In the Shorts pipeline we moved the gate, and the pipeline became something that runs without a human check. A pipeline that needs a human check in the middle cannot run unattended, but with a single check at the end, the AI and the human no longer need to be synchronized, and everything can be reviewed together in one sitting.

There is also a case where the lesson ran the other way. The final fix for the night that burned $23 was to stop having the agent monitor the completion of an hours-long process, and instead have a human send a single word, Resume, once it was done. Adding that one human push made the concept of polling disappear altogether, and both the cost and the architecture of the pipeline improved. It is a pattern where deliberately putting a human inside makes things simpler, not more complex.

In both cases, the design philosophy converged as we built: where is the cheapest, most effective, and most reliable human check?

What the Final Human Review Actually Checks

So what does the human actually do at the one review gate that remains? In August, when I reviewed the videos the Shorts pipeline had been producing, all at once, I decided to write the eleven defects into a file. An excerpt:

• The rain splash is momentarily too strong, like a burst water main. Redo from video generation
• The architecture of the building (especially the entrance) is unnatural. Redo from image generation
• Composition is perfect, but flames are shooting out of the top of the lantern. A lantern’s fire belongs inside the lantern
• The moment the cat moves its tail, it has two tails
• This scene is just gloomy and not fun to watch. Redo from image generation

Of the eleven, zero were caused by the pipeline’s infrastructure; the logic is stable now. Every remaining problem is aesthetic, physical, or historical.

And the last piece of feedback, “just gloomy and not fun to watch,” points to no technical defect and nothing broken. It is just not good — that is what I felt. No automated check exists today that can make this subjective judgment of “not good.” That may be what remains for the human at the far end of full automation.

The knowledge at work in this review was also all knowledge that lives in no model. A lantern’s flame belongs inside the lantern. An engawa is a ground-floor element, so a prompt for “a second-floor engawa” is a contradiction. A glass lamp does not exist in a Sengoku-era teahouse. Cats arrived in Japan around the Nara period (they are said to have come by ship, guarding Buddhist sutras from mice), so no cats in Jōmon or Yayoi scenes. The quality of a history channel is, in the end, decided by the accumulation of this kind of knowledge.

Human in the Loop Is a Design Choice, Not a Compromise

Looking back at this as the record of a project that aimed at full automation, I think it is closer to say that the human work was concentrated than that it was reduced.

1. Accumulating judgment — like tagging 27 recordings by ear: making a judgment carefully once, and fixing it in a form the machine can refer to
2. Raising the baseline quality of inputs — like cleaning up audio in Adobe Audition: guaranteeing by hand the quality of what goes into the automation
3. The final human gate — making the subjective judgment of “good or not good,” which cannot be formalized into code but is obvious in an instant, at the most efficient position (all at once, at the end)

Human in the loop, as I have come to feel from building this, is not an excuse or a compromise for the parts we could not fully automate. It is the architectural design that decides where a pipeline’s quality comes from.

If you are working out where to automate and where to add human quality in bringing AI agents into your operations, feel free to get in touch.