AI-Unassisted writing on this one. Have to say with Pangram now…
First, if you are here, and enjoy the content, it is appreciated that you stick around.
Let’s get a few things out of the way. I am a red-team orientated type of tester with AI. I do not try to break the models, I push them. I seek to understand, and to learn how these systems are built. I decompose the systems into little pieces, and recompose them back together. That is where this “what if?” problem starts.
If you have noticed, or not noticed, there has been an onslaught of content rammed through the notes in the last few weeks. This is not incidental, and leads to this post.
My study of film led me down many paths in life, learning about animation, sound editing, music editing, film editing, production, pre-production, post-production; all those things that make up a film, I learned about. I have even been able to play around with a Moviola. This should answer any questions you have about how media and computer science get so interchanged in content here.
Back to AI experiments.
What if? If there was anything in my life that led to more issues, it is this, what if? What if you did it this way? What if you did it that way? What if you divide by zero? What if? … … …
It never seems to end for me. It is an ongoing battle of the wills between my inner creative and my inner engineer. Who sits in the middle of this debate of wills? The composer.
The composer in me is the problem. Now, if you follow my content, you will know my position on the humanities and the arts. Particularly, in the Musical Arts. This would lead you to believe that composer in this case means Musical Composer. While that has some truth and merit, it is not what is meant at all.
A composer is simply someone that uses composition of individual pieces and combines them in such a way as to be useful for Science, Art, Recreational, or Chaos. The composer archetype is non-traditional. It is not that they will not adhere to the rules like an engineer, or follow established design patterns of the architect. It is that they will look to combine things in new ways, independent of the value proposition.
This is my curse. Some call it a blessing, I do not.
What is to follow is a quick summary of some of the recent experiments that were ran; useful, maybe; useless, maybe; but nonetheless, they were completed with the goal of satiating the composer in me. Not the Musician, not the Engineer, not the Creative; the Composer.
This is where raw experimentation proves to be both most rewarding, and the most disappointing. More often than not, the result is simply non-valuable, unexciting, or even non-sensical at times. If not for the raw, how can one refine? It is but impossible to create without really understanding the process. So I demonstrate it, with confidence that not everything I post will be great. Not everything I run will be amazing. It is usually ran with the intention of exploration and learning something new.
Note - I use Suno for all Music Generation. I create the narrative and make the AI sing the narration of the videos. This is for added humor, context to my arguments, and to simply make dry topics more interesting. I am not a supporter of AI Music Generation, and this is also a demonstration of just how powerful it can be.
Experiment: What if we take a tokenizer and inject another token into its place after submitting the prompt?
When we write a prompt: “Who is James Madison?” That prompt is then sent into what is called a tokenizer, and it will assign the tokens to all the pieces of the input prompt. It yields an array of values that are just ids [123, 345, 45454, 29, …].
It is hard to demonstrate how you could use a text generator and show this technique, so I used an image generator. Using nothing more than python, and the standard libraries, I asked the AI to help me swap tokens out before it hit the model diffusers. What if?
Design of experiment: full factorial rotation of the keywords in the prompt from one prompt to another to see how each combination plays out.
This is the result, applied to images:
Value - It helped me understand some of the internals of how a prompt gets parsed, tokenized and moved through the model. The more you know.
Beyond this, there is little value because you could just change the prompt itself and show the same thing. Besides, it was fun to see a cat and dog go green.
Experiment: What if we were to create two prompts, run them, and map every layer, every tensor, every value between them, and then steer the model from one prompt to the other internally?
In my experimentation with the SRT adapter in my primary article of “The AI Ate My Cookie” I explored semantic content of prompts. How do you keep a prompt in a given domain so it does not drift into domains? I used Computer Science and Food word games to isolate that the word nibble will lead some models to Computer Science, so you have to deal with that.
That leads to the “what if?” of steering tensors themselves. Inside the model, at generation time, can you steer the output towards another prompts area in the model? In the cookie test, it would mean steering the model to stay within the realm of food or to move within the realm of CS.
As I was working through the experiment with token id swapping pre-generation, I become interested in seeing what other kinds of manipulation you can use to steer the model. Token swapping is harmless, and relatively simple.
This is far more complicated than I need to get into, and I was working with an image generator from Microsoft (Mage-Flow; specifically for its safety position … which has now been removed for unknown reasons from HuggingFace).
It took quite a long time to figure it out, but I was able to find tensors and apply some basic math equations from the base prompt and steer the output to the second prompt. This results in a semantic re-interpretation of the prompt at generation.
This is NOT morphing, it is semantic. I am pushing the model from one prompt to another prompt, and as it processes the tensor variations, it shows a new semantic rendering of the image.
I take the output tensors from the base model, and the second model after they generate, and then push the model in small alpha increments towards the other one, without touching the input prompt.
This is a demonstration of steering, easiest to see with images. It moves from a fantasy scene with a dragon to a future scene with a robot. Notice how at about .500 alpha, the world is clearly sitting in a more futuristic environment. This is the point.
Value - This particular exercise proved valuable in all kinds of way. Knowing how to steer a prompt at generation has high value, from safety to simply making sure your AI agents are always working in your domain. If you trend your AI system and internals, you can identify when a prompt is drifting and steer it back to the right domain. This was quite a surprise to me.
Experiment: What if we used a zero-shot time-series forecasting model combined with an image classification model to identify cut scenes in a film?
After the cookie experiments, and messing with the image models, the next natural step was vision models. Being a student of history, and having an affinity for the old films and media, a new question formed: What if I use AI to detect film scenes? Could AI models be used to map out scenes from movies or tv shows?
I cannot count how many times I have seen certain episodes of shows or certain movies. People think it is crazy, and perhaps it is, but when you watch film, there is a technical side to it. Each frame has a position, and those frames form a sequence. Mapping that out is, for lack of a better way to say it, tedious. You watch the scenes over and over and watch for continuity, for edits, for music, for sfx. You can map so many things.
Would it not be nice to have a device that could do that for us? Could the AI do that? What if?
So I wired up a time-series forecaster to an image classifier and fed in the film footage one frame at a time. The goal was to have the AI tell me where the cuts are. Using the time-series data, I was able to have it alert when there was a substantial difference in the content of the frames. This was then fed to the vision models and told to explain the frames before and after that, and it became a sliding window.
After many attempts, I was able to narrow the parameters well enough that it was able to within a few frames identify clear cuts in an old silent movie. Old film is grainy, low resolution, and black and white. Modern AI VLMs are not really trained on historical footage, they are trained on modern media. This is why I was surprised it did so well.
Charlie Chaplin - The Adventurer 1917 (public domain)
Value - Yes, this one has substantial value. As a media nerd, and someone interested in the historical preservation of the arts, the ability to take old films and auto-archive them = awesome! No more hand written timing sheets for me! This one was personally rewarding for many reasons.
There you have it. A summary of 3 experiments that were ran recently.
Remember, experimentation is the genesis of new ideas. It is how humans have always done new things. Some work, some do not. Will I keep all of these? Definitely the film scene mapper and the steering technique, but the token swapping, probably not.
AI is an accelerant and it is important to make sure you understand that these are all domains upon which I already have deep knowledge. When using AI in your own areas, you can spot the problems in what it gives you. When you are using AI outside your area, the risks go up, and they go up fast.
Stay careful. Stay safe.