Iterative multimodal prompting leverages different input types—such as data, images, and text—in a dynamic back-and-forth process to refine AI-generated outputs. This approach enables users to guide AI assistants through multiple revisions to better align results with specific goals and context.
Key Insights
- Used a data table as an initial input to generate an infographic visualizing satisfaction rates by transportation mode, demonstrating the first step in multimodal prompting.
- Analyzed the AI-generated image using a text-based prompt to assess whether the intended message—year-over-year improvement—was clearly communicated.
- Iteratively refined the output through follow-up prompts, ultimately producing a more accurate infographic and generating a social media caption tailored to the content.
This lesson is a preview from our AI Prompt Engineering for the Government Workforce Course. Enroll in a course for detailed lessons, live instructor support, and project-based training.
To kind of bring this notion of multimodal prompting together, let's look at that most complex use case, the iterative multimodal prompting, and we'll see it at play, and hopefully that'll help synthesize for you the power of using multiple modalities. So I have a demo I'm going to do here. I'm sharing my screen.
I'm in ChatGPT. Since I'm not going to ask about engineering anymore, I'm switching contexts completely, so I'm going to start a brand new chat. And I am going to start with some data.
So I have some data here that I am going to copy and drop into ChatGPT, our AI assistant. There it goes. So this was a table that showed some different modes of transportation and the satisfaction rating for that mode of transportation, and its exponential increase or its increase over the previous year.
So I just fed it some data, so that's one modality, right? Or that's one input type. And I am now going to ask it to take that data and create an infographic that visualizes these satisfaction rates by mode of transport for a public-facing report. And let's see what it does.
Okay. So here, through the magic of video editing, we now have our image. So we started with some data, and then we asked our AI assistant to generate an image from that data.
But watch what we do here. I am going to, this is the iterative multimodal flow here. I am going to ask it, now that we have an image, I'm going to use some text to ask it something else.
I'm going to say no, I want to make sure that my meaning is being conveyed, right? Before I release this publicly, does it get it? So I'm going to ask my assistant here, now, describe this infographic as if you were explaining it to somebody who can't see it at all. And let's see if that matches my intended tone. Okay.
It says, this infographic is titled satisfaction rates by mode of transport. Yes, that's correct. It uses clean modern flat design with blue and green horizontal bars, each representing a different mode of transportation.
Five modes are listed: bus, subway, tram, bicycle, and car, and that's all correct. Each mode is paired with a circular graphic, similar to a donut chart that displays two percentages in bold, dark green numbers. The circles are half-filled, giving a quick visual sense of satisfaction.
So the bus has 45% in one circle and 55 in another, 55 and 60 for the subway, and so on and so on and so on. And it says that the colors alternate and things. So that's good.
That's good to get that description from our AI assistant, but I can tell from that output that my initial intent is not coming through in what it generated. Because again, the point from this data up here is what I really wanted to highlight, which is that it's getting better, you know, that it's improved. And specifically from last year, it's not like it's improved all time or something like that.
Or yeah, well, of course, in 10 years, it was bound to get a little bit better, you know, by chance maybe. I want to show the exponential change from just last year. So that's good to know.
I'm using the AI sort of against and with itself to help me get a better result. So I'm going to ask it something else then. So I'm going to copy a canned response here, but it's also interesting to see because, you know, again, the AI is different every time.
It doesn't have the same deficiencies when I asked it to do this exercise before. It doesn't have the same deficiencies as the last time I used it. So hopefully this demonstration is showing you, again, how it can kind of work together to iterate to get a better result.
So I could now, if we cared to, say, okay, go back to that infographic and highlight that it's a change in just a year, you know, make that more prominent. And then I would, you know, wrestle with the AI some more to eventually get that result. But let's say, just for the purposes of keeping the example fairly short, that I was, okay, I'm happy with it.
I used different modes of input, after maybe several versions, to finally get the infographic that I was looking for. And then I can, just to put a little icing on the cake, I can now generate a short caption summarizing this infographic for our social media page. And I'll use this as the last step because presumably it will generate this quite quickly.
So here's its response. I asked it to generate a short caption summarizing the infographic for the city's social media page. And I didn't ask it to do this, but it actually gave me a couple of options, which is really nice.
One, two, three here. And let's say I'm not even reading them, but since this one uses emojis, and it's going on social media, you know, maybe this is the perfect thing. So there's a demonstration of how you can use these different inputs to improve upon one another.
And it gets really, really powerful from there.
