Building AI-powered tools that aid in visualisation and empower design exploration

This started as a side project born out of curiosity, but I didn't know where it was going to take me, or if it would even work. To break down this ambitious idea of mine, I wanted to get a proof of concept working first.

First, Cursor needed to understand my project before it could help execute it. Designers should be able to screenshot any component they liked from another app, drop it into a website, and have it converted into an image of the same component style using our new design system's colours, font styles, and spacing.

I chose to build this as a standalone website rather than a more AI-native approach which would have been much simpler to setup such as a Cursor skill. AI adoption was still early at the company, and I wanted anyone to be able to use it, I believed this tool had the potential to be especially useful to PMs, who have ideas about their products but have a harder time visualising it than designers.

For this setup to work, Cursor needed a source of truth, the design system variables. So I asked it to generate a markdown file describing them, pulled directly from our Figma design system file via Figma's MCP. Then I tested whether Cursor actually understood what I wanted: I dropped in a component image and asked it to output an HTML preview styled with our design system, viewed it on localhost, and tested it across images of different components to check accuracy.

From there, I built the rough outline of the site using Github pages. Drop an image in, hit convert, see the preview. The next step was getting this tool to run on the live site. I needed the website to securely call our company's Claude, hosted on Bedrock, without exposing credentials to anyone visiting the page. A personal access token worked at first, but new company security measures broke that setup, so I had to get the site whitelisted with the security team before it could call the model again.

Once that was working, I added extra features that helped the AI build better context, such as letting users describe the component before converting, and a feedback loop for correcting output.

At this point, we had Screenshot → Preview. I wanted to stretch that further and get the output directly in Figma. If designers could actually use this in their own files, the barrier to exploration would drop even further. I discussed with Cursor what Figma's MCP could support, and landed on the most seamless path. Generating code from the website that could be pasted into a custom Figma plugin. I iterated on the plugin until it produced accurate output: frames using auto layout, with spacing, typography, and colour all pulled from our actual design system variables.