I am working on a similar thing for jsoncanvas (.canvas files in Obsidian). My takeaway is that agents are heavily trained on specific formats (e.g. svg), and are conversely pretty bad at niche formats. My strategy was to create a native renderer [1] so that at least they can rapidly iterate "visually" by reading svg and png renders instead of trying to one-shot.

I have yet to see an AI generated visual diagram that doesn't feel like slop though... I'd be very interested in examples of actually-good diagrams workflows if anyone has seen them.

[0] https://jsoncanvas.org/

[1] https://github.com/dtkav/canvas-native

tldraw has a lot of programmatic access I have not compared it to excalidraw but there is still no equivalence between the model capabilities to build amazingly detailed infographics and diagraming tool use which is a gap I hope will narrow