(1) build a bunch of automation scripts that we can reliably run as part of our product. This is closest to my day-to-day, you're (mis?)using selenium or playwright to make a known set of "click here, fill that, click there" scripts, and then expose some interface that calls them. At a company usually this will be a fastAPI / microservice, and your set of automation scripts is part of what makes the product possible. In my case, I'm currently working on registering my Tkinter/Pyinstaller app with windows so people can run my .exe and click the "do the script" button. There's also a slightly different approach to the selenium/playwright jam that runs curl requests which mimic the network requests of performing some action. You'll be able to see what I mean by clicking into your browser devtools "network" tab and clicking around the web. Imagine having a library of known API calls, you save a users cookie, and run script to achieve result.

(2) build a browser agent that we can use to run all of our (1) scripts. This is stuff that I iterate on as needed, and in my experience is best done by having a bunch of "helper functions" that spawn/kill/season different selenium webdrivers. It will depend on whether your (1) scripts are dealing with JS-heavy websites, bot detection, or just constantly changing UI's, but the deep rabbit holes here end up in places where you're building your own browser agent on top of webkit/chromium, implementing some kind of captcha solver, or trying to automagically discover what buttons exist by fuzzing the API or DOM.

(3) use a (2) to get us every PDF we need for our upcoming RAG chatbot. This is something I do by executing on (1), and I only bother to note the difference because it's a great example of the kind of actualy end product goal that all of this leads to.

Academically, the problems happening in web automation broadly fall into discovery and reproduceability. Discovery, meaning API fuzzing (how do we get that library of known API calls? / the equivalent buttons in a DOM?); Reproduceability, meaning running-without-errors. (How do we wait for the target's server to be ready to send the next command? How do we avoid getting blocked? How do we detect/recover when the target updates their website?) The most interesting opportunity IMO is inserting an LLM to build a self-healing scraper, and the edges of your tes/prod environments will be defined by your product's tolerance for wrong/nondeterministic behavior. I've got a great blogpost draft about a "railroad model of software development", where an LLM is a hammer nondeterministically pounding in railway ties, and an end product is a deterministic piece of code that can have trains run over it all day long. (effectively, LLM as test-environment devtool thesis, I don't think I'm saying anything that hasn't been said before.)

Practically, the problems that are facing me as an engineer are in packaging these tools for sale/distribution. My current state-of-the-art is to wrap up a .exe with Pyinstaller, build a GUI with Tkinter, and register with Windows so I'm not showing scary "This program is made of evil" messages when people try to run it. From there the plan is to give away free trials and after a month of people clicking their magic buttons they all disable and demand you purchase a license to re-enable (like if winRar was evil, but sorry yall I gotta eat). I'm also trying to sell building these tools as a service but that's very word of mouth, I haven't found a viable web/storefront model for that yet.

IM-Practically, most companies with a browser automation component are struggling with the same HR/Onboarding issues everyone else is. In the past ~2 months I've Interviewed at ~4 serious companies that profit from their browser automation, and every process has been unique: 1) firecrawl.dev; a black mirror hourlong AI-chatbot-zoom-interview, followed by a human call scheduled 3 weeks out, only to then be told that really they want someone fully specialized on breaking captchas, with a vaugely condescending suggestion that I'm "customer facing" and no followup when I lean into that 2) atomic.financial; actual-human-zoom-calls with engineers who I get along with great only to have no idea why I'm getting a rejection email the next week 3) sheer.health; a very contentious first call that demands I name a salary, followed by a trivially easy take-home test, followed by my 3rd round, 1/2 hour call with the CEO being cancelled the morning of because they filled the role. 4) Mozilla; where a principal/staff engr cold DM'd me, to schedule a call with an HR rep that told me they're paying 350k base 420k total and another staff eng who's leaving to start a startup, only to then tell me I'm not senior enough but could maybe come on as a contractor, only to then tell me they're using internal resources for the contractors.

Overall, I think the best opportunity in the space is going to look something like https://ui.vision/, which is an open-source tool!