Using LLMs to enhance our testing practices

Posted by johnjwang 10/24/2024

Using LLMs to enhance our testing practices(www.assembled.com)

186 points | 77 commentspage 2

simonw 10/26/2024||

If you add "white-space: pre-wrap" to the elements containing those prompt examples you'll avoid the horizontal scrollbar (which I'm getting even on desktop) and make them easier to read.

johnjwang 10/26/2024|

Thanks for the suggestion -- I'll take a look into adding this!

apwell23 10/26/2024||

i would love to used to use it change code in ways that compiles and see if test fails. Coverage metric sometimes doesn't really tell you if some piece of code is covered or not.

sesm 10/26/2024||

Coverage metric can tell if lines of code were executed, but they can't tell if execution result was checked.

taberiand 10/26/2024||

I believe that's called mutation testing. Using an LLM to perform the mutation sounds like a great idea

rgmerk 10/27/2024||

LLMs are not suitable for mutation testing. Mutation testing needs to be fast to be useful (because you need to generate and test a lot of mutated versions); an LLM-based mutator would be extremely slow as well as error-prone.

taberiand 10/27/2024||

Set aside LLMs, why does mutation testing need to be fast? It would be fine to have mutation tests run slowly, out-of-band of the main CI pipeline. They aren't mission critical, they're smoke tests for your unit tests.

Also you only need to generate a set of mutations for any particular unit once, and then again when the test code or the code under test changes.

rgmerk 10/28/2024||

Because mutation testing (should) generate a lot of mutants, which you then need to run your unit tests against.

taberiand 10/28/2024||

Yes but my point is it doesn't have to do that fast, as in unit test speeds.

dfilppi 10/27/2024||

[dead]