A real-world benchmark for AI code review

Posted by benocodes 2 days ago

A real-world benchmark for AI code review(www.qodo.ai)

52 points | 26 commentspage 2

mohsen1 2 days ago|

> Qodo takes a different approach by starting with real, merged PRs

Merged PRs being considered good code?

esafak 2 days ago|

What do you suggest they use for ground truth?

mohsen1 2 days ago||

I thought about this quite a bit. There are some nuggets in the open source code:

- vX.X.1 releases. when software was considered perfect but author had to write a fast follow up fix. very real bugs with real fixes

- Reverts. I'm sure anyone doing AI code review pays attention to this already. This is a sign of bad changes, but as important.

- PRs that delete a lot of code. A good change is often deleting code and making things simpler

esafak 1 day ago||

For the first, your signal would be weak, for those events are rare. I don't think deleting and reverting is a signal of quality. Rather, it demonstrates bad changes, as you said. This does not tell the model what good code is, just what it is not.

aetherspawn 2 days ago|

Your pricing page has a bug on it, the annual price is higher than the monthly price.

zamadatix 2 days ago|

I'm seeing $30/m at annual and $38/m at monthly? (maybe already fixed, hard to tell)