I don't see LOC as that different from number of hours in the office. They'd always say pre-pandemic "If they're not in the office, how will I know they're working?" Simple, use the output metrics that you use to evaluate all of your workers to see what they contribute to the business.

More that LoC is a simple metric that has always been a problem.

Non-Functional requirements is a vestigial term from ‘function point analysis’ which is from the late 70s, and which also ended up being a proxy for LoC.

The entire industry is so focused on measuring now, and incentives are so skewed to short term that lagging indicators like maintainability are a non starter in many organizations that it will be challenging to fix this time.

Large enterprises class systems are notoriously difficult to work Ai. It’s the context window limitation. Assuming 10 tokens per LoC. The best models today cannot wrangle 100k distributed LOC across multiple repos. It’s great for building new and maintaining smallish codebases. All this code being written is fantastic, but maintaining them efficiently over the code lifecycle is tricky.

I think a better metric these days is what percentage of code is not reviewed / understood by humans. That is the real bottleneck. Until we can stop looking at the code, AI barely matters - you are just trading quality for quantity.

Thats why it is so amazing for speed runs and prototypes. Here it is legitimately > 10X faster.

This is already changing again now that CEOs have wised up to the fact that they're paying for code by the line but these lines don't translate to profit.

It seems to naturally follow that a company that sells lines of code would want to measure success in lines of code.

If developers burn through thousands in AI tokens a day, does it really matter, and is it a good spend? Are the outputs actually checked for sanity, fitness, qa/qc, security etc. How much rework is coming out because of lack of validation, or too much automation in the soup.

The more I read, the more I feel that 1 dev, 1 ai agent with the dev as a gatekeeper is probably the most appropriate workflow. Where you now treat the single dev + ai as a team in terms of planning and cost analysis and you get about 1.2-1.3x the throughput compared to a traditional team of 3-5 devs with partial PM and partial QA where the Dev now needs to take on those roles too.

The output should include more/better testing, examples, demos etc... since the bus factor is now 1, but AI is expected to be able to do the heavy lift.

Not enough people read The Goal.

Ugh. Just imagine the following on a normal curve:

Pre-AI: The goal is to make more money.

With-AI: The goal is to ship more code.

Post-AI: The goal is to make more money.

Can't wait to see how we get there...

LoC by itself is useless and so is AI LoC, it doesn't really show anything by itself.

But if you pair AI LoC in a range and also task completed in the same range and then compare that with historical data over a similar range without AI, then you have something tangible.

You also need to look at defect reports to understand the full picture of is AI being helpful.

So, we do need to measure AI LoC and AI PR counts, but we also need to make sure we are using other metrics to help paint the full picture.

We're still in the FA phase of FAFO when it comes to LLM code generation, aren't we?