Posted by theanonymousone 16 hours ago
create a plan with SOTA, execute with this.
The specific agent is focused on getting precise and on point answers about a codebase.
The starting point was nowhere near. E.g. asked why was X implemented in a certain way it would give bogus answers when the real answer was that there was no reason at all.
The benchmark included more than 50 questions or different difficulty.
But when the agent was improved in its prompt and rooting it was impossible to have it perform worse than closed source sota.
Just to say that the quality of the harness is as important as agents intelligence.
The ban on these open models is coming within weeks, if not days. As usual, the excuse will be "national security".
For example: no government contract to any company who uses even one vendor in it's entire chain of dependencies, who uses such open models.
They can extend this further by laying more conditions, such as: any company dealing in this-this field can only use models "officially" approved as "safe". Rest you can guess how easy it would be to get that "safe" rating for such open models.
I'm not sure the outcome would be beneficial for the US as a whole here. But perhaps that is not their priority.
- the US strictly regulate cryptography https://en.wikipedia.org/wiki/Export_of_cryptography_from_th...
- some prime numbers are considered illegal https://en.wikipedia.org/wiki/Illegal_number#Illegal_primes
I don’t think banning open weight is that far fetch compared to those
I claim the CCP will wise up within 2 years, possibly much much sooner, and ban their own companies from open sourcing to prevent the Americans from acquiring the capabilities.
Despite all the nonsense claims of China distilling US models, the reality is that the Americans absolutely do distill these free Chinese models, and distillation when full logprobs are available (i.e. you have access to the weights of the model) is an order of magnitude better than when you don't.
Yes, Chinese open weight models in the short term harm US closed source model providers bottom line. In the slightly longer term, "showing your hand" and publishing both the architecture innovations and the models weights will be too dangerous for the CCP to allow. This is triply true if they can release a model that beats the Americans on most benchmarks.
I've already warned investors that this is probably the closest open weight models will ever get to closed access.
https://www.businessinsider.com/xi-jinping-open-source-ai-us...
People on HN downvote objectively correct information because they don't like it 24/7. There's a reason the creator of Zig left and gave the computer version of a middle finger on the way out to HN!
Since when have hackernews started to become toxic like stackoverflow used to be?
commenting about voting is also something the HN guidelines warns against:
> Please don't comment about the voting on comments. It never does any good, and it makes boring reading.
Or a benchmark to benchmark benchmarks?
benchmark website benchmark is indeed a benchmark that benchmarks websites with benchmarks (but it can be shown outside websites as well, it's not picky)
They can't compete, they have bills. By the end of the year, if they can't react, it's game over.
Maybe US clients could be a little patriotic here, but money is money. They won't give them free money forever.