Posted by futurisold 4 days ago
I didn't get very far because I had difficulty piping it all together, but with something like this I might give it another go. Cool stuff.
---
class Merge(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being merged.")
relations: list[SubClassRelation] = Field(
description="A list of superclass-subclass relations chosen from the existing two clusters in such a way that they merge."
)
@field_validator("indexes")
@classmethod
def is_binary(cls, v):
if len(v) != 2:
raise ValueError(
f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
)
return v
class Bridge(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being bridged.")
relations: list[SubClassRelation] = Field(
description="A list of new superclass-subclass relations used to bridge the two clusters from the ontology."
)
@field_validator("indexes")
@classmethod
def is_binary(cls, v):
if len(v) != 2:
raise ValueError(
f"Binary op error: Invalid number of clusters: {len(v)}. The merge operation requires exactly two clusters."
)
return v
class Prune(LLMDataModel):
indexes: list[int] = Field(description="The indices of the clusters that are being pruned.")
classes: list[str] = Field(description="A list of classes that are being pruned from the ontology.")
@field_validator("indexes")
@classmethod
def is_unary(cls, v):
if len(v) > 1:
raise ValueError(
f"Unary op error: Invalid number of clusters: {len(v)}. The prune operation requires exactly one cluster."
)
return v
class Operation(LLMDataModel):
type: Merge | Bridge | Prune = Field(description="The type of operation to perform.")
---I hope you keep at this, you may be in the right place at the right time.
It's getting to the point where some of the LLMs are immediately just giving me answers in Python, which is a strong indication of what the future will look like with Agents.
I built a version of this a few years ago as a LISP
Two years ago, we built a benchmark to evaluate multistep reasoning, tool use, and logical capabilities in language models. It includes a quality measure to assess performance and is built on a plugin system we developed for SymbolicAI.
- Benchmark & Plugin System: https://github.com/ExtensityAI/benchmark
- Example Eval: https://github.com/ExtensityAI/benchmark/blob/main/src/evals...
We've also implemented some interesting concepts in our framework: - C#-style Extension Methods in Python: Using GlobalSymbolPrimitive to extend functionalities.
- https://github.com/ExtensityAI/benchmark/blob/main/src/func.py#L146
- Symbolic <> Sub-symbolic Conversion: And using this for quality metrics, like a reward signal from the path integral of multistep generations.
- https://github.com/ExtensityAI/benchmark/blob/main/src/func....For fun, we integrated LLM-based tools into a customizable shell. Check out the Rick & Morty-styled rickshell:
- RickShell: https://github.com/ExtensityAI/rickshell
We were also among the first to generate a full research paper from a single prompt and continue to push the boundaries of AI-generated research:
- End-to-End Paper Generation (Examples): https://drive.google.com/drive/folders/1vUg2Y7TgZRRiaPzC83pQ...
- Recent AI Research Generation:
- Three-Body Problem: https://github.com/ExtensityAI/three-body_problem
- Primality Test: https://github.com/ExtensityAI/primality_test
- Twitter/X Post: https://x.com/DinuMariusC/status/1915521724092743997
Finally, for those interested in building similar services, we've had an open-source, MCP-like API endpoint service available for over a year:- SymbolicAI API: https://github.com/ExtensityAI/symbolicai/blob/main/symai/en...
// read files
const file = await workspace.readText("data.txt");
// include the file
content in the prompt in a context-friendly way def("DATA", file);
// the task
$`Analyze DATA and extract data in JSON in data.json.`;
https://reference.wolfram.com/language/guide/FreeFormAndExte...
It can (in theory) do very similar things, where natural-language input is a first class citizen of the language and can operate on other objects. The whole thing came out almost a decade before LLMs, I'm surprised that they haven't revamped it to make it really shine.
No worries! I can't find it right now, but Wolfram had a stream (or short?) where he introduced "Function". We liked it so much we implemented it after one day. Usage: https://github.com/ExtensityAI/symbolicai/blob/main/tests/en...