Posted by mfkhalil 4/3/2025
Search could be better? Yes, yes it could.
I search for words, can even indicate I want search results with a keyword included and it will be ignored. And then I have to sift between what is the search result, and what is an ad.
And if I get another quora answer....
But, this post? it was a waste. We do some hand wavy stuff, come try us.
MatterRank uses LLMs to rank pages based on criteria you provide it with, not SEO tricks. It’s not meant to replace Google, but helps when you're looking for something specific and don't want to wade through tons of results that you don't care about. Still early, but useful for deeper searches.
I cant even imagine why they got rid of that, unless hundreds of thousands of people started pasting 1000-character search terms removing all the known ads currently flying around
That seems extremely unlikely as the reason.
It's far more likely that some executives looked at the numbers and decided that removing search operators would make people more likely to click on ads, while leaving them in would make people click on the actual results that they were searching for.
make search worse = more searches are performed = more ads shown
It's anti-consumer but every company is like this now.
I am unsure if it is possible to run a "free" web search without having a benevolent benefactor paying for the scraping and maintenance and staffing. Furthermore, someone has to play mouse and mousetrap with the "gaming" of whatever "algo" one chooses to use to rank results. Maybe a list is the wrong way to display search results. maybe a contemporary snapshot of the page with the search text highlighted might work better. It might even convince a lot of sites to clean up their landing pages and their blog/article formats.
I know how to stand up and start a web search engine, and probably could implement a decent chunk of functionality myself. it'd be slow and fall down if 100,000 people hit it at once, but nonetheless, the hard part isn't getting one running and starting the scraping. The hard part is results and funding.
I envisioned, last night, in a fever dream: maybe some metadata that the crawler and the sites share, to encourage Value for Value. If a site is willing to be scraped, but would like some nominal bandwidth costs recouped, or perhaps some sort of data agreement that is mutually - mutually - beneficial; or a site like NYT chipping in to the search hosting costs if the search company has really good results, like better than NYT could implement, then there could be some value for value there, too.
Search engines provide a valuable service for humanity, as a general concept. Search engines as they exist now provide a valuable service for their shareholders. remove the shareholders, make the service valuable for humans, and the human stakeholders in the search company (employees, vendors, etc) might not be so greedy or "legally obligated to make numbers go up".
Encarta and Britannica existed. Wikipedia exists - as well as the forks and archives.
Does it? I understand there are issues with spam in search, but assuming we don't know what we want is not at all the conclusion I draw from using search engines.
For the former, I'm intrigued but unconvinced that it's what I actually want in a search engine.
For the latter, I imagine that's something that this search engine will need to contend with, although it could "just" be an LLM compute trade-off, where if you give enough results to an LLM to analyse you'll eventually find the good stuff. That said, SEO is going to rapidly become LLMEO and ruin the day again.
https://github.com/rumca-js/Internet-Places-Database
I must admit, that this is a difficult task. There are many domains for "hotels", "casinos", so I have to protect myself, just as google agains spam.
Remove all of this, just let me directly use your app, I want to search and create engines on the fly.
I don't need to save them for future uses, if I am not going to use your app even once.
If you want this to take off, it needs to just work, no extra steps unless I want to.
The results for me were fairly high quality and moderately relevant but I think they could be improved as well.
You get pretty far by just blocking low quality blogspam and Medium, which would be a lot faster and could even be done on the frontend with a chrome plugin.
As for the results, it's tough because we've made the deliberate decision to have no control over the reranking. What that means is that if your criteria is "written by a woman", for instance, then any result that meets that will be ranked equally at the top. In all engines I've built for myself, I have a relevance criteria that's weighted relative to how much I care that the result is exactly what I'm looking for. It's probably important to make that clearer to the end user.
I'm hoping that as LLMs become more mainstream more functionality is built into tech that doesn't treat consumers as idiots. This is one stab at it, but there's so many other opportunities imo.
What I prefer is interfaces that are more systematic and based on comprehensible principles. Like, for search (as someone mentioned in another comment), I want to be able to search for pages (or records, or whatever) that contain the text I searched for. I don't want an interface that tries to understand what I mean, I just want it to use the data I give it in a way that's deterministic enough that I can figure out how to make it do what I want.
In my opinion, the best applications of LLM UX will have full clarity for the end user (something we're trying to do with MatterRank). The non-determinism should be something the user can control to get better results, not something the engineer has prompted that takes control away from the user.
Now, if the use case you're looking for is "give me results with x text", then yes I agree with you that LLMs are just getting in the way. But that's not always the case.
It doesn't even explain why it's better than Perplexity.
anyway, my test was to search for FOSS software, explicitly asking for "not big tech" and no ads. the contents of the results were fine, if repetitive - but i was a bit sad to see a lot of youtube and reddit in the results. does the " algorithm" not look at the actual domains?