Like a personal Google? How do you bypass all the captcha, ip bans, cloudflare turnstile antibot stuff etc?
sandeepkd 3 hours ago||
Thats the fun part, the user just went with happy path. Javascript, captchas, cloudflare protected content did not made to the catalogue. This sort of use case exists in LLM training data a lot which makes it easier. The data gathered by the user is not really practically useful cause there are way too many gotchas when it comes to web scraping and building a catalogue (source: I have done scraping for a particular domain data and had to do at least 10+ iterations to get it >90 right)
voidUpdate 3 hours ago|||
They don't: "skips the model entirely if the page is empty, parked, or a bot-challenge wall"
dreamforever 42 minutes ago||
despite this limitation, there is still some good stuff out there, and with the priority steering, you can focus compute on what you actually want, fast and cheap.
dewey 4 hours ago||
I think Kagi Small Web filter would give you very similar results.
dreamforever 41 minutes ago|
I'll check them out!
dreamforever 5 hours ago||
Check out my latest project! You can fork it, tweak the policy manually or with AI, run the system and watch the data come in! It's engineered to keep a low data footprint, so 500k domains fits into 1GB on disk. If you have local models it's free! You just might not get the best throughput depending on your GPU. My production data is not exposed anywhere yet, and I may never expose it. The point is for you to fork and make your own policy, and thus your own personal search engine! The article covers basic analysis on my data, so it's worth a read if you're interested! A deeper analysis may arrive with V2 if I ever do it
elorant 2 hours ago||
Domains are way more than just 40M though.
BaudouinVH 2 hours ago|
From what I understand the aim was not to collect all the domains on the web but focus on personal website, etc. and avoid corporate web sites.