Why “Crawler” is being discussed
The most recent Hacker News stories and comments contributing to this topic's mentions.
Show HN: I built a crawler that finds personal websites by a /.well-known/josh
by TheWoollyBully · Oct 31, 2026 · 3 points
All - I’ve been wanting to use Jev in a project, and I recently saw Firecrawl use it in an interesting way. They use their crawler to pull company data and then use Jev to class…
by bilater · Oct 31, 2026
A browser does not send 10k requests per minute to open a webpage. Even in the most and modern bloated webpages. And even if it did it wouldn't be to a single website. A user op…
by AlienRobot · Oct 31, 2026
I disagree, the moat will be of those holding rights for valuable problem domain specific training data. Examples I can think of: Adobe sits on a treasure trove of: - Copyrighte…
by DoingIsLearning · Oct 30, 2026
- "Vibe-coded": A shopping list that sort by aisle in the order I go through them on my local store. It is _wildly_ successful. I got my wife to use it. - When reading technical…
by jomi-se2 · Oct 30, 2026
> That costs the crawler money They're for-profit operations. They have money, and it doesn't cost that much. Proof of work is significantly more expensive (but then you put del…
by AnthonyMouse · Oct 30, 2026
Interest
Proportion of Hacker News items mentioning "crawler" over time.
Mentions
Total number of Hacker News items mentioning "crawler" over time.
Crawler may refer to: Read more on Wikipedia