How to scrape Hacker News comments

Reading a thread by hand

For a single thread you can just read it on the site and copy what matters. HN's threading is legible in the browser, so this works fine when you only care about one discussion. It stops scaling the moment you want to compare what people say across every thread about your category, or capture the deep replies that get collapsed on a long thread.

The official API route: easy access, no analysis

This is the rare case where the API route is genuinely the developer-friendly one. The Firebase API exposes item and comment endpoints with no auth and no key; you fetch an item, get its list of child ids, and recurse to rebuild the comment tree. The Algolia HN Search API lets you find threads by keyword so you are not guessing at URLs. Both are free.

But access is only half the job. You still have to reconstruct the nesting from ids, dedupe, strip the meta-discussion, and then read the whole thing to find the pattern. Five hundred comments of 'I'd just self-host it' and 'the pricing page buries the per-seat cost' is data, not an angle. The API hands you the raw material and stops there.

The paste-a-URL route

Adlicio reads the public comments under a thread through your own browser session and skips the recursion entirely. Paste a thread URL and it collects the comments, replies included, in about 60 seconds, no id-walking and no scripts.

Then the comments come back clustered into ranked angles: the objections technical buyers raise first, the tools they say they would reach for instead, and the trade-offs that decide whether they take you seriously, each backed by verbatim quotes. It turns 500 comments into the handful of angles worth testing.