How to scrape a Discourse forum
Here is the trick most people miss: many public Discourse forums will hand you JSON if you just append .json to a topic URL. Discourse is one platform running under thousands of communities, from official product forums to open-source projects and hobby hubs, so the same endpoint tends to work across all of them. For a developer, the raw posts are one request away.
The friction is everything around that endpoint. Each forum configures visibility, rate limits, and pagination its own way, long topics stream posts in batches you have to page through, and the brand-community forums where the customer language is richest are exactly the ones worth reading in bulk. Here are the routes, and the fastest path to usable language.
★ 4.5 on the Chrome Web Store · no card needed
Reading a topic by hand
You can open a topic and copy the posts that matter. On a short thread this is quick and requires nothing technical. It falls apart on the long, most-replied topics, which is a shame, because those are the ones that hold the switching stories and feature-gap complaints you actually want. Copying a 200-post topic by hand is not a plan.
The .json endpoint route: works, but quirky per forum
Append .json to a topic URL and most public Discourse forums return the posts as structured JSON, which makes this a genuinely accessible developer route. In practice you also page through it: a long topic exposes its post ids in a stream and serves them in chunks, so a full pull means following that pagination rather than one call.
The catch is that every forum is configured differently. Some require a login for certain categories, some throttle requests, some tweak pagination or rate limits, and the endpoint behavior you tuned for one community may not hold on the next. You script it once, then you maintain a slightly different version per forum. Access is mostly solved; the per-forum babysitting and the analysis are not.
The paste-a-URL route
Adlicio reads the public posts under a topic through your own browser session, the same content any visitor sees, with no login, no API key, and no per-forum scripting. Paste a public topic URL and it collects the posts in about 60 seconds.
Then the posts come back clustered into ranked angles: the recurring feature gaps, the switched-from stories, and the complaints that make engaged users leave, each backed by verbatim quotes. Brand-community forums are gold for this, because the most invested users post long, specific threads in the exact words your buyers use.
The 60-second version
- 01Find the forum where your audience already argues
A competitor's official Discourse forum, an open-source project's community, or a niche hobby hub. The most-replied topics carry the desires and dealbreakers that define the category.
- 02Paste the topic URL into Adlicio
Any public Discourse topic. The scrape reads the posts under it and finishes in about 60 seconds, no .json wrangling and no pagination to handle.
- 03Read the clusters, write in their words
Work from the ranked angles: the feature gap power users keep raising becomes your hook, and the verbatim post gives you copy that sounds native to the community.
Questions people also ask
Can you really get JSON by adding .json to a Discourse URL?+
On most public Discourse forums, yes, appending .json to a topic URL returns the posts as structured JSON. It is a real developer route, but behavior varies by forum, and long topics still have to be paged through.
Why are brand-community forums so good for ad research?+
The most invested users gather there and post long, specific threads about what is missing, what they switched from, and what finally worked. That is customer language in the exact words your buyers use, and few competitors mine it.
How do I scrape a long Discourse topic with many pages?+
A big topic serves its posts in chunks rather than all at once, so a manual or scripted pull means following that pagination. Adlicio reads the posts under the topic for you, so you paste the URL instead of paging through it.
Do all Discourse forums allow this?+
Not uniformly. Some forums gate certain categories behind a login or throttle requests, so the .json endpoint is not guaranteed everywhere. Adlicio reads what is publicly visible to a normal visitor and takes no action on the forum.
More guides
Discourse scraperAll scrapersSee pricing
Run this play on your own Discourse page.
Paste one public URL. Adlicio returns the angle, hook, and proof to test next.