Duplicate Articles in Your RSS Reader and How to Fix It
One Tuesday in September I opened my reader to 140 unread items and found that 23 of them were the same nine articles.

One Tuesday in September I opened my reader to 140 unread items and found that 23 of them were the same nine articles. A piece about shipping delays appeared four times: once from the publication itself, once from a syndication partner that had republished it under a different byline, once from a Google News query feed I had set up and forgotten, and once from a generated feed I had built months earlier when the site had briefly dropped its own. I had engineered the mess myself and then blamed the reader.
Duplicates are the thing most people quit RSS over, and they almost never complain about it in those words. They say the reader feels noisy. Noise and duplication look identical from the unread count, which is why the fix starts with working out which copy came from where rather than hunting for a setting that will make it stop.
The four ways duplicates get in
I sorted every repeat in my own list over about six weeks, and all of them fell into four buckets. Knowing which bucket you are looking at decides the fix, and three of the four have nothing to do with your app.
The same article from two different sources. This is the biggest cause and the least technical. Wire services, partner networks and aggregators republish the same text, sometimes with a new headline. Your reader has no way to know these are one piece of writing, because they arrive from unrelated addresses with different titles and different links. No deduplication feature can catch this reliably.
The same article arriving twice from one feed. This is a publisher problem. Every item in a feed carries an identifier, and a well behaved site keeps that identifier fixed for the life of the post. Some content systems regenerate it when an editor fixes a typo, changes a category or republishes with a new date. Your reader sees a new identifier and files a new item. Sites that run a staging to production workflow do this constantly, and news sites that update a running story do it on purpose.
Two subscriptions to one feed. Easier to do than it sounds. A feed served at the address with a trailing slash and the same feed served without one count as two subscriptions in most apps. So do the secure and insecure versions, the version with the www prefix and the version without, and the comments feed that sits one path segment away from the posts feed. I found three of these in my list, all added on different devices.
Generated or bridged feeds overlapping a real one. If you built a feed with a scraper service for a site that later shipped its own, you now have both. The same goes for bridge services that turn a social account into a feed when the person posts links to their own blog. The bridge gives you the post and the blog feed gives you the post.
Finding out which one you have in about five minutes
Open two of the duplicate items and compare their link addresses first. If the links are different domains, you have a syndication overlap and the answer is unsubscribing from one source. If the links are identical, the problem is upstream of your reader or inside your subscription list.
For identical links, check your subscription count for that site. Most readers have a search box over the feed list. Search the domain name rather than the publication name, because the two subscriptions usually have slightly different titles pulled from different versions of the feed metadata.
If there is only one subscription and the items still repeat, open the raw feed in a browser tab and look at the item identifiers. In an RSS feed the tag is guid. In an Atom feed it is id. Compare the identifier on the duplicate pair. Two identical identifiers mean your reader has a bug worth reporting. Two different identifiers for the same article mean the publisher is regenerating them, and the only real options are tolerance or leaving.
Here is how the symptoms map, condensed from my own notes.
| What you see | Most likely cause | Fix |
|---|---|---|
| Same headline, different sites in the source line | Syndication or aggregator overlap | Drop the aggregator, keep the original publisher |
| Identical item, same source, arrives hours apart | Identifier regenerated after an edit | Nothing you can change, decide if the feed is worth it |
| Everything from one site doubles, permanently | Two subscriptions to the same feed | Search the feed list by domain, delete one |
| Doubles only on posts the author also shared socially | A bridge feed overlapping the blog feed | Keep the blog feed, narrow the bridge to replies only |
| Old posts reappear as unread in a batch | Feed migrated and changed its identifier scheme | Mark all as read once, then leave it alone |
| One article appears five or six times in a news folder | Multiple keyword query feeds matching the same story | Collapse the queries into one, make it narrower |
Why turning on deduplication is the wrong first move
The standard answer in support forums is to enable duplicate filtering. A handful of readers offer it, usually matching on title or on the link address, and the self hosted ones let you write rules. I ran title based filtering for about three months and turned it off.
Matching on title hides legitimately different articles. Two publications writing about the same quarterly results will often land on the same seven word headline, and only one of them will have the detail you wanted. Matching on the link address catches almost nothing, because the duplicate copies that actually bother you come from different addresses by definition. The feature works on the narrow case where a publisher pushes the identical item twice, which is the case you can most easily live with.
The deeper issue is that duplication is a signal about your subscription list, and filtering silences the signal. Nine copies of one story means I subscribed to nine sources covering the same beat. The honest response is to pick two. I cut my news folder from 31 feeds to 11 that autumn and my daily unread count dropped by about 60 percent, while the number of stories I actually learned about stayed the same as far as I could tell.
There is one exception I keep. For a running news story that updates through the day, repeated items are useful, and I would rather see the third version than miss the correction. That folder gets no filtering and no pruning, and I read it once in the evening instead of all day.
What to do this week
Export your subscriptions as an OPML file, open it in a text editor, and sort the lines. Exact duplicate addresses jump out immediately, and near duplicates with a slash or a protocol difference sit next to each other once sorted. That alone caught five of mine. Then skim the list for query feeds and generated feeds, and ask of each one whether the site it points at now publishes a feed of its own.
After that, go folder by folder and count how many feeds in each cover the same subject. Anything above four in one subject area is where your duplicates live. Pruning overlapping sources beats every filter I have tried, and it is the same exercise described in separating daily, weekly and reference feeds, approached from the other end.
Do this in the same sitting as your dead feed sweep, since you have the OPML file open anyway. The routine for that is in finding feeds that stopped publishing. Between the two passes, a reader that felt unusable usually comes back to something you can finish in fifteen minutes a day. The rest of our reader coverage gets into the app specific settings.
FAQ
Does any reader handle duplicates well?
The self hosted ones give you the most control, since you can write rules against the title, the author and the content body rather than one field. Among hosted apps, the ones aimed at news professionals tend to cluster similar stories rather than hide them, which is more useful. None of them solve syndication overlap on their own.
Why do old posts suddenly show up as unread?
Usually a site changed platforms and the new system issues fresh identifiers for everything. Your reader cannot tell the difference between a new post and an old post with a new identifier. Mark the folder read and it will settle within a day.
Should I subscribe to a site and its newsletter?
Pick one. The newsletter is almost always the same posts with a wrapper, and you end up reading the one that arrives first and deleting the other. I keep the feed and unsubscribe from the newsletter, except where the newsletter carries extra writing that never appears on the site.
Can I merge two feeds into one deduplicated feed?
Some feed building services will combine sources and drop items with matching titles, and it works acceptably for a tight topic with two or three sources. The failure mode is quiet: a dropped item you never knew existed. I would rather see the duplicate than trust a middle layer to choose.
More in RSS Readers
RSS Readers
How to Sync an RSS Reader Across Phone and Laptop
I cleared 46 items on the train on a Tuesday, closed my phone, opened the laptop at my desk, and there they were again.
RSS Readers
How to Find and Remove Dead Feeds From Your RSS Reader
Last month I exported my subscriptions and counted them for the first time in two years. There were 214 feeds.
RSS Readers
Getting Full Text RSS From Feeds That Only Send Summaries
You open your reader on the train, tap a post from a blog you like, and get two sentences and a "Read more" link.