Feed Viewer logo Feed ViewerRead the web on your terms
RSS Readers

Duplicate Articles in Your RSS Reader and How to Fix It

One Tuesday in September I opened my reader to 140 unread items and found that 23 of them were the same nine articles.

An RSS reader window showing the same headline repeated several times from different sources in one unread list

One Tuesday in September I opened my reader to 140 unread items and found that 23 of them were the same nine articles. A piece about shipping delays appeared four times: once from the publication itself, once from a syndication partner that had republished it under a different byline, once from a Google News query feed I had set up and forgotten, and once from a generated feed I had built months earlier when the site had briefly dropped its own. I had engineered the mess myself and then blamed the reader.

Duplicates are the thing most people quit RSS over, and they almost never complain about it in those words. They say the reader feels noisy. Noise and duplication look identical from the unread count, which is why the fix starts with working out which copy came from where rather than hunting for a setting that will make it stop.

The four ways duplicates get in

I sorted every repeat in my own list over about six weeks, and all of them fell into four buckets. Knowing which bucket you are looking at decides the fix, and three of the four have nothing to do with your app.

The same article from two different sources. This is the biggest cause and the least technical. Wire services, partner networks and aggregators republish the same text, sometimes with a new headline. Your reader has no way to know these are one piece of writing, because they arrive from unrelated addresses with different titles and different links. No deduplication feature can catch this reliably.

The same article arriving twice from one feed. This is a publisher problem. Every item in a feed carries an identifier, and a well behaved site keeps that identifier fixed for the life of the post. Some content systems regenerate it when an editor fixes a typo, changes a category or republishes with a new date. Your reader sees a new identifier and files a new item. Sites that run a staging to production workflow do this constantly, and news sites that update a running story do it on purpose.

Two subscriptions to one feed. Easier to do than it sounds. A feed served at the address with a trailing slash and the same feed served without one count as two subscriptions in most apps. So do the secure and insecure versions, the version with the www prefix and the version without, and the comments feed that sits one path segment away from the posts feed. I found three of these in my list, all added on different devices.

Generated or bridged feeds overlapping a real one. If you built a feed with a scraper service for a site that later shipped its own, you now have both. The same goes for bridge services that turn a social account into a feed when the person posts links to their own blog. The bridge gives you the post and the blog feed gives you the post.

Finding out which one you have in about five minutes

Open two of the duplicate items and compare their link addresses first. If the links are different domains, you have a syndication overlap and the answer is unsubscribing from one source. If the links are identical, the problem is upstream of your reader or inside your subscription list.

For identical links, check your subscription count for that site. Most readers have a search box over the feed list. Search the domain name rather than the publication name, because the two subscriptions usually have slightly different titles pulled from different versions of the feed metadata.

If there is only one subscription and the items still repeat, open the raw feed in a browser tab and look at the item identifiers. In an RSS feed the tag is guid. In an Atom feed it is id. Compare the identifier on the duplicate pair. Two identical identifiers mean your reader has a bug worth reporting. Two different identifiers for the same article mean the publisher is regenerating them, and the only real options are tolerance or leaving.

Here is how the symptoms map, condensed from my own notes.

What you seeMost likely causeFix
Same headline, different sites in the source lineSyndication or aggregator overlapDrop the aggregator, keep the original publisher
Identical item, same source, arrives hours apartIdentifier regenerated after an editNothing you can change, decide if the feed is worth it
Everything from one site doubles, permanentlyTwo subscriptions to the same feedSearch the feed list by domain, delete one
Doubles only on posts the author also shared sociallyA bridge feed overlapping the blog feedKeep the blog feed, narrow the bridge to replies only
Old posts reappear as unread in a batchFeed migrated and changed its identifier schemeMark all as read once, then leave it alone
One article appears five or six times in a news folderMultiple keyword query feeds matching the same storyCollapse the queries into one, make it narrower

Why turning on deduplication is the wrong first move

The standard answer in support forums is to enable duplicate filtering. A handful of readers offer it, usually matching on title or on the link address, and the self hosted ones let you write rules. I ran title based filtering for about three months and turned it off.

Matching on title hides legitimately different articles. Two publications writing about the same quarterly results will often land on the same seven word headline, and only one of them will have the detail you wanted. Matching on the link address catches almost nothing, because the duplicate copies that actually bother you come from different addresses by definition. The feature works on the narrow case where a publisher pushes the identical item twice, which is the case you can most easily live with.

The deeper issue is that duplication is a signal about your subscription list, and filtering silences the signal. Nine copies of one story means I subscribed to nine sources covering the same beat. The honest response is to pick two. I cut my news folder from 31 feeds to 11 that autumn and my daily unread count dropped by about 60 percent, while the number of stories I actually learned about stayed the same as far as I could tell.

There is one exception I keep. For a running news story that updates through the day, repeated items are useful, and I would rather see the third version than miss the correction. That folder gets no filtering and no pruning, and I read it once in the evening instead of all day.

What to do this week

Export your subscriptions as an OPML file, open it in a text editor, and sort the lines. Exact duplicate addresses jump out immediately, and near duplicates with a slash or a protocol difference sit next to each other once sorted. That alone caught five of mine. Then skim the list for query feeds and generated feeds, and ask of each one whether the site it points at now publishes a feed of its own.

After that, go folder by folder and count how many feeds in each cover the same subject. Anything above four in one subject area is where your duplicates live. Pruning overlapping sources beats every filter I have tried, and it is the same exercise described in separating daily, weekly and reference feeds, approached from the other end.

Do this in the same sitting as your dead feed sweep, since you have the OPML file open anyway. The routine for that is in finding feeds that stopped publishing. Between the two passes, a reader that felt unusable usually comes back to something you can finish in fifteen minutes a day. The rest of our reader coverage gets into the app specific settings.

FAQ

Does any reader handle duplicates well?

The self hosted ones give you the most control, since you can write rules against the title, the author and the content body rather than one field. Among hosted apps, the ones aimed at news professionals tend to cluster similar stories rather than hide them, which is more useful. None of them solve syndication overlap on their own.

Why do old posts suddenly show up as unread?

Usually a site changed platforms and the new system issues fresh identifiers for everything. Your reader cannot tell the difference between a new post and an old post with a new identifier. Mark the folder read and it will settle within a day.

Should I subscribe to a site and its newsletter?

Pick one. The newsletter is almost always the same posts with a wrapper, and you end up reading the one that arrives first and deleting the other. I keep the feed and unsubscribe from the newsletter, except where the newsletter carries extra writing that never appears on the site.

Can I merge two feeds into one deduplicated feed?

Some feed building services will combine sources and drop items with matching titles, and it works acceptably for a tight topic with two or three sources. The failure mode is quiet: a dropped item you never knew existed. I would rather see the duplicate than trust a middle layer to choose.

MO
Mateus Oyelaran

Mateus ran social media accounts for brands before deciding most feeds were designed to be endless. He now writes about curating them back into something useful.

More posts by Mateus

More in RSS Readers

01doorganico 084goad 3hot8 4465456 after5internprogram ahandfulofblessings ahmadpc ahomcybersecurity allodialresources annodex antiquesandperiodcollectibles aotc asianworldkashmir beedlectrics bloggertemplatefree bluefeatheragency bodyweightsolution campaign08blog carfixadvisor catstreetcollective cbstvsports cnfanspurchasing committeeof500yearsofdignityandresistance covertopsdjs cs-kaiyuangame cuestacomunicaciontotal cxoutlet dancebetgame deskgearhub destinyvillastay detectmagazine di33 diamondsteelbuildings donmo dyanadropship emarsysfinance emimattl escapechan eskitarih evautoplay everythingweddings-orlando farmzone foxstudiosscratch gamevui123 gleasongrade greetingsday hacklabasu haskell iw-kaiyun jagoanproperti jilidskripsi joltbusinessfinance juegodepokergratis juegospucca kaeru-festival latestmehndidesigns lestanzediottaviano litemagazines localcommunityhero lustrsazan mariashebets menswheels newwebstechs1 nikkit ootheday oxigenvpn plurfitness prefetial probudgetcarpetcleaning psychicmrsmartin pyaten qvantor rejoignezlogis rexzbot riselayouts ruchin rwxqsgd5 sakaltime shibiaowang shop-bot skrtips spotlesshaven strcoding tetapatrium topselller uneplanteartificielle vipertechcr webtech1223411 wo-kygaming yamaha-mekar yeknava yoybuysheets zenunwa yamaha-mekar jilidskripsi ankertoto ankertoto ankertoto