The Internet is at a tipping point. The continued rise of adblocking has put an end to the revenue model that relies solely on ad dollars to operate websites and businesses.
Especially news sites have started to experiment with ways to diversify income sources, and one prominent option that sites like The Wall Street Journal, Financial Times, The New York Times or The Washington Post have all implemented is the paywall system.
There are different types of paywalls but they all have in common that they block access to content either directly or after a certain number of articles have been read on site.
Visitors are then asked to subscribe to the site to continue reading articles on it.
It may make sense from a business point of view, and may be more lucrative than battling it out with users who run adblockers, but there is a downside to it both for the paywalled site and the blocked user.
Sites lose a high percentage of visitors if they implement a paywall system. It is unclear how high the percentage really is, and it probably varies from site to site, but it is likely a lot higher than the percentage of visitors who subscribe to the site after being presented with the choice to subscribe to read the desired article.
It is no secret that news sites allow access to news aggregators and search engines. If you check Google News or Search for instance, you will find articles from sites with paywalls listed there.
In the past, news sites allowed access to visitors coming from major news aggregators such as Reddit, Digg or Slashdot, but that practice seems to be as good as dead nowadays.
Another trick, to paste the article title into a search engine to read the cached story on it directly, does not seem to work properly anymore as well as articles on sites with paywalls are not usually cached anymore.
Update: The Wall Street Journal announced that it will plug the hole described below. You can still read articles behind the site's paywall however using the following method:
The page should reload, and the article should be loaded in full. You could also post the article link on Facebook, for instance in a new post that only you can see. Clicking on the posted link should load the article in its entirely on The Wall Street Journal website.
User-Agent and Referrer
You are probably wondering how sites block or allow access to the site's content. The methods have have improved over the years, and it is no longer enough to simply change the referrer of the browser to https://www.google.com/ to gain full access to a site's content.
Instead, sites use various checks that include user-agent, referrer and cookies, and sometimes even more than that, to determine the legitimacy of access.
Probably the best way to masquerade the browser is to make it appear to be Googlebot.
Firefox users need two browser add-ons for that: the first, RefControl, to change the referrer value when visiting news sites, the second, User Agent Switcher, to change the user agent of the browser.
Update: RefControl is no longer available. You may try this alternative instead. End
This is all there is to it. It is a bit unfortunate that there is no extension for Firefox that changes the user agent automatically based on the sites you visit.
There is however another possibility, and that is to create a custom extension which automates the process in the browser.
Instructions are provided on Elaineou. All it takes, basically, is to create a new directory on the local computer, create the two files background.js and manifest.json inside it, and copy and paste the code found on the site into the files.
You need to enable "developer mode" on chrome://extensions/, and can then select "load unpacked extension" to pick the folder you have created the two files in to load the extension in Chrome.
You may modify the list of sites it supports to add new ones.Advertisement
Ghacks is a technology news blog that was founded in 2005 by Martin Brinkmann. It has since then become one of the most popular tech news sites on the Internet with five authors and regular contributions from freelance writers.