Tuesday, May 14, 2013

The reasoning behind Web Cookies

I get a  fair number of questions about cookies from individuals and the press.    I thought I would try and explain some of the motivation and history behind Web cookies as well as some of the design behind them.      Feel free to post more questions and I will try and expand this article with more details.

See my recent post on 3rd party cookies as well!


Motivation
HTTP is the networking language behind your browser.    HTTP, or Hyper Text Transport Protocol, was designed and introduced as part of the WWW project that started with Tim Berners-Lee at CERN and was expanded upon by several universities, corporations and private individuals.    The WWW and HTTP are open standards which are published and given into the public trust in order to foster interoperability and to create products that can become ubiquitous.

One of the problems faced in the early years of the web was how to create websites that had "memory" for individual users.    The uses of "memory" on a website are many: shopping carts for shopping, personalized content, logging in, and many other interactive features require memory.    The problem in 1994 was a lack of mechanisms to identify a user individually.   HTTP was designed to be fast and efficient and part of its design was to connect to a website, grab a document and then disconnect.   This freed up the website to serve other customers, but it also meant that there was no concept of a session.     Without a session, each time a user clicked to move to a different page they would become just another random user with no way to associate them with an action they had done just moments ago.     This is a bit like talking to someone with Alzheimer disease.    Each interaction would result in having to introduce yourself again, and again, and again.

History
By the summer 1994 the idea of adding some form of "memory" to HTTP had been kicking around the WWW design groups for a while.   I'm not sure exactly how long but I would guess for 1 or 2 years.    There were some interesting proposals, but one of the popular ones that kept coming up was to add a unique identifier to every web browser so that a web site could individually identify each user use that to build a session.    I was very much against this concept because the unique identifier could be used to track a user at every website.     Without another proposal the idea of adding sessions or memory remained dormant.
 
Sometime around July of 1994 I went to a meeting to talk about a shopping server that another group at Netscape was working on.     They needed to build shopping cart functionality into the server, but they didn't have a good way to make it work with the existing technology.    They explained what they wanted to do and I had to shake my head and agree with them and tell them that they could not do it in any reasonable way with what existed.     I promised to think about the problem and get back to them.    At some point during the next week the general concept of Web Cookies formed in my head.    The idea of allowing a single web site to send a session identifier to the browser that would get sent back only to the server appealed to me and prevented cross site tracking.    I wanted to create something that had more utility so that we could do a lot more than just shopping carts, so I extended the concept of the session identifier into a general payload that would get sent back to the server.   

Design
With a rough sketch in my head I worked on the general design of cookies.    The goal was to create a session identifier and general "memory" mechanism for websites that didn't allow for cross site tracking.    I discussed the design with other members of the engineering team and got useful feedback, especially with regard to denial of service attracts and other security concerns.   John Giannandrea was especially helpful with the design.   The end result was the cookie specification that still defines 95% of what cookies do today.     A Web server can return information to the browser that the browser should give back to the server each time it contacts the server in the future.     Within the Web Cookie, a server can embed a random session identifier, a username, a shopping cart, or anything it wants as long as it doesn't try to send something too big or set too many cookies.    There are also restrictions on how cookies can be transferred, either over secure connections or not and if the cookie should be erased when the user closes the browser or reboots their computer.  


The name

I had heard the term "magic cookie" from an operating systems course from college.     The term has a somewhat similar meaning to the way Web Cookies worked and I liked the term "cookies" for aesthetic reasons.     Cookies was the first thing I came up with and the name stuck.   


Usefulness

We released the Netscape browser in the fall of 1994 and within a year it was the most popular browser in the world.     We also released the Cookie specification so that other browsers could implement it and so that web sites would know how to use it.    With most of the world using a browser that supported Web Cookies, web sites started using cookies for many different things and in ways that we could not have predicted.      Most of those uses were fantastic, some of them were concerning.

By 1996 and 1997 the web had grown dramatically and was a big business.    Most websites offered their content for free and were advertising supported.     Advertisers were looking for ways to increase the effectiveness of their ads and they looked to Web Cookies to help them with that.    


An Unforeseen Problem
A problem that I missed during the Cookie design phase was an interaction between cookies and embedded content within a webpage.    Webpages start as a single HTML document on one server.   That one HTML document contains references to other resources that are loaded to display the site that the user sees.    Images, videos, more text, and plug-ins are all references within an HTML document and any of those resources can be loaded from anywhere in the world.     This referencing technique is one of the things that make the Web so amazingly powerful.     When Web Cookies are combined with embedded references that point to other websites they are called "3rd party cookies" and they represent a new way in which users can be tracked across multiple web sites.

Advertising uses
Big ad companies, in particular DoubleClick, started using 3rd party cookies to track a browser uniquely across all of the sites that used DoubleClick to serve ads.     The use of the tracking wasn't to actually identify the name of a user, but to make sure they didn't see the same ad every time or to track the number of unique users that saw a particular ad.     Eventually the ads were customized to reflect the browsing habits of the unique identifiers.    If a browser had looked at kayaks and racquetball racquets in the past then they were given ads for sporting goods and kayaks and racquets.    

Around 1996 ad tracking via Cookies became a hot topic.    Users were upset with the practice and asking for change.     Tracking across websites was certainly not what cookies were designed to do, they were designed with the opposite intention, but what could be done at that point to fix the problem?        One of the solutions we came up with was to add controls to give each user control over what cookies to accept and from whom.    The big question at the time was whether or not to disable 3rd party cookies completely or not.     The decision had wide ranging effects since advertising was paying for most of the web and disabling 3rd party cookies would also disable many legitimate uses for cookies on embedded resources.    

An uncomfortable decision
In the end the decision to disable 3rd party cookies or keep them on was left to me.     I agonized about the decision for weeks and in the end I chose to keep them.   My logic was two fold:

Any company that had the ability to track users across a large section of the web would need to be a large publicly visible company.   Cookies could be seen by users so a tracking company can't hide from the public.    In this way the public has a natural feedback mechanism to constrain those that would seek to track them.

If 3rd party cookies were disabled ad companies would use another mechanism to accomplish the same thing, and that mechanism would not have the same level of visibility and control as cookies.   We would be trading out one problem for another.

Today, I still believe that it was the correct decision.     Governments have an ability to regulate the collection of data by large visible companies and has shown a willingness to do so.    The public has a responsibility to keep pressure on both the companies that have the ability to track users and governments to enact reasonable privacy regulations and enforce them.      Most importantly, there are other mechanisms that can replace Web cookies for tracking if they are universally disabled, and those mechanisms would be much harder to observe and disable.

Durability
The design of Cookies have remained fairly unchanged for the last 19 years.   The biggest changes have been to how users can view and control their cookies.     Many people have proposed alternatives, but none have caught on.     Cookies are not perfect, but they have certainly proved good enough and much of the functionality of the web depends on them.   

Summary
Web Cookies find themselves in the midst of a very controversial area, and have gained a level of notoriety because of it.    Even though they were designed to protect privacy they have been used in ways that are sometimes infringing upon it.    Many efforts have been taken to protect the privacy of Web users and Cookie controls have been put into the hands of every user.    The future of Cookies and user privacy is surely to continue as an interesting news item for a very long time and rightfully so.    The nature of the advertising business is to collect as much information as it possibly can, so the public needs to push back when it goes too far. 




15 comments:

  1. Hi There,
    Thank you for sharing the knowledgeable blog with us I hope that you will post many more blog with us:-
    Our Minneapolis bakery lobby reopens on Monday, August 23. We're excited to welcome you back for walk-in cookie purchases.
    Email:sales@big-smokes.com
    Click here for more information:- more info

    ReplyDelete
  2. Howdy! I could have sworn I've visited your blog before but after looking at some of the articles I realized it's new to me. Anyways, I'm certainly happy I found it and I'll be book-marking it and checking back often! Check out our Hosting Coupons

    ReplyDelete
  3. Bubble Shooter Games 2 has lots of bubbles, pro color and ball shooting games in 2022. Bubble Shooter Classic is best played with pro friends. Match 3 to pop origin bubbles. This new game is first launched in 2022 game.

    ReplyDelete
  4. charlidameliomerch.shopHello, thank you for communicating your article with us and I would return to your website

    ReplyDelete
  5. I really appreciate to read this blog it is very impressive and informative content. If you want to know how to make a YouTube video into a ringtone for iPhone? if yes, then unlike the simple and almost effortless solution for iOS, converting YouTube videos to ringtones for iPhone can be quite challenging, but with the help of the iPhone ringtone site you can easily convert your YouTube videos into a ringtone, for additional info I recommended you can read one article about make ringtone for iPhone, in this article you there are various easy ways to create and set the iPhone ringtone, Without wasting time you can go and create ringtones now and enjoy them.

    ReplyDelete
  6. The goal of Routers.Guru is to assist people in extracting the most value possible from the technology that permeates every aspect of our lives today. We offer context for the most recent technological news, in addition to an extensive library of up-to-date educational how-to articles and impartial product advice and reviews.동대문스웨디시
    중랑스웨디시
    성북스웨디시
    강북스웨디시
    도봉스웨디시
    노원스웨디시
    은평스웨디시

    ReplyDelete
  7. so Legit Arms shows you everything related to this world to help you decide on the one that best suits your
    taste and needs. Please feel free to visit our website
    https://www.legitarmsdealer.com/
    https://t.me/buyfirearmsonline
    https://www.buycounterfeitbills.com/
    https://mrgoldenretriever.com/
    https://capuchinmonkey.company.com/

    ReplyDelete
  8. Hollywood is worlds bigger and premium industry which adds many faces from all walks of lives each year. Hollywood industry is having very lengthy list, however we tried to bring you some of the very popular & world-famous Hollywood actress list.문경피쉬아로마
    상주피쉬아로마
    안동피쉬아로마
    영천피쉬아로마
    영주피쉬아로마

    ReplyDelete
  9. The Zillow API Python is an invaluable tool for real estate enthusiasts and developers. Its seamless integration with Python allows for easy access to Zillow's vast database of property information, empowering users to extract and analyze data with efficiency. A game-changer for those seeking real estate insights and building innovative applications.

    ReplyDelete