"Good artists borrow, great artists steal", Pablo Piccasso
It's not a perfect analogy for software, but it sure gets your attention. A different way to say this is "Don't borrow the ideas of other products, steal the building blocks themselves instead of rebuilding them."
Software teams are able to get new products and updates to market much faster than they were 10 and 20 years ago. Much of the productivity gains have come from "stolen"
building blocks that have been made open source or integrated into the development stacks that we use.
When I helped build Netscape Navigator nearly 20 years ago we had to start at a very low level and build everything up. The networking libraries started with TCP sockets and even then there was not a consistent API. The UI was built up from fonts, pixels, primitive menu API's and drawing commands. When we wanted to display a GIF or JPEG we wrote all the code ourselves. Everything was built using plain old "C" and we had to build all of our support libraries from scratch. Simple list structures, hashing functions, time functions, threads, better memory allocation to deal with 16 bit machines. We eventually built a cross platform support infrastructure called the Netscape Portable Runtime (NSPR) that encapsulated all the support code into one library and let anyone build tools on top of it. We also gave it away, so anyone could use it. Thousands of people over the last 20 years have been building similar libraries and making them available through open source for others to build upon.
Today when we build an application we can code the UI in HTML and let the browser figure out how to render it. We can build the business logic in a high level scripting language that handles threading, memory management and a whole host of other complex low level problems for us. We have access to thousands of open source libraries that can do incredibly complex tasks. On top of all that we can even host our applications in the cloud and let someone else build, maintain and monitor all our hardware for us.
New code is still hard
UI's are easier, infrastructure is easier, memory management is easier, but coding is still pretty much the same. We have lots of new buzzwords for software development these days: Scrum, Agile, Extreme, Waterfall. It all sounds very exciting, but I can tell you that team programming is still pretty much the same. I have been doing versions of Agile programming for more than 20 years and we have now codified the methodology significantly, but the nature of the game has not changed very much. New code is still hard. It takes longer that we estimate and getting it bug free is very hard. Keeping it bug free while extending it is even harder.
Don't build anything you don't have to
Every single thing you build will take about 10x the time in maintenance
than the time it takes to build. The things we build tend not to be as fully featured or as well thought out as a successful open source library and likely won't have half
the features of the components you could have "stolen".
So when do you decide to build something?
Deciding when to build something new versus using something that exists is really tricky. One of the hardest parts is knowing what is out there. There are literally tens of thousands of open source libraries that we could grab that do virtually anything. If you want to make smart decisions about when to "steal" a component you need to educate yourself and get to know what is available. It may seem like a lot of time spent researching and reading, but imagine how much time you are going to save by not having to build a major component for your development project. And how much time you will have later on by not have to fix the bugs and add features because the component you found is already feature complete and stable. Get on the internet and ask people if searching doesn't yield results.
Spend all your time wisely
In a start-up, and in any company, but especially a start-up, you should spend 100% of your time building unique value. Time spent reinventing the wheel does nothing to generate unique value. Investors won't care that you built the coolest new version of crontab to control your back end processes, they care how many users you have and how much money you can make.
Use your components wisely
Don't try to use components in ways they were not designed to be used. First, you will likely find bugs that no one else has encountered because they have never used that component in that particular way. Second, it probably won't scale and perform the way you expect.
Don't try to second guess the design unless you are an expert
Understand and follow the best practices for each component. You might not understand all the reasons for the best practices, but they are there. If you want to use the Zend PHP framework you are going to have to use a model-view-controller paradigm. You may think that it's too complicated for the simple website that you are trying to get done right now and you don't want to spend the time to learn how Zend's M-V-C model works, but you need to do it. There are really good reasons why you should do these things, but they really don't manifest themselves until you start maintaining the code or decide to change the UI later in the product life cycle. Or you might decide that SQL is way too complicated and design an intermediate language to sit on top of it. If you think that's a good idea and your not a bona fide SQL and relational DB expert you are going to get in trouble. SQL got complicated for a reason, trying to second guess 30+ years of iteration is very dangerous.
Don't use components that suck
I have a whole separate blog article on this one. "For software development tools it's the ecosystem that matters most". Make sure all the components and tools you use are bulletproof. Time spent fixing someone else's code takes away from your product.
Give back
On a final note, I would like to encourage you to give back when you can. All of the code that we "steal" comes from somewhere and wouldn't be there without the hard work and generosity of those who put it there. Recognize that and give back to the open source community by contributing new libraries and enhancements and bug fixes. I have open sourced almost all the code I have written and it has been a highly rewarding experience.
Showing posts with label Software Development. Show all posts
Showing posts with label Software Development. Show all posts
Tuesday, June 18, 2013
Wednesday, June 12, 2013
For software development tools it's the ecosystem that matters most
In this article I will be exploring the different types of software development tools and how to choose the right ones for your next start-up or project. In my career I have had the pleasure of working with several successful software teams on a variety of projects. I have also had the opportunity to mentor or review more than a hundred software development teams. I have written programs for Windows, Mac and Unix, Android and a bunch of Web delivered apps. At Netscape I was part of a small cross platform development team that wrote Netscape Navigator which at that time was the most used application in the world.
In this article I would like to concentrate on platforms and tools. Platforms and tools are one of the least understood or appreciated since they technically focused and most engineers don't get the opportunity to choose their tools. Whenever a new software project is created we make important choices about what platform and tools we are going to build with. I'm defining the platform to be the hardware and operating system and the tools to be everything else above the operating system.
Platform
Platform choices used to be pretty easy: Windows, Mac or Unix. When the world was 90% Windows, most software developers knew right away that they would be using Windows as their platform. Today we have many choices. Win, Mac, Linux, Android, iOS, Blackberry, Windows Mobile or HTML. Platform choices are usually driven by business reasons, but there are still technical choices to be made. For instance should you have a native windows or Mac app? Web applications are preferable for most since they solve the distribution and update problem, but they are not able to integrate seamlessly with the computer and can't handle long running background tasks. Music players are a great example of this. Lots of music companies have cloud players that run in the browser, but most choose to do a native app to support offline play and to have a fancy native UI. Mobile app companies also have an important choice about what platforms to support. Many mobile start-ups I see are just concentrating on their mobile apps, and not creating a web based experience. Unless they are a gaming company this seems extremely short sighted. Most of us live and work in a heterogeneous environment, so a web based experience allows us to work with a service no matter where we are. GoodReads and Google Maps are good examples of this.
Tools
I find that tools are the most difficult choices that we make at the beginning of a project. There are many different classes of development tools. These are just a few:
- Software language: C++, Java, PHP, Ruby, ...
- Framework: LAMP, Zend, Rails, Tomcat, ...
- Compilers & IDEs: G++, MSVC, Eclipse, XCode, ...
- Debuggers: GDB
- Databases: Oracle, MySQL, Postgres, Mongo, Cassandra
- Bug tracking: Bugzilla, Jira, FogBugz, ...
- Revision Control: SVN, Git, CVS, Perforce, ...
- Integration: Jenkins, Bamboo, ...
- Unit Testing: CppUnit, Google Test, JUnit
- Static Analysis: Lint, Klocwork, Coverity, ...
- Hosting: Equinix, Amazon, RackSpace, Google, ...
Many software tools are chosen on the basis of what was used on the last project. Familiarity is an important consideration, but should not be the only one. If we simply go with what we know we are missing opportunities to choose a tool that has advanced or is simply better suited to the current project. Here are some important criteria to consider:
- How suited is the tool to what I am building?
- Will the tool handle the scalability that I need?
- Maturity
- Supporting tools
- Community
How suited is the tool to what I am building?
To a hammer the whole world looks like a nail. Perhaps you really need a wrench or a screwdriver? If you are building a financial application, don't try to use a fancy new noSQL database. Use a good old fashioned conservative SQL database. If the type of app you are writing has a large portion of users using one tools set, you should probably choose that. Why try and build an Android app with MSVC when everyone else does it with Eclipse?
Scalability
If you are building a web application for a small group, performance probably doesn't matter. If you are trying to be the next Facebook, you might want to think about how the tools you are using will scale up to millions of transactions and a big design team. Facebook is experiencing this problem in spades with their choice of PHP as a language. Their improvements to PHP are very impressive, but the effort could have been avoided by choosing a different language. Scalability also applies to team programming. Dynamically typed languages, such as PHP, Ruby and Python can be challenging for very large software projects since the functions are not explicitly typed. This makes it hard to detect software problems at link time and requires engineers to document their code in ways that are not enforced by a compiler.
Maturity
Tool maturity is too often overlooked. I think this one should be easy, but I see start-ups making the wrong choices over and over again. If a tool isn't 99.999% stable, you probably should not use it. All of your time should spent solving technical and business challenges. If a start-up needs to spend time debugging their tools they are wasting valuable time and resources solving a problem that they didn't need to solve. How should you evaluate maturity? Past personal experience or a referral by someone you know and trust is best. Otherwise, look to see who else is using it. If you can't find a large list of other companies that are using the tool in a production environment, stay away. Even the most promising tool can turn into a nightmare if you make it part of your product and it starts to crash or otherwise misbehave. Building tools is hard, not just because they are hard to build, but because they are very hard to perfect. You need tools that work every time, not just most of the time.
Supporting tools
This is probably the most important tool trait. Every tool class listed above comes with an ecosystem of tools that support it. With a database you need backup tools, profiling tools, monitoring tools, API libraries and documentation. New tools often look promising until you look under the covers and find out that it is lacking basic support infrastructure. Do you really want to find out after you go into production that your tool needs to be taken offline just to back it up?
Profiling and debugging tools are critically important and are often the last thing on a product road map. If you have ever experienced a critical performance bottleneck and have been without a real profiling tool you will understand how important a profiler is. How about finding a thread deadlock with a debugging tool that doesn't properly deal with threads? Memory corruption detection without a dedicated memory debugger? These are support tools that usually take many years if not decades to get right in a major tool class, but they are absolutely necessary.
Community
Last but not least is the community of people that can support you for questions and problems. The absence of a strong community should be a huge red flag. Poke around and ask questions, make sure that there is still active development of the tool that you are choosing and lots of active users. Look for example code and places where people congregate to ask and answer questions. You will save a lot of time if you can draw upon the community to figure out problems for you. Open source libraries built for your tool can also make your job a lot easier. Plugins to support your other tool choices and layer in great product features.
Looking at the whole stack
Now that you have a framework for evaluating individual tools, you now need to go up a level and look at your entire tool stack. Almost all of the development tools you use will interact with each other so they need to be compatible. Want to use a Java based messaging framework but you use C++ everywhere else? Think again. Love that C++ static analysis tool, but you use Python? Sorry no dice. Your choices on one tool effect every other tool you will use. Make sure that you evaluate the whole stack before you commit to any one tool. Otherwise your one choice will likely dictate and limit all your other tool choices.
Current Experience Levels
Last but not least please be realistic about your team's experience relative to the tools you are choosing. It might seem like a great idea to use Ruby on Rails for your next major web project, but if there is nobody on your team that has any experience with Ruby you are probably going to run into trouble. If you already know PHP the benefits to moving to Ruby are probably overshadowed by all the mistakes that you will make learning a whole new framework. Remember that the end product is more important than the language used to express it and the user isn't going to care if you are using the latest and coolest technology to pull it off. Conversely, using a technology that very few people know, even if your team is very proficient with it, will limit your ability to hire new people in the future.
Apply this same methodology to support libraries as well
All of the points above also apply to support libraries that you will use in your product. Here are a few examples.
- Logging
- Messaging frameworks
- General Utility libraries such as boost for C++
- Caching and proxy libraries and servers
- XML & JSON libraries
Summary
Look at all your tools carefully before you start using them. Make sure that they are all of the highest quality and maturity and that they come with the supporting infrastructure to handle the whole development process: Coding - Testing - Debugging - Profiling - Monitoring
Doing this will save you and your team countless hours of frustration relative to less mature tools.
Monday, June 10, 2013
An HTML5 fancy user interface project
| Sunburt graph |
History
HTML has come a long way from the early days when I was working on Lynx. The latest and greatest features are pushing graphics and user interfaces that are approaching native code levels of complexity and performance. I'm particularly fascinated by these features because they represent the realization of a vision that we had in the early days of Netscape where HTML could take over as a complete application programing interface.
WWW applications?
Using the Web just like a computer program seems pretty obvious these days, we use it that way on many if not most web sites, but in the beginning it was really just a document publishing platform. Hypertext, the basis for the web, was designed as a way to make documents more powerful by linking them together with references that could be accessed incredibly easily. The idea of putting powerful application logic behind a hypertext interface was pretty far fetched in the early '90s. I happened to have a need to do just that due to a need at my university. I came up with some ideas and put forth an idea to the other developers I knew that were working on the web. This idea soon sparked the addition of form elements to the web. Later while at Netscape, we took the next step by adding a scripting language, Javascript, to add even more client side capabilities and responsiveness.
Full GUI support
One of the visions we had at Netscape was to create a platform that had a fully complete and functional set of graphical elements so that other developers could create any application imaginable. The foundational elements are reasonably well known and are included in all modern operating systems and major graphics libraries. The difficulty in delivering GUI primitives to the Web was figuring out a way to efficiently add them to HTML and to make them massively cross platform. When I say "massively cross platform" I don't mean just Windows and Mac. We were thinking about operating systems, screen sizes, resolution, and a whole bunch of other things. The web was designed from the ground up to try to be device neutral and that makes the implementation very hard. CSS is one technology that supports multiple displays by providing different layout and styling for different displays as required. Netscape as a company ran out of life before it was able to complete its roadmap for a full complement of GUI primitives, but it did make a good effort. A technology called XUL (pronounced Zool) was introduced that allowed a large variety of GUI to be expressed, unfortunately, XUL didn't move forward fast enough and was essentially dead on arrival. Simultaneously Microsoft smothered the market with Internet Explorer and tried its best to keep the Web from moving forward with HTML.
| Tree graph |
Enter HTML5
Fast forward several years from the Netscape era and the world has changed substantially. Firefox, Chrome and a variety of Webkit based browsers have wrestled away the strangle hold that IE6 held on the market and began to make substantial improvements to the abilities of HTML. HTML5 was soon born and with it a whole bunch of new graphical technologies that enable many new types of GUI applications. SVG, advanced CSS, Canvas and Javascript now combine to create an ability to create very rich interfaces that are entirely driven within the Web browser.
My experiment
A while back I came across a group that was creating a set of tools that harnessed the new power of HTML5. The project is called D3 for Data Driven Documents. I saw what D3 could do graphically and wanted to try it out and do something personally useful with the technology. Since my current company Zetta.net is in the cloud storage space I decided to write a tool to visualize the size of the data on a computer hard drive. One of the difficult issues with a tool to analyze disk space is that it needs access to all of the data on a drive and that data is sensitive and private. I needed to figure out a way to scan the data and not violate a user's privacy. Standard HTML applications run in the cloud and couldn't do something like this without sending data to a server, but the new world of HTML5 can make everything happen locally in the browser.
Some complications
At this point in my project I had an idea of what I wanted to do, and a fancy new graphical library for graphing, but I still needed a way to collect the disk space data to be graphed. Javascript has chosen to stay away from local file access to simplify the security model, but Java long ago added the ability to ask the user for local disk access. I ended up writing a Java beans integrated module that spawns a Java application to scan the local disk and return the data to Javascript running in the browser for analysis. The beauty of the Java/Javascript integration is that none of the data ever leaves the local computer that it is running on. The downside is that Java is often broken on user machines, or is disabled due to security concerns. I hope at some point Javascript figures out a workable security model to allow some forms of local disk access.
| Tree map graph |
The finished project
Please check out the finished project if you are interested. I have made a video demo and a walk through to explain the use and features. There is also sample data to play around with that skips the scanning process. The code has been published as open source with a BSD license. This project just might come in handy for you the next time you run out of disk space and need to know what is using it all up.
http://wheresmydiskspace.com/
Saturday, May 18, 2013
It's the end of <blink>!
Firefox was the last holdout of the blink tag, but has finally removed the evil. For sentimental reasons I will show you a method for bringing it back. Also see my other post on the creation of the blink tag.
CSS to the rescue! Did I mention that I helped with the creation of the first version of CSS? CSS has gotten quite a bit fancier in recent years and now supports animations, which allow you to do many very fancy tricks, one of the easiest is blinking:
<style>
.keyframeBlink {
animation-name: onOff;
animation-duration: .8s;
animation-iteration-count: infinite;
animation-direction: alternate;
}
@keyframes onOff {
from { opacity: 1; }
to {opacity: 0; }
}
</style>
<div class="keyframeBlink">This should blink with css animations, images work too: </div>
You can also get fancier and change the rate of blinking and the style of the fade.
Here is a 1 sec duration and the use of: "
Now kids don't try this at home, people will make fun of you. I am definitely not encouraging anyone to do this and if you ask me I will disavow any knowledge of this post. In fact, just working on this post, which I disavow working on, is giving me a headache.
NOTES: On chrome you will need to make the CSS a bit uglier since they are not yet supporting the new naming scheme. They are using the -webkit- prefix on all the animation keywords. I would expect that to change soon so that the -webkit- prefix will be unnecessary. Here is a great site for an explanation: http://www.css3files.com/animation/
Your style element will look like this to make it work cross browser for now:
<style>
.keyframeBlink {
-webkit-animation-name: onOff;
-webkit-animation-duration: .8s;
-webkit-animation-iteration-count: infinite;
animation-name: onOff;
animation-duration: .8s;
animation-iteration-count: infinite;
}
@keyframes onOff {
from { opacity: 1; }
to {opacity: 0; }
}
@-webkit-keyframes onOff {
from { opacity: 1; }
to {opacity: 0; }
}
</style>
CSS to the rescue! Did I mention that I helped with the creation of the first version of CSS? CSS has gotten quite a bit fancier in recent years and now supports animations, which allow you to do many very fancy tricks, one of the easiest is blinking:
<style>
.keyframeBlink {
animation-name: onOff;
animation-duration: .8s;
animation-iteration-count: infinite;
animation-direction: alternate;
}
@keyframes onOff {
from { opacity: 1; }
to {opacity: 0; }
}
</style>
<div class="keyframeBlink">This should blink with css animations, images work too: </div>
This is an example of text and an image blinking! 
You can also get fancier and change the rate of blinking and the style of the fade.
Here is a 1 sec duration and the use of: "
animation-timing-function: ease-in-out"
Fancy blinking example, notice the nice fade in and out!
Now kids don't try this at home, people will make fun of you. I am definitely not encouraging anyone to do this and if you ask me I will disavow any knowledge of this post. In fact, just working on this post, which I disavow working on, is giving me a headache.
NOTES: On chrome you will need to make the CSS a bit uglier since they are not yet supporting the new naming scheme. They are using the -webkit- prefix on all the animation keywords. I would expect that to change soon so that the -webkit- prefix will be unnecessary. Here is a great site for an explanation: http://www.css3files.com/animation/
Your style element will look like this to make it work cross browser for now:
<style>
.keyframeBlink {
-webkit-animation-name: onOff;
-webkit-animation-duration: .8s;
-webkit-animation-iteration-count: infinite;
animation-name: onOff;
animation-duration: .8s;
animation-iteration-count: infinite;
}
@keyframes onOff {
from { opacity: 1; }
to {opacity: 0; }
}
@-webkit-keyframes onOff {
from { opacity: 1; }
to {opacity: 0; }
}
</style>
Friday, May 17, 2013
Why blocking 3rd party cookies could be a bad thing
3rd party Cookies are used to facilitate targeted advertising and can be used to track your browsing across multiple Web sites.
I am not comfortable with being tracked across the web by Cookies, and Cookies were specifically designed to try and prevent this sort of behavior, so why would I be against disabling the major mechanism that is used for web tracking?
The answer is pretty simple:
The evil you know is better than the one you don't
Right now we know that advertising companies use 3rd party cookies to track behaviors and serve custom ads. The companies that use these methods are well known to us and use similar methods. They almost exclusively use 3rd party cookies. With this knowledge it is very easy to disable tracking: Go into your browser preferences and disable 3rd party cookies. For those who care there is a simple and reliable method to disable tracking.
Now lets suppose that someone decides to turn off 3rd party cookies by default in most browsers. What do you think will happen? Will the advertising industry just close up shop and move on to other businesses? Will web tracking just cease to happen? Of course not! Advertising companies will move to other technical methods for user tracking. They exist already, they are just a bit less convenient than 3rd party cookies to use. The new situation is one that will be much harder for an individual user to disable tracking.
This is probably a race we can't win
Think of this battle similar to the challenges of DRM. Companies spend lots of time and money to design a super secure system to protect their digital content and within a week or two the DRM is broken and the content is all over the internet. DRM is typically broken by small groups of unpaid hackers who do it for a hobby and they win just about every time. Now think about how difficult it will be to stop a multi-billion dollar industry from winning a similar war. If you are a browser company how are you going to stop well funded companies from coming up with technical ways to serve targeted ads? There are already multiple ways to do tracking without 3rd party cookies and countless others could be created if enough effort is put into it.
What can we do?
My suggestion is to handle this politically not technically. Keep working on Do-Not-Track and other mechanisms. Keep working on legislative methods to restrict the policies. Those are wars that we can win and those policies can have real teeth to them. One of the benefits of the 3rd party cookie tracking techniques is that it is really easy to see who is doing the tracking. If we make the societal choice to prohibit tracking then it will be easy to track down offenders.
User tracking is a hot issue and should continue to be discussed and debated. It is important to keep in mind that targeted ads are paying for almost all of the things that are on the Internet today. If we make a wholesale decision to stop serving ads are we all prepared to start paying for all of the services that we are getting for free now? Perhaps many people would agree to some level of targeted advertising if they saw the reasons and cost benefits clearly enough. What I don't think will be useful is to start a technical war that disables the current working methods for disabling tracking and replaces them with poorly understood and less visible techniques.
I am not comfortable with being tracked across the web by Cookies, and Cookies were specifically designed to try and prevent this sort of behavior, so why would I be against disabling the major mechanism that is used for web tracking?
The answer is pretty simple:
- The evil you know is better than the one you don't.
- This is probably a race we can't win.
The evil you know is better than the one you don't
Right now we know that advertising companies use 3rd party cookies to track behaviors and serve custom ads. The companies that use these methods are well known to us and use similar methods. They almost exclusively use 3rd party cookies. With this knowledge it is very easy to disable tracking: Go into your browser preferences and disable 3rd party cookies. For those who care there is a simple and reliable method to disable tracking.
Now lets suppose that someone decides to turn off 3rd party cookies by default in most browsers. What do you think will happen? Will the advertising industry just close up shop and move on to other businesses? Will web tracking just cease to happen? Of course not! Advertising companies will move to other technical methods for user tracking. They exist already, they are just a bit less convenient than 3rd party cookies to use. The new situation is one that will be much harder for an individual user to disable tracking.
This is probably a race we can't win
Think of this battle similar to the challenges of DRM. Companies spend lots of time and money to design a super secure system to protect their digital content and within a week or two the DRM is broken and the content is all over the internet. DRM is typically broken by small groups of unpaid hackers who do it for a hobby and they win just about every time. Now think about how difficult it will be to stop a multi-billion dollar industry from winning a similar war. If you are a browser company how are you going to stop well funded companies from coming up with technical ways to serve targeted ads? There are already multiple ways to do tracking without 3rd party cookies and countless others could be created if enough effort is put into it.
What can we do?
My suggestion is to handle this politically not technically. Keep working on Do-Not-Track and other mechanisms. Keep working on legislative methods to restrict the policies. Those are wars that we can win and those policies can have real teeth to them. One of the benefits of the 3rd party cookie tracking techniques is that it is really easy to see who is doing the tracking. If we make the societal choice to prohibit tracking then it will be easy to track down offenders.
User tracking is a hot issue and should continue to be discussed and debated. It is important to keep in mind that targeted ads are paying for almost all of the things that are on the Internet today. If we make a wholesale decision to stop serving ads are we all prepared to start paying for all of the services that we are getting for free now? Perhaps many people would agree to some level of targeted advertising if they saw the reasons and cost benefits clearly enough. What I don't think will be useful is to start a technical war that disables the current working methods for disabling tracking and replaces them with poorly understood and less visible techniques.
Tuesday, May 14, 2013
The reasoning behind Web Cookies
I get a fair number of questions about cookies from individuals and
the press. I thought I would try and explain some of the motivation
and history behind Web cookies as well as some of the design behind
them. Feel free to post more questions and I will try and expand
this article with more details.
See my recent post on 3rd party cookies as well!
Motivation
HTTP is the networking language behind your browser. HTTP, or Hyper Text Transport Protocol, was designed and introduced as part of the WWW project that started with Tim Berners-Lee at CERN and was expanded upon by several universities, corporations and private individuals. The WWW and HTTP are open standards which are published and given into the public trust in order to foster interoperability and to create products that can become ubiquitous.
One of the problems faced in the early years of the web was how to create websites that had "memory" for individual users. The uses of "memory" on a website are many: shopping carts for shopping, personalized content, logging in, and many other interactive features require memory. The problem in 1994 was a lack of mechanisms to identify a user individually. HTTP was designed to be fast and efficient and part of its design was to connect to a website, grab a document and then disconnect. This freed up the website to serve other customers, but it also meant that there was no concept of a session. Without a session, each time a user clicked to move to a different page they would become just another random user with no way to associate them with an action they had done just moments ago. This is a bit like talking to someone with Alzheimer disease. Each interaction would result in having to introduce yourself again, and again, and again.
History
By the summer 1994 the idea of adding some form of "memory" to HTTP had been kicking around the WWW design groups for a while. I'm not sure exactly how long but I would guess for 1 or 2 years. There were some interesting proposals, but one of the popular ones that kept coming up was to add a unique identifier to every web browser so that a web site could individually identify each user use that to build a session. I was very much against this concept because the unique identifier could be used to track a user at every website. Without another proposal the idea of adding sessions or memory remained dormant.
Sometime around July of 1994 I went to a meeting to talk about a shopping server that another group at Netscape was working on. They needed to build shopping cart functionality into the server, but they didn't have a good way to make it work with the existing technology. They explained what they wanted to do and I had to shake my head and agree with them and tell them that they could not do it in any reasonable way with what existed. I promised to think about the problem and get back to them. At some point during the next week the general concept of Web Cookies formed in my head. The idea of allowing a single web site to send a session identifier to the browser that would get sent back only to the server appealed to me and prevented cross site tracking. I wanted to create something that had more utility so that we could do a lot more than just shopping carts, so I extended the concept of the session identifier into a general payload that would get sent back to the server.
Design
With a rough sketch in my head I worked on the general design of cookies. The goal was to create a session identifier and general "memory" mechanism for websites that didn't allow for cross site tracking. I discussed the design with other members of the engineering team and got useful feedback, especially with regard to denial of service attracts and other security concerns. John Giannandrea was especially helpful with the design. The end result was the cookie specification that still defines 95% of what cookies do today. A Web server can return information to the browser that the browser should give back to the server each time it contacts the server in the future. Within the Web Cookie, a server can embed a random session identifier, a username, a shopping cart, or anything it wants as long as it doesn't try to send something too big or set too many cookies. There are also restrictions on how cookies can be transferred, either over secure connections or not and if the cookie should be erased when the user closes the browser or reboots their computer.
The name
I had heard the term "magic cookie" from an operating systems course from college. The term has a somewhat similar meaning to the way Web Cookies worked and I liked the term "cookies" for aesthetic reasons. Cookies was the first thing I came up with and the name stuck.
Usefulness
We released the Netscape browser in the fall of 1994 and within a year it was the most popular browser in the world. We also released the Cookie specification so that other browsers could implement it and so that web sites would know how to use it. With most of the world using a browser that supported Web Cookies, web sites started using cookies for many different things and in ways that we could not have predicted. Most of those uses were fantastic, some of them were concerning.
By 1996 and 1997 the web had grown dramatically and was a big business. Most websites offered their content for free and were advertising supported. Advertisers were looking for ways to increase the effectiveness of their ads and they looked to Web Cookies to help them with that.
An Unforeseen Problem
A problem that I missed during the Cookie design phase was an interaction between cookies and embedded content within a webpage. Webpages start as a single HTML document on one server. That one HTML document contains references to other resources that are loaded to display the site that the user sees. Images, videos, more text, and plug-ins are all references within an HTML document and any of those resources can be loaded from anywhere in the world. This referencing technique is one of the things that make the Web so amazingly powerful. When Web Cookies are combined with embedded references that point to other websites they are called "3rd party cookies" and they represent a new way in which users can be tracked across multiple web sites.
Advertising uses
Big ad companies, in particular DoubleClick, started using 3rd party cookies to track a browser uniquely across all of the sites that used DoubleClick to serve ads. The use of the tracking wasn't to actually identify the name of a user, but to make sure they didn't see the same ad every time or to track the number of unique users that saw a particular ad. Eventually the ads were customized to reflect the browsing habits of the unique identifiers. If a browser had looked at kayaks and racquetball racquets in the past then they were given ads for sporting goods and kayaks and racquets.
Around 1996 ad tracking via Cookies became a hot topic. Users were upset with the practice and asking for change. Tracking across websites was certainly not what cookies were designed to do, they were designed with the opposite intention, but what could be done at that point to fix the problem? One of the solutions we came up with was to add controls to give each user control over what cookies to accept and from whom. The big question at the time was whether or not to disable 3rd party cookies completely or not. The decision had wide ranging effects since advertising was paying for most of the web and disabling 3rd party cookies would also disable many legitimate uses for cookies on embedded resources.
An uncomfortable decision
In the end the decision to disable 3rd party cookies or keep them on was left to me. I agonized about the decision for weeks and in the end I chose to keep them. My logic was two fold:
Durability
The design of Cookies have remained fairly unchanged for the last 19 years. The biggest changes have been to how users can view and control their cookies. Many people have proposed alternatives, but none have caught on. Cookies are not perfect, but they have certainly proved good enough and much of the functionality of the web depends on them.
Summary
Web Cookies find themselves in the midst of a very controversial area, and have gained a level of notoriety because of it. Even though they were designed to protect privacy they have been used in ways that are sometimes infringing upon it. Many efforts have been taken to protect the privacy of Web users and Cookie controls have been put into the hands of every user. The future of Cookies and user privacy is surely to continue as an interesting news item for a very long time and rightfully so. The nature of the advertising business is to collect as much information as it possibly can, so the public needs to push back when it goes too far.
See my recent post on 3rd party cookies as well!
Motivation
HTTP is the networking language behind your browser. HTTP, or Hyper Text Transport Protocol, was designed and introduced as part of the WWW project that started with Tim Berners-Lee at CERN and was expanded upon by several universities, corporations and private individuals. The WWW and HTTP are open standards which are published and given into the public trust in order to foster interoperability and to create products that can become ubiquitous.
One of the problems faced in the early years of the web was how to create websites that had "memory" for individual users. The uses of "memory" on a website are many: shopping carts for shopping, personalized content, logging in, and many other interactive features require memory. The problem in 1994 was a lack of mechanisms to identify a user individually. HTTP was designed to be fast and efficient and part of its design was to connect to a website, grab a document and then disconnect. This freed up the website to serve other customers, but it also meant that there was no concept of a session. Without a session, each time a user clicked to move to a different page they would become just another random user with no way to associate them with an action they had done just moments ago. This is a bit like talking to someone with Alzheimer disease. Each interaction would result in having to introduce yourself again, and again, and again.
History
By the summer 1994 the idea of adding some form of "memory" to HTTP had been kicking around the WWW design groups for a while. I'm not sure exactly how long but I would guess for 1 or 2 years. There were some interesting proposals, but one of the popular ones that kept coming up was to add a unique identifier to every web browser so that a web site could individually identify each user use that to build a session. I was very much against this concept because the unique identifier could be used to track a user at every website. Without another proposal the idea of adding sessions or memory remained dormant.
Sometime around July of 1994 I went to a meeting to talk about a shopping server that another group at Netscape was working on. They needed to build shopping cart functionality into the server, but they didn't have a good way to make it work with the existing technology. They explained what they wanted to do and I had to shake my head and agree with them and tell them that they could not do it in any reasonable way with what existed. I promised to think about the problem and get back to them. At some point during the next week the general concept of Web Cookies formed in my head. The idea of allowing a single web site to send a session identifier to the browser that would get sent back only to the server appealed to me and prevented cross site tracking. I wanted to create something that had more utility so that we could do a lot more than just shopping carts, so I extended the concept of the session identifier into a general payload that would get sent back to the server.
Design
With a rough sketch in my head I worked on the general design of cookies. The goal was to create a session identifier and general "memory" mechanism for websites that didn't allow for cross site tracking. I discussed the design with other members of the engineering team and got useful feedback, especially with regard to denial of service attracts and other security concerns. John Giannandrea was especially helpful with the design. The end result was the cookie specification that still defines 95% of what cookies do today. A Web server can return information to the browser that the browser should give back to the server each time it contacts the server in the future. Within the Web Cookie, a server can embed a random session identifier, a username, a shopping cart, or anything it wants as long as it doesn't try to send something too big or set too many cookies. There are also restrictions on how cookies can be transferred, either over secure connections or not and if the cookie should be erased when the user closes the browser or reboots their computer.
The name
I had heard the term "magic cookie" from an operating systems course from college. The term has a somewhat similar meaning to the way Web Cookies worked and I liked the term "cookies" for aesthetic reasons. Cookies was the first thing I came up with and the name stuck.
Usefulness
We released the Netscape browser in the fall of 1994 and within a year it was the most popular browser in the world. We also released the Cookie specification so that other browsers could implement it and so that web sites would know how to use it. With most of the world using a browser that supported Web Cookies, web sites started using cookies for many different things and in ways that we could not have predicted. Most of those uses were fantastic, some of them were concerning.
By 1996 and 1997 the web had grown dramatically and was a big business. Most websites offered their content for free and were advertising supported. Advertisers were looking for ways to increase the effectiveness of their ads and they looked to Web Cookies to help them with that.
An Unforeseen Problem
A problem that I missed during the Cookie design phase was an interaction between cookies and embedded content within a webpage. Webpages start as a single HTML document on one server. That one HTML document contains references to other resources that are loaded to display the site that the user sees. Images, videos, more text, and plug-ins are all references within an HTML document and any of those resources can be loaded from anywhere in the world. This referencing technique is one of the things that make the Web so amazingly powerful. When Web Cookies are combined with embedded references that point to other websites they are called "3rd party cookies" and they represent a new way in which users can be tracked across multiple web sites.
Advertising uses
Big ad companies, in particular DoubleClick, started using 3rd party cookies to track a browser uniquely across all of the sites that used DoubleClick to serve ads. The use of the tracking wasn't to actually identify the name of a user, but to make sure they didn't see the same ad every time or to track the number of unique users that saw a particular ad. Eventually the ads were customized to reflect the browsing habits of the unique identifiers. If a browser had looked at kayaks and racquetball racquets in the past then they were given ads for sporting goods and kayaks and racquets.
Around 1996 ad tracking via Cookies became a hot topic. Users were upset with the practice and asking for change. Tracking across websites was certainly not what cookies were designed to do, they were designed with the opposite intention, but what could be done at that point to fix the problem? One of the solutions we came up with was to add controls to give each user control over what cookies to accept and from whom. The big question at the time was whether or not to disable 3rd party cookies completely or not. The decision had wide ranging effects since advertising was paying for most of the web and disabling 3rd party cookies would also disable many legitimate uses for cookies on embedded resources.
An uncomfortable decision
In the end the decision to disable 3rd party cookies or keep them on was left to me. I agonized about the decision for weeks and in the end I chose to keep them. My logic was two fold:
Any company that had the ability to track users across a large section of the web would need to be a large publicly visible company. Cookies could be seen by users so a tracking company can't hide from the public. In this way the public has a natural feedback mechanism to constrain those that would seek to track them.Today, I still believe that it was the correct decision. Governments have an ability to regulate the collection of data by large visible companies and has shown a willingness to do so. The public has a responsibility to keep pressure on both the companies that have the ability to track users and governments to enact reasonable privacy regulations and enforce them. Most importantly, there are other mechanisms that can replace Web cookies for tracking if they are universally disabled, and those mechanisms would be much harder to observe and disable.
If 3rd party cookies were disabled ad companies would use another mechanism to accomplish the same thing, and that mechanism would not have the same level of visibility and control as cookies. We would be trading out one problem for another.
Durability
The design of Cookies have remained fairly unchanged for the last 19 years. The biggest changes have been to how users can view and control their cookies. Many people have proposed alternatives, but none have caught on. Cookies are not perfect, but they have certainly proved good enough and much of the functionality of the web depends on them.
Summary
Web Cookies find themselves in the midst of a very controversial area, and have gained a level of notoriety because of it. Even though they were designed to protect privacy they have been used in ways that are sometimes infringing upon it. Many efforts have been taken to protect the privacy of Web users and Cookie controls have been put into the hands of every user. The future of Cookies and user privacy is surely to continue as an interesting news item for a very long time and rightfully so. The nature of the advertising business is to collect as much information as it possibly can, so the public needs to push back when it goes too far.
Monday, May 13, 2013
A short history of the "about:" URL
Here is a fairly recent musing on "about:" URLs that I did just a few months ago.
Due to my lack of a real blog this may be the first time anyone will read it.
http://www.montulli.org/lou/about_urls
In case you are wondering, about URL's are used by your web browser to do things that URL's really should not do.
There is more info on about URLs on Wikipedia of course: http://en.wikipedia.org/wiki/About_URI_scheme
The origins of the <blink> tag
Here is an article I posted in 2009. I think it got picked up by a few of the tech sites as well.
http://www.montulli.org/theoriginofthe<blink>tag
Gizmodo reference: http://gizmodo.com/5903827/the-humble-origins-of-the-html-blink-tag
Recently Google named their new HTML engine after blink: http://blog.chromium.org/2013/04/blink-rendering-engine-for-chromium.html
And I'm definitely enjoying the name of this company: http://blinktag.com/
It seems that blink is having a resurgence in 2013, I guess it's old enough now to be retro cool.
http://www.montulli.org/theoriginofthe<blink>tag
Gizmodo reference: http://gizmodo.com/5903827/the-humble-origins-of-the-html-blink-tag
Recently Google named their new HTML engine after blink: http://blog.chromium.org/2013/04/blink-rendering-engine-for-chromium.html
And I'm definitely enjoying the name of this company: http://blinktag.com/
It seems that blink is having a resurgence in 2013, I guess it's old enough now to be retro cool.
Subscribe to:
Posts (Atom)


