Another housing bubble in the Bay Area

I moved to the Bay Area in 1994.

Source: Zillow.com

I bought my first house in 1996.

I sold that house in 2000 for more than twice what I paid for it, and moved up to Truckee, CA.

I moved back to the Bay Area in 2006 during the last housing bubble and felt that the market was overvalued, so I decided to rent.    I found a high end rental house and have been renting for the last 7 years.  I have saved roughly $600K in housing costs during that time vs owning the same house.

Record growth this year
The market this year is red hot.    Year over year home prices are up about 31%.    Across the street from me a developer built a 4,000 sqft house on a 8,000 sqft lot and sold it for $4 million.    The most expensive home on the street previously had been only $2.5 million.   The owner of the house I have been leasing noticed and has wisely decided to cash in on the current market, so now I need to find a new place to live.   I have been making the tour of houses in Menlo Park for the last two months and I'm exasperated.  Very average houses are going for well above asking price and land values are crazy.    Complete tear downs are selling for close to $2 million.  How does this make sense?    I created a spread sheet to try and analyze the market and see if it makes any sense.

According to Zillow the average house in Menlo Park is currently $1.46 million dollars, that is up from $1.04 million in May of 2011.   Keep in mind that these are averages and include condos and distressed properties and houses literally right next to highway 101.    The real truth is that you can't buy a "normal" house for less than $1.9 million and that "normal" house will be about 1,600 sqft.   The average cost per square foot is at or above $1000 sq/ft.   Any house that you might consider "fancy" is 3 million or more.

The recommended guidelines for housing costs are about 28% of after tax income.   In California, people are used to spending more income on housing so doubling the figure is about 55%.   I built a spread sheet that calculates the full cost of owning a home and will calculate the income required for 28%, 55% and 75% of income going towards housing.   The spreadsheet takes into account the mortgage interest tax deduction and the property tax deductions.

If I run the numbers for $1.46 million and $1.9 million here are incomes necessary to have 28%, 55% and 75% of income going to housing

Necessary income for $1.4 Million house

28.00%	55.00%	75.00%
$688,198.86	$322,812.16	$221,766.95

Necessary income for $1.9 Million house

28.00%	55.00%	75.00%
$885,066.96	$421,091.92	$292,782.77

Now lets look at the average income in Menlo Park.    Average income can be misleading since it also includes people who are not home owners.   The published average household income for Menlo Park is about $110,000.    If I dig a little deeper and take just the most affluent area of Menlo Park we can find that the household income for West Menlo is about $220,000.  (The data source is city-data.com)

So we can see that at $1.4 million the average household would be spending 75% of income on housing.   The real reality of $1.9 million would find that average household would have to spend 94.5% of their income on housing.

I think that it is pretty clear from the data that the average person cannot afford the average house right now.    But what does that mean?

What is causing the spike in prices
If the average person living in an area could not afford to re-buy the house that they are living in, what does that mean?   

• The neighborhood could have a recent influx of new buyers that are more wealthy than the existing base.  (This is usually called gentrification)
• Supply is constrained and this is temporarily pushing up prices.
• Interest rates are about to rise and people are buying before they go up.
• The Facebook effect?
• The tulip craze is back.

Is it really Gentrification?
Gentrification is a possibility, but is it really happening?  How many people really make $885K a year?     Let's me more reasonable, lets allow for 55% of income for affordability, so the real question is how many people really make over $420K.    According to the IRS in 2010, 800,000 people in the US make $500K or more.   How many of them live in California?   I can't find a direct source for this, but we can estimate it.    According to the IRS in 2009 about 20% of Millionaires live in California.   If I generously estimate that 75% of them live in the Bay Area and not LA then we have about 15% of all the Millionaires living in the Bay Area.    That's 120,000 people (very approximately) in the Bay Area with household income above $500,000 per year.     So I guess this is  a possibility.  

Similar Neighborhoods in the penninsula:  (number of households)

• Menlo Park: (12,000)
• Palo Alto:   (25,000)
• Atherton:   (2,500)
• Los Altos:  (10,000)
• Hillsborough: (3,600)
• Los Gatos:  (12,000)
• Woodside:   (2,000)
• Total: (67,100)  peninsula only

Constrained Supply
This does seem to be a serious problem.   There just are not enough houses for all the people moving here.    Additionally, Prop 13 encourages people to stay in their homes and never move, so there is very little turn over in the market.   High prices combined with taxes hurt turnover as well.    If you are living in a house that is suddenly worth 2 million dollars it is very hard to sell because taxes will take half the proceeds.   Then you need someplace else to live and everything costs more than a million dollars.

Basic economics says that prices will go up if there is more demand than supply.    But how high and why now?    Supply has been a problem for the last 20 years.   Why is the market up 30% in one year?

Interest rates on the rise
I think the recent spike may be related to interest rates.   As interest rates climb the monthly cost of a home goes up substantially.    I did all my affordability calculations at 4% interest.   Here is a table of annual cost for a 1.9 million dollar house at different interest rates: 

4%	5%	6%	7%
$146,000	$160,000	$173,000	$188,000

That equates to more than a thousand dollars a month in difference for each percentage point.    At 7% you would spend $42,000 more per year on the same house than at 4%.    If interest rates return to normal levels of 7% over the next few years, it should cool the market considerably.


Facebook?
There is more than 75 Billion dollars worth of new value in the Menlo Park area.   The vast majority of that is owned by a small number of people, but there are at least a few hundred new Millionaires in the last year.

Tulip craze?
Are we in another temporary bubble?    I really don't know, but I hope so.    Nobody benefits from sky high real estate prices, except the real estate industry itself.    Surrounding neighborhoods are also experiencing record setting prices.    The Zillow Palo Alto index is over $1.7 Million and even towns like Sunnyvale and Mountain View, which are traditionally far less expensive, are approaching $1 million.     Local companies will not be able to attract talent to the area if people can't find an affordable place to live.   Keep in mind that the national average for a house is $158,300.    I would never expect the Bay Area market to be average, but I would expect it to be closer to 5x the norm rather than 10x.

What am I going to do?
I love living in the Bay Area and I love Menlo Park most of all.    The combination of an amazing climate, proximity to fantastic natural wonders like Tahoe, Marin and Napa and last but certainly not least the amazing people who live here.     If I could convince myself that the market won't fall again in the near future I would be happy to buy an overpriced home here.   Unfortunately I'm convinced of the opposite.   I think the market will correct downward at least 25% in the future, but I don't know how long it will take.

Some people called me crazy back in 2006 when I predicted a correction.   They said that real estate in California never goes down and even bet me that it wouldn't.  I won money on those bets, but I was polite enough not to collect since the folks who said it lost a lot in the down turn.    I was complacent in 2009-2010 and didn't buy then, so now I'm in a difficult spot, my mistake.

Barring a happy accident falling into my lap with an under priced gem, I will be renting again.    The rental market is just recently starting to correct closer to ownership prices, but it is still a much better value than owning, it also carries no risk of a massive loss if the market corrects back to normal levels again.

Here is my spreadsheet that has the calculations

Please leave a comment with your favorite theory and prediction for housing prices in the future.

Why great software artists steal rather than borrow

"Good artists borrow, great artists steal", Pablo Piccasso

It's not a perfect analogy for software, but it sure gets your attention.   A different way to say this is "Don't borrow the ideas of other products, steal the building blocks themselves instead of rebuilding them."

Software teams are able to get new products and updates to market much faster than they were 10 and 20 years ago.   Much of the productivity gains have come from "stolen" building blocks that have been made open source or integrated into the development stacks that we use.

When I helped build Netscape Navigator nearly 20 years ago we had to start at a very low level and build everything up.    The networking libraries started with TCP sockets and even then there was not a consistent API.   The UI was built up from fonts, pixels, primitive menu API's and drawing commands.    When we wanted to display a GIF or JPEG we wrote all the code ourselves.   Everything was built using plain old "C" and we had to build all of our support libraries from scratch.    Simple list structures, hashing functions, time functions, threads, better memory allocation to deal with 16 bit machines.  We eventually built a cross platform support infrastructure called the Netscape Portable Runtime (NSPR) that encapsulated all the support code into one library and let anyone build tools on top of it.   We also gave it away, so anyone could use it.     Thousands of people over the last 20 years have been building similar libraries and making them available through open source for others to build upon.

Today when we build an application we can code the UI in HTML and let the browser figure out how to render it.   We can build the business logic in a high level scripting language that handles threading, memory management and a whole host of other complex low level problems for us.   We have access to thousands of open source libraries that can do incredibly complex tasks.    On top of all that we can even host our applications in the cloud and let someone else build, maintain and monitor all our hardware for us.

New code is still hard
UI's are easier, infrastructure is easier, memory management is easier, but coding is still pretty much the same.     We have lots of new buzzwords for software development these days:   Scrum, Agile, Extreme, Waterfall.   It all sounds very exciting, but I can tell you that team programming is still pretty much the same.    I have been doing versions of Agile programming for more than 20 years and we have now codified the methodology significantly, but the nature of the game has not changed very much.    New code is still hard.   It takes longer that we estimate and getting it bug free is very hard.   Keeping it bug free while extending it is even harder.

Don't build anything you don't have to
Every single thing you build will take about 10x the time in maintenance than the time it takes to build.   The things we build tend not to be as fully featured or as well thought out as a successful open source library and likely won't have half the features of the components you could have "stolen".   

So when do you decide to build something?
Deciding when to build something new versus using something that exists is really tricky.   One of the hardest parts is knowing what is out there.    There are literally tens of thousands of open source libraries that we could grab that do virtually anything.    If you want to make smart decisions about when to "steal" a component you need to educate yourself and get to know what is available.     It may seem like a lot of time spent researching and reading, but imagine how much time you are going to save by not having to build a major component for your development project.    And how much time you will have later on by not have to fix the bugs and add features because the component you found is already feature complete and stable.    Get on the internet and ask people if searching doesn't yield results.   

Spend all your time wisely
In a start-up, and in any company, but especially a start-up, you should spend 100% of your time building unique value.    Time spent reinventing the wheel does nothing to generate unique value.    Investors won't care that you built the coolest new version of crontab to control your back end processes, they care how many users you have and how much money you can make.  

Use your components wisely
Don't try to use components in ways they were not designed to be used.   First, you will likely find bugs that no one else has encountered because they have never used that component in that particular way.   Second, it probably won't scale and perform the way you expect.   

Don't try to second guess the design unless you are an expert
Understand and follow the best practices for each component.   You might not understand all the reasons for the best practices, but they are there.    If you want to use the Zend PHP framework you are going to have to use a model-view-controller paradigm.   You may think that it's too complicated for the simple website that you are trying to get done right now and you don't want to spend the time to learn how Zend's M-V-C model works, but you need to do it.    There are really good reasons why you should do these things, but they really don't manifest themselves until you start maintaining the code or decide to change the UI later in the product life cycle.   Or you might decide that SQL is way too complicated and design an intermediate language to sit on top of it.   If you think that's a good idea and your not a bona fide SQL and relational DB expert you are going to get in trouble.     SQL got complicated for a reason, trying to second guess 30+ years of iteration is very dangerous.

Don't use components that suck
I have a whole separate blog article on this one.  "For software development tools it's the ecosystem that matters most".   Make sure all the components and tools you use are bulletproof.    Time spent fixing someone else's code takes away from your product.

Give back
On a final note, I would like to encourage you to give back when you can.   All of the code that we "steal" comes from somewhere and wouldn't be there without the hard work and generosity of those who put it there.   Recognize that and give back to the open source community by contributing new libraries and enhancements and bug fixes.    I have open sourced almost all the code I have written and it has been a highly rewarding experience.

For software development tools it's the ecosystem that matters most

In this article I will be exploring the different types of software development tools and how to choose the right ones for your next start-up or project.

In my career I have had the pleasure of working with several successful software teams on a variety of projects.  I have also had the opportunity to mentor or review more than a hundred software development teams. I have written programs for Windows, Mac and Unix, Android and a bunch of Web delivered apps. At Netscape I was part of a small cross platform development team that wrote Netscape Navigator which at that time was the most used application in the world.

In this article I would like to concentrate on platforms and tools. Platforms and tools are one of the least understood or appreciated since they technically focused and most engineers don't get the opportunity to choose their tools.  Whenever a new software project is created we make important choices about what platform and tools we are going to build with.  I'm defining the platform to be the hardware and operating system and the tools to be everything else above the operating system.

Platform
Platform choices used to be pretty easy:  Windows, Mac or Unix. When the world was 90% Windows, most software developers knew right away that they would be using Windows as their platform.  Today we have many choices. Win, Mac, Linux, Android, iOS, Blackberry, Windows Mobile or HTML.  Platform choices are usually driven by business reasons, but there are still technical choices to be made.  For instance should you have a native windows or Mac app?  Web applications are preferable for most since they solve the distribution and update problem, but they are not able to integrate seamlessly with the computer and can't handle long running background tasks.   Music players are a great example of this.    Lots of music companies have cloud players that run in the browser, but most choose to do a native app to support offline play and to have a fancy native UI.  Mobile app companies also have an important choice about what platforms to support. Many mobile start-ups I see are just concentrating on their mobile apps, and not creating a web based experience. Unless they are a gaming company this seems extremely short sighted.  Most of us live and work in a heterogeneous environment, so a web based experience allows us to work with a service no matter where we are. GoodReads and Google Maps are good examples of this.

Tools
I find that tools are the most difficult choices that we make at the beginning of a project. There are many different classes of development tools. These are just a few:

• Software language:  C++, Java, PHP, Ruby, ...
• Framework: LAMP, Zend, Rails, Tomcat, ...
• Compilers & IDEs: G++, MSVC, Eclipse, XCode, ...
• Debuggers: GDB
• Databases: Oracle, MySQL, Postgres, Mongo, Cassandra
• Bug tracking: Bugzilla, Jira, FogBugz, ...
• Revision Control: SVN, Git, CVS, Perforce, ...
• Integration: Jenkins, Bamboo, ...
• Unit Testing: CppUnit, Google Test, JUnit
• Static Analysis: Lint, Klocwork, Coverity, ...
• Hosting: Equinix, Amazon, RackSpace, Google, ...

Many software tools are chosen on the basis of what was used on the last project.    Familiarity is an important consideration, but should not be the only one.    If we simply go with what we know we are missing opportunities to choose a tool that has advanced or is simply better suited to the current project.  Here are some important criteria to consider:

• How suited is the tool to what I am building?
• Will the tool handle the scalability that I need? 
• Maturity
• Supporting tools
• Community

How suited is the tool to what I am building?
To a hammer the whole world looks like a nail. Perhaps you really need a wrench or a screwdriver?   If you are building a financial application, don't try to use a fancy new noSQL database.   Use a good old fashioned conservative SQL database.  If the type of app you are writing has a large portion of users using one tools set, you should probably choose that.   Why try and build an Android app with MSVC when everyone else does it with Eclipse? 

Scalability
If you are building a web application for a small group, performance probably doesn't matter.   If you are trying to be the next Facebook, you might want to think about how the tools you are using will scale up to millions of transactions and a big design team.  Facebook is experiencing this problem in spades with their choice of PHP as a language.  Their improvements to PHP are very impressive, but the effort could have been avoided by choosing a different language.  Scalability also applies to team programming.    Dynamically typed languages, such as PHP, Ruby and Python can be challenging for very large software projects since the functions are not explicitly typed.  This makes it hard to detect software problems at link time and requires engineers to document their code in ways that are not enforced by a compiler.

Maturity
Tool maturity is too often overlooked.  I think this one should be easy, but I see start-ups making the wrong choices over and over again.  If a tool isn't 99.999% stable, you probably should not use it.  All of your time should spent solving technical and business challenges.  If a start-up needs to spend time debugging their tools they are wasting valuable time and resources solving a problem that they didn't need to solve.    How should you evaluate maturity?   Past personal experience or a referral by someone you know and trust is best.   Otherwise, look to see who else is using it.    If you can't find a large list of other companies that are using the tool in a production environment, stay away.    Even the most promising tool can turn into a nightmare if you make it part of your product and it starts to crash or otherwise misbehave.  Building tools is hard, not just because they are hard to build, but because they are very hard to perfect. You need tools that work every time, not just most of the time.

Supporting tools
This is probably the most important tool trait. Every tool class listed above comes with an ecosystem of tools that support it.  With a database you need backup tools, profiling tools, monitoring tools, API libraries and documentation.  New tools often look promising until you look under the covers and find out that it is lacking basic support infrastructure.  Do you really want to find out after you go into production that your tool needs to be taken offline just to back it up?

Profiling and debugging tools are critically important and are often the last thing on a product road map.  If you have ever experienced a critical performance bottleneck and have been without a real profiling tool you will understand how important a profiler is.  How about finding a thread deadlock with a debugging tool that doesn't properly deal with threads?  Memory corruption detection without a dedicated memory debugger?  These are support tools that usually take many years if not decades to get right in a major tool class, but they are absolutely necessary.   

Community
Last but not least is the community of people that can support you for questions and problems.  The absence of a strong community should be a huge red flag. Poke around and ask questions, make sure that there is still active development of the tool that you are choosing and lots of active users.  Look for example code and places where people congregate to ask and answer questions.   You will save a lot of time if you can draw upon the community to figure out problems for you.  Open source libraries built for your tool can also make your job a lot easier.  Plugins to support your other tool choices and layer in great product features.

Looking at the whole stack
Now that you have a framework for evaluating individual tools, you now need to go up a level and look at your entire tool stack.  Almost all of the development tools you use will interact with each other so they need to be compatible.  Want to use a Java based messaging framework but you use C++ everywhere else?   Think again.  Love that C++ static analysis tool, but you use Python?  Sorry no dice.  Your choices on one tool effect every other tool you will use.   Make sure that you evaluate the whole stack before you commit to any one tool.  Otherwise your one choice will likely dictate and limit all your other tool choices.    

Current Experience Levels
Last but not least please be realistic about your team's experience relative to the tools you are choosing.  It might seem like a great idea to use Ruby on Rails for your next major web project, but if there is nobody on your team that has any experience with Ruby you are probably going to run into trouble.  If you already know PHP the benefits to moving to Ruby are probably overshadowed by all the mistakes that you will make learning a whole new framework.  Remember that the end product is more important than the language used to express it and the user isn't going to care if you are using the latest and coolest technology to pull it off.  Conversely, using a technology that very few people know, even if your team is very proficient with it, will limit your ability to hire new people in the future.  

Apply this same methodology to support libraries as well
All of the points above also apply to support libraries that you will use in your product.   Here are a few examples.

• Logging
• Messaging frameworks
• General Utility libraries such as boost for C++
• Caching and proxy libraries and servers
• XML & JSON libraries

Summary
Look at all your tools carefully before you start using them.   Make sure that they are all of the highest quality and maturity and that they come with the supporting infrastructure to handle the whole development process:    Coding - Testing - Debugging - Profiling - Monitoring
Doing this will save you and your team countless hours of frustration relative to less mature tools.

An HTML5 fancy user interface project

Sunburt graph

I recently finished a new project that uses HTML5 to display and graph file system data and I thought I would write a blog post about it.
 
History
HTML has come a long way from the early days when I was working on Lynx. The latest and greatest features are pushing graphics and user interfaces that are approaching native code levels of complexity and performance. I'm particularly fascinated by these features because they represent the realization of a vision that we had in the early days of Netscape where HTML could take over as a complete application programing interface.

WWW applications?
Using the Web just like a computer program seems pretty obvious these days, we use it that way on many if not most web sites, but in the beginning it was really just a document publishing platform. Hypertext, the basis for the web, was designed as a way to make documents more powerful by linking them together with references that could be accessed incredibly easily.  The idea of putting powerful application logic behind a hypertext interface was pretty far fetched in the early '90s.  I happened to have a need to do just that due to a need at my university. I came up with some ideas and put forth an idea to the other developers I knew that were working on the web. This idea soon sparked the addition of form elements to the web. Later while at Netscape, we took the next step by adding a scripting language, Javascript, to add even more client side capabilities and responsiveness.

Full GUI support
One of the visions we had at Netscape was to create a platform that had a fully complete and functional set of graphical elements so that other developers could create any application imaginable. The foundational elements are reasonably well known and are included in all modern operating systems and major graphics libraries.  The difficulty in delivering GUI primitives to the Web was figuring out a way to efficiently add them to HTML and to make them massively cross platform.    When I say "massively cross platform" I don't mean just Windows and Mac. We were thinking about operating systems, screen sizes, resolution, and a whole bunch of other things.  The web was designed from the ground up to try to be device neutral and that makes the implementation very hard.  CSS is one technology that supports multiple displays by providing different layout and styling for different displays as required.  Netscape as a company ran out of life before it was able to complete its roadmap for a full complement of GUI primitives, but it did make a good effort.  A technology called XUL (pronounced Zool) was introduced that allowed a large variety of GUI to be expressed, unfortunately, XUL didn't move forward fast enough and was essentially dead on arrival.     Simultaneously Microsoft smothered the market with Internet Explorer and tried its best to keep the Web from moving forward with HTML.

Tree graph

Enter HTML5
Fast forward several years from the Netscape era and the world has changed substantially.    Firefox, Chrome and a variety of Webkit based browsers have wrestled away the strangle hold that IE6 held on the market and began to make substantial improvements to the abilities of HTML.  HTML5 was soon born and with it a whole bunch of new graphical technologies that enable many new types of GUI applications.    SVG, advanced CSS, Canvas and Javascript now combine to create an ability to create very rich interfaces that are entirely driven within the Web browser.

My experiment  
A while back I came across a group that was creating a set of tools that harnessed the new power of HTML5.  The project is called D3 for Data Driven Documents.  I saw what D3 could do graphically and wanted to try it out and do something personally useful with the technology.   Since my current company Zetta.net is in the cloud storage space I decided to write a tool to visualize the size of the data on a computer hard drive.  One of the difficult issues with a tool to analyze disk space is that it needs access to all of the data on a drive and that data is sensitive and private.  I needed to figure out a way to scan the data and not violate a user's privacy. Standard HTML applications run in the cloud and couldn't do something like this without sending data to a server, but the new world of HTML5 can make everything happen locally in the browser.

Some complications
At this point in my project I had an idea of what I wanted to do, and a fancy new graphical library for graphing, but I still needed a way to collect the disk space data to be graphed.  Javascript has chosen to stay away from local file access to simplify the security model, but Java long ago added the ability to ask the user for local disk access.  I ended up writing a Java beans integrated module that spawns a Java application to scan the local disk and return the data to Javascript running in the browser for analysis.  The beauty of the Java/Javascript integration is that none of the data ever leaves the local computer that it is running on. The downside is that Java is often broken on user machines, or is disabled due to security concerns.  I hope at some point Javascript figures out a workable security model to allow some forms of local disk access.

Tree map graph

The finished project
Please check out the finished project if you are interested. I have made a video demo and a walk through to explain the use and features.   There is also sample data to play around with that skips the scanning process.  The code has been published as open source with a BSD license.    This project just might come in handy for you the next time you run out of disk space and need to know what is using it all up.

http://wheresmydiskspace.com/

A phone for every feature. Where is Apple?

6/12/2013: Update:  The Galaxy Zoom was announced today.   This is a pretty great looking device.    If you are serious about photography and have been carrying around a Canon s100 plus a phone like I have, this could potentially make your pockets lighter.  (in two ways)

Today Samsung announced the Galaxy S4 active, a dust proof, water resistant, rugged phone.    Motorola is talking about the new X phone, a phone with a battery big enough to last for days and a host of new sensors to know when it's in your pocket, traveling in a car or just hanging out.    The latest generation of phones from Samsung and HTC include infrared transmitters and NFC and a screen sizes for every hand size and preference.  I'm overjoyed with the choices and features in phones these days, but in all of this I have to wonder where is Apple?  The former leader in phone innovation which introduced and dominated the smart phone market seems to have quietly slipped into an underdog role.   


A bet on the future
A few years back I had a thoughtful debate with a VC friend of mine about the future of the phone market.  I had just gotten a shiny new HTC Android phone and was a member or a small minority.    I was looking at the fairly unpolished features of Android 2.3 and seeing all the untapped potential.   My friend was basking in the glow of the polished iPhone experience and confident that Apple would continue its dominance in the smart phone marketplace.  Being an open source advocate certainly influenced my thinking, but I was confident that the open model would eventually produce a phone that would outclass the iPhone.    I was so confident in my new Android phone I bet one whole dollar that Android would eventually rule the market.  (wow!)
 
Android turns the tables
New Android phones today ship with a version of Android (4.2) that is significantly more polished than a few years ago.   Android is now a legitimately beautiful and elegant solution that still offers the flexibility and power to customize your phone endlessly to your own whims.
Android phones currently outsell the iPhone 3 to 1 in total units, but the iPhone is still a dominant player in the premium market.  Here in Silicon Valley most of the affluent people I know are still iPhone users and Apple still has a fantastic brand and an ability to move the entire market, but how long will that last?   If Apple continues to sell only one form factor and one premium phone can it really compete in a marketplace full of specialized choices?   Should I have to wrap my phone in a big case if I want to make it rugged?  Should I have to add an external battery case if I want the battery to last?  I want choices and I want my phone to fit my lifestyle.


What could Apple do?
I would like to see Apple learn from its previous mistakes and open up its ecosystem with other hardware vendors.    I want iOS to be set free.   I believe that if Apple had started licensing iOS several years ago and satisfied the demands of hardware manufacturers it would never have lost market share to Android and would control nearly the entire mobile market.    As a consumer I'm quite happy that there isn't one dominant closed source vendor, but from Apple's perspective iOS is really missing the market.    Waterproof rugged iPhones,  iPhones with 6 inch screens and a whole host of other features would be fantastic, and I would expect that at the top of the food chain Apple would still be selling the highest quality, highest margin iPhone.

Now that I've said it, forget it.
Apple will never do this.   They will argue that it would make it impossible to have a seamless experience, that they tried it before with Mac clones and it failed then, and that they are a hardware company.    Too bad, opportunity lost.    My prediction is that Apple's mobile market a few years from now will look a lot like their desktop market.   10-20% market share on the high end and no real influence in the true direction of the market.   They will still command high prices with reasonably high margins, but they will not have a dominate position in anything.

Is Apple losing market share a good thing?
Yes!   A market controlled by Apple or any other closed ecosystem vendor would be very scary to me.   Apple has a history of telling it's users what they want and ignoring market data.    Apple also is trying to extract high margins from its products and benefits by keeping prices high.    Fortunately for us the Android alternative is here and has fantastic fundamentals.    Android is, at its core, a true open source and truly open technology.   Android is built upon Linux, and most of the code is released under the Apache license.  That means that we can take the code and use it even if Google doesn't want us to.    There are parts of Android that are closed and proprietary, but the meaty core is open to us and essentially free forever.   So lets say Google goes all evil on us in the future?   Will they be able to pry my phone from my cold dead hands?   No, we the consumers, and any other company that wants to, can take the core of Android and release something even better.    

OK I'm biased, I admit it
Yes, I admit it, I'm shamelessly writing a blog article about a subject that people just love to argue about.   I'm also a huge open source bigot so I'm naturally inclined to like Android.     I really just wanted to write about how much I love the progress that Android has made and the fact that for the first time in history an open source operating system has dominate market share.   It really is about time!


Flame on!
Feel free to start a debate in the comment section, but please try and put forth reasonable arguments.   Nobody benefits from name calling and bickering.    I'm sure lots of people will disagree with my predictions, if you have your own, call them out and tell us why you think they are right.

Value based home music recording

Value based credentials
Everyone who knows me knows that I'm obsessed with getting a good value. I probably look for good deals a little too much. My wife has said to me on multiple occasions that she would really appreciate if I would stop telling everyone how much I like to get good deals. To put it bluntly, I'm a bit cheap sometimes, although I do prefer "value based" or possessing "Yankee ingenuity". The whole idea of being cheap isn't consistent with the rest of my life, since I have been known to make a fair bit of money and have spent quite a bit of it as well. The long and short of things is that I like to get the best value I can from the money I spend and I spend a fair amount of time researching things to find the right items to buy. I should probably write a separate blog just on my neurotic need to find good value, but let's just establish for now my credentials for finding good deals.  

The challenge
About 9 months ago I started the process of recording a set of songs. I'm not a particularly good musician and the other guys in our casual band are definitely not professionals either, but having set out to do something I always try to do as good a job as I can with the time and resources available.

The band
Our band is not a traditional band. We write and perform parodies for our school auction. The idea started with my good friend Jeremy and fellow school parent David who decided, off the cuff, to do a few songs during the school live auction just to be a bit different and entertaining. I joined the band the following year and it grew from there. Each fall we get together for 3 or 4 nights and write some new lyrics to classic kitschy tunes and figure out how to hack our way through them. Our lack of great talent is part of the charm of the whole affair. (Or so we like to think) I mostly think that Jeremy's jovial personality and overwhelmingly funny and positive stage presence is the only thing that makes it work. So the raw talent that we are working with is limited, but there is a fair amount of enthusiasm and positive energy that we want to capture in a musical album of the songs we have performed over the last few years.

Equipment
Knowing that I need to take the reigns and figure out how to record our band, I set out to find out how the process works and what equipment I would need to pull it off.     I'm a drummer and I have always had the additional task of managing the PA and mixing equipment.    At this point, having done it for many years, I'm pretty good at setting up microphones, speakers and mixing consoles.   I have run audio boards for other fairly decent bands in the past so I know the live performance side of things well.     Recording is a different beast and requires different tools.    Here are the things that I knew I needed:

• All the standard live equipment: Instruments, amps and microphones
• A hardware device that takes analog input from the microphones and other instruments and translates it into separate recordings of each instrument.  Each one of those is called a "track".
• A computer program that can take those separate tracks and mixes them together to create a master track that is the finished product.    Ideally this software would work similar to Photoshop and can be used to touch up and improve the tracks after they are recorded.

Microphones
Microphones are a bit like your favorite food or wine.   There is something for everyone and everyone will have a different opinion about what quality, price and sound they prefer.    Microphones are a cheap as a few dollars and can run into the thousands for bespoke models for professional studios.    If you already have a band, you probably already have some microphones.    The singer will probably care the most about what their mic sounds like.    If you don't have any mics I recommend looking at Shure, AKG, Sennheiser and Audix.    For this project I used Audix OM-5 microphones.   They are an inexpensive durable mic made for live giging.   If you are only going to do studio recording you can probably find something less durable and better sounding.   I also used a Shure wx-20 headset mic for myself (the drummer).    There are specialty mics you can get for guitar amps, bass drums, snare drums, harmonicas and every other type of instrument.     One mic I really wish I had was something like this AMT 15G for recording acoustic guitar.    A regular mic on a boom stand can work great for guitars, but the guitarist needs to have the discipline to not move around during the recording session.    A mic attached to the guitar solves those problems.    I don't really like the sound of the mics and pickups that are inside the guitar body.

Drums
Recording drums is an art unto itself.   Ideally you would mic every drum head and each cymbal separately.   Practically speaking most people just can't get there.    If you want to record an acoustic drum set properly you will need to spend more than I did on my entire recording set up.    I got around this problem by using a Roland TD-30.   High end Roland drums are pretty amazing to play, they really reproduce the feel of an acoustic set.   In live settings they are definitely a different type of instrument and don't feel quite as edgy and real, but for a recording session, they are really hard to beat, and the benefits for recording are amazing.   Everything you play will be recorded properly and if you record using MIDI you can go back and tweak the track losslessly.   You can also tailor the drum sounds to the music you are playing.    I'm a pretty weak drummer, but the TD-30 makes me sound better than I am.  (Don't worry, I'm not planning on quitting my day job)   I didn't buy the TD-30 for recording, I bought it so that I can play in my house everyday without driving my family crazy.    If you don't have an electronic drum set to record with I recommend two things.   1)  Isolate the drummer as much as possible from the rest of the band so that the drums don't bleed into the rest of the band's mics.   2) Record with a single overhead mic to capture the entire sound in one track.    You will have to play with the height to get it to sound right and the acoustics of the space you are in will color the sound, so you might have to figure out how to soften the room by hanging blankets or carpet on the walls.

Multitrack recording hardware
Your microphones have to plug into something.    At a live gig you plug all the sources into one mixing console and have someone adjust the levels of each source to sound reasonably good.     Mixing a live gig is very subjective and results vary wildly.   The nature of the live gig is that people expect to hear a more raw sound and it is often played very loudly which gives music a very different feel.   When making a recording, listeners tend to expect a more polished sound and they tend to be more critical of the recording since they listen to it over and over again.    In order to get a more polished sound it is necessary to get a clean recording of each instrument and save each one separately so that they can be mixed together into a final form later.     Traditionally, artists recorded onto tape.   8 track tapes from the 70s were are a commercialization of technology that artists used to record their individual tracks.    Today most people digitize into a computer and store each track as a file on disk.    There are lots of different products that do this.   Some are fully integrated and look like a mixing console, but also record each track to an internal hard drive.   A lot of products use FireWire to record to any computer often times a Macintosh, since they have supported FireWire for a long time.    Recent products use USB to interface with the computer and are compatible with any computer you have.  (PC, Mac or Linux)     I wanted to find a USB product since FireWire is hard to find these days and is almost impossible to get on a laptop.    The product I chose is the M-Audio C600.    The C600 can record 6 tracks simultaneously and comes from a company that has a high regard for audio quality.   There are lots of similar products out there from Tascam, Behringer, Roland, Alesis, Mackie, Motu and others.    The products are improving and changing quickly, so as you are reading this, chances are that there is or will be better products out already.   The product class is called a "USB audio interface".   Here are the basics of what you should look for:

• Number of simultaneous tracks that can be recorded.
• 1 is the minimum you can buy.   You can buy multiple single track recorders and attach them all to a computer to get multiple tracks.   This can work, but can also be a source of frustration if your computer can't handle the multiple hardware sources properly.  The cheapest ones you can buy are on Ebay for $12 each. 
• Figure out how many you tracks will need at one time.   The bigger your band the more you will need.    I figured that I would need at least 6 to mic a 4 piece band.  
• Figure out how many you can afford.   This technology is getting cheaper all the time, but there is still a pretty significant price jump from the 2 up to 6 and again up to 8 or 12 inputs.
• Overall quality.   There is a big difference in the mic pre-amps between the high and low end of the market.    Read the reviews carefully before you buy a bargain priced unit.
• Phantom power.    Transducer mics require power to operate and your audio interface should supply it.   You can plug your mics into a mixer and then key off the audio inserts on the mixer, but that represents one more step for noise to creep in.
• Rack mount or desktop.   The higher end units and especially the 8-12 input units are usually rack mount and integrate into a "professional" studio environment.    
• Headphone and audio sends.      Your audio interface will be used to provide monitoring for your band as you are recording.    The audio sends can go to speakers or you can use the headphones as a monitor.    Headphones work best since you don't want to have extra sound sources in your recording room.

Computer
Almost any reasonable computer should work.    I used a Thinkpad X220 laptop.    You will only need about 30 gigs of free disk space for most uses.    Obviously if you choose a Firewire solution, you need to have a computer that has a Firewire interface.    If you have a USB audio interface, you can use pretty much anything.    Almost everything is USB 2.0 still.   USB 3.0 is coming but isn't really necessary for most uses.

Software
Software is where you can really spend or really save, the type of software you need is call a DAW or "Digital Audio Workstation".   The 800 pound gorilla in the space is Protools.   Protools is what the pros really use and has pretty much every feature you could imagine.    Protools is expensive and has a really onerous DRM system that involves physical keys that you have to plug into your computer, it also retails for about $700 for just the base software.   There are "starter" versions of Protools and I received one with my M-Audio C600, but it turns out that the way they restrict the usage is to limit the number of simultaneous tracks you can record to two.    With only two simultaneous tracks you really can't record a band, you can only record yourself.   I'm an open source software guy and have given away nearly everything I have ever written, so I looked around to see what I could use.    I tried to use a product called Audacity.   It seemed to be the best of the open source tools, but it had a real problem interfacing with my hardware.    Most of the audio interfaces work best with a Windows driver interface called ASIO, and I couldn't make ASIO work with Audacity.    There are many DAW's out there, but I really wanted something that had a bit more polish to it and supported ASIO.   During my research I came across Reaper.   As it turns out Reaper was everything I was looking for.   It is a nicely polished product that works very reliably, supports ASIO and has very attractive pricing.    Reaper is free to try and nags you if you don't buy it.    Once you decide to buy it there are two options.   It's only $60 if you are a small time user like me, and $225 if you are a big time commercial outfit.    Most of us will fit into the $60 tier and that is a tremendous value in the DAW space.     One of the biggest benefits of Reaper, besides that fact that it works fast and reliably, is that it supports VST plugins.   VST plugins are the same things that make Protools such a powerful platform.    The DAW software is not just a recording, mixing and editing solution, it is also a platform for adding lots and lots of 3rd party effects to make your tracks sound amazing.

The recording process
Every band will do it different, but here is what I experienced.   We only had about 8 weeks from the start of the process to being completely done and we were trying to record more than 10 songs.    A real band would take longer than that just for one song in most cases.     On many of the songs we recorded the drums and rhythm guitar in one session and added the rest of the instruments as overlays later.    In some songs we tried to do all the instruments together and then went back and rerecorded some parts that we didn't like.    The complications that we ran into were many.     Sometimes one or two of our band members were not available, so we had to try and make progress.    Sometimes we would all get together and waste a bunch of time just trying to get ready.    We had problems with bleed when we tried to record multiple acoustic guitars.     Most of us didn't like our performance and wanted to go back and try it again and again.    The most important thing we did, was to keep making progress and keep making recordings.    Eventually we were able to get about 12 songs recorded in various states of completeness.   

After the band had recorded the major bits, I went back and overlayed backup vocal tracks, bongos and other percussions and started playing around with effects to try and make us sound better.

VST Plugins
After you capture your recording you will start mixing and editing.   A short while later you may  realize that you want to add effects to voices and instruments to enhance the sound.    VST plugins is where to turn for amazing effects.    Start searching the internet for VST plugins and you will find hundreds of choices for free and commercial plugins that do virtually everything.    Want to get that Autotune sound to be just like T-Pain, you can get the original commercial version and plug it right in.    Some really great free plugins are gVST and BetaBugs. 

Final cut
You may be listening to your songs on headphones and love the sound only to be surprised later on that it doesn't sound nearly as good in your car or on your iPod.    It turns out that one of the secrets of good audio engineers is that they know how to mix a song so that it sounds pretty good on all audio systems, even though it won't sound perfect on any of them.    One of the secrets is that they compress the music substantially.    Compression removes dynamic range from a song, but it also makes it sound louder overall in general.   It raises the volume of the quite parts and lowers the volume of the loud parts.   Compression is very controversial in the music world, but I will tell you that if you don't do it, people will probably tell you that your song doesn't sound right, since they will probably be listening to your song with poor quality speakers.    If your audience is all audiophiles or people with high quality headphones then leave out the compression.

Total cost and equipment list

Here is the list of equipment and approximate price for them:

• $0  A laptop or desktop computer (assumed that you already owe this) 
• $250  M-Audio C600
• $360  3 -Audix OM-5 microphone (3 x $120)
• $80    Shure WX20 headset microphone
• $200  Mic stands, cables, beer holders, etc...
• $60    Reaper DAW software

Total cost, just under $1000 USD.   The experience of recording and producing music, priceless.


The result
I know it's not terribly good, but I'm proud of the result and we had a great time making the music.   I hope these tracks are received in the spirit in which they were recorded.   I'll just post a few.

All of these songs were written for our nursery school, a COOP school in Menlo Park called the Menlo Atherton Cooperative Nursery School.   

Stop. Get Right Down and work it out
    This is a song about two toddlers battling it out on the nursery school playground.   It was inspired by a parents meeting in which we all learned the best way to try to resolve conflicts between two  children on the playground.

Can't you see what that COOP means to me
     This is the last song we wrote and is a look back at how the school had effected our lives and how special the time was.     Listening to it now brings back fond memories of all the friends I met and the great experiences helping to teach the kids in the school.

Wow, Flickr is giving away 1TB of storage for personal photo hosting

I just wrote an article for our Zetta.net blog: Cloud Storage Explained: How Can Yahoo Give Flickr Users 1TB Free?

Let me know what you think...

It's the end of <blink>!

Firefox was the last holdout of the blink tag, but has finally removed the evil.     For sentimental reasons I will show you a method for bringing it back.    Also see my other post on the creation of the blink tag.

CSS to the rescue!    Did I mention that I helped with the creation of the first version of CSS?      CSS has gotten quite a bit fancier in recent years and now supports animations, which allow you to do many very fancy tricks, one of the easiest is blinking:

<style>
.keyframeBlink {
animation-name: onOff;
animation-duration: .8s;
animation-iteration-count: infinite;
animation-direction: alternate;
}
@keyframes onOff {
    from { opacity: 1; }
    to {opacity: 0; }
}
</style>

<div class="keyframeBlink">This should blink with css animations, images work too:  </div>

This is an example of text and an image blinking!

You can also get fancier and change the rate of blinking and the style of the fade.
Here is a 1 sec duration and the use of:  "animation-timing-function: ease-in-out"

Fancy blinking example, notice the nice fade in and out!

Now kids don't try this at home, people will make fun of you. I am definitely not encouraging anyone to do this and if you ask me I will disavow any knowledge of this post.    In fact, just working on this post, which I disavow working on, is giving me a headache.



NOTES:   On chrome you will need to make the CSS a bit uglier since they are not yet supporting the new naming scheme.   They are using the -webkit- prefix on all the animation keywords.    I would expect that to change soon so that the -webkit- prefix will be unnecessary.   Here is a great site for an explanation:  http://www.css3files.com/animation/

Your style element will look like this to make it work cross browser for now:

<style>
.keyframeBlink {
-webkit-animation-name: onOff;
-webkit-animation-duration: .8s;
-webkit-animation-iteration-count: infinite;

animation-name: onOff;
animation-duration: .8s;
animation-iteration-count: infinite;
}

@keyframes onOff {
    from { opacity: 1; }
    to {opacity: 0; }
}

@-webkit-keyframes onOff {
    from { opacity: 1; }
    to {opacity: 0; }
}
</style>

Why blocking 3rd party cookies could be a bad thing

3rd party Cookies are used to facilitate targeted advertising and can be used to track your browsing across multiple Web sites.

I am not comfortable with being tracked across the web by Cookies, and Cookies were specifically designed to try and prevent this sort of behavior, so why would I be against disabling the major mechanism that is used for web tracking?   

The answer is pretty simple:

• The evil you know is better than the one you don't.
• This is probably a race we can't win.

The evil you know is better than the one you don't

Right now we know that advertising companies use 3rd party cookies to track behaviors and serve custom ads.  The companies that use these methods are well known to us and use similar methods.  They almost exclusively use 3rd party cookies.    With this knowledge it is very easy to disable tracking:  Go into your browser preferences and disable 3rd party cookies.    For those who care there is a simple and reliable method to disable tracking.   

Now lets suppose that someone decides to turn off 3rd party cookies by default in most browsers.   What do you think will happen?    Will the advertising industry just close up shop and move on to other businesses?    Will web tracking just cease to happen?    Of course not!   Advertising companies will move to other technical methods for user tracking.    They exist already, they are just a bit less convenient than 3rd party cookies to use.     The new situation is one that will be much harder for an individual user to disable tracking.  


This is probably a race we can't win

Think of this battle similar to the challenges of DRM.   Companies spend lots of time and money to design a super secure system to protect their digital content and within a week or two the DRM is broken and the content is all over the internet.    DRM is typically broken by small groups of unpaid hackers who do it for a hobby and they win just about every time.     Now think about how difficult it will be to stop a multi-billion dollar industry from winning a similar war.     If you are a browser company how are you going to stop well funded companies from coming up with technical ways to serve targeted ads?     There are already multiple ways to do tracking without 3rd party cookies and countless others could be created if enough effort is put into it.


What can we do?

My suggestion is to handle this politically not technically.   Keep working on Do-Not-Track and other mechanisms.    Keep working on legislative methods to restrict the policies.    Those are wars that we can win and those policies can have real teeth to them.     One of the benefits of the 3rd party cookie tracking techniques is that it is really easy to see who is doing the tracking.    If we make the societal choice to prohibit tracking then it will be easy to track down offenders.   


User tracking is a hot issue and should continue to be discussed and debated.    It is important to keep in mind that targeted ads are paying for almost all of the things that are on the Internet today.    If we make a wholesale decision to stop serving ads are we all prepared to start paying for all of the services that we are getting for free now?      Perhaps many people would agree to some level of targeted advertising if they saw the reasons and cost benefits clearly enough.     What I don't think will be useful is to start a technical war that disables the current working methods for disabling tracking and replaces them with poorly understood and less visible techniques.

Google "Play All" music service initial review

Today (5/15/2013) Google announced the "Play All" subscription music service.   Play All adds unlimited streaming to the Google Play Music store and app.     I have previously tried to use Google Play to store and playback music and had some problems, but I was willing to give it another go to try out this new Spotify killer service.

My history with cloud music
I am currently a Spotify premium subscriber, a Pandora premium subscriber and I have tried to use Google music and Amazon Music with limited success.    I have a very large MP3 collection that I started back in 1998 when I started ripping my own CD collection to my computer.   I have been using network based streaming music players and portable MP3 players for about 15 years.    Putting music in the cloud makes my life easier because it is available wherever I go.

The Good
The terms and signup are good: 30 days free to try it out and only $7.99 if you sign up this month.    I already had a Play account so registration was super simple.   I simply agreed to a new TOS and approved a zero dollar Google Wallet transaction.   (The same process as buying an app on the Google app store)


I started to browse around and found basically the same music selection as Spotify.    I did find some surprises when I searched on Led Zepplin and The Beatles.   In both cases I got music while Spotify is missing those artists.    As it turns out Google doesn't have them either, but music that was previously uploaded to Google Music is automatically searchable and playable.     This turns out the be the best feature of Google Music in my opinion.    All of the subscription music services are missing important artists like The Eagles, Led Zepplin, The Beatles and many others.    With Google music you can upload your own collection, up to 20,000 songs and fill in the holes that are left by the licensing problems that the music world is experiencing.    With this feature I can finally have a fully complete library with cloud convenience.

The mobile app is not as polished as Spotify is yet, but the basic features are there.   Search is good, playback is fast and I can sync and store my playlists for offline playback.    So no major problems for me, and my experience with Google apps on an Android phone has been generally very good.   I don't know how good the app is on iOS.

There is a radio feature that seems to work well.   I haven't had a chance to listen to it over a long period of time so I don't know if the algorithm is good, but time will tell.

As an experiment I started converting some of my large Spotify playlists to Google Music.  There is no automation for this, so it involves looking at the Spotify list in one window and then searching for the same song on Google Music and then adding it to a playlist.    The process is long and boring, but it did confirm that Google Music has most of the same catalog as Spotify.    There were some missing songs like "Big Yellow Taxi" by the Counting Crows but nothing really surprising or significant.   Converting playlists did bring up an important switching cost.   I have lots of playlists on Spotify and moving them will be very hard.

The Bad
I already mentioned that converting my own playlists is hard, but the other feature that I love about Spotify is that many of the people I know are also on Spotify and they share playlists with me.     Without a playlist converter I'm losing one of the best features of a socially connected music service.    This is a huge problem for Google.   If they can solve playlist sharing I can coexist peacefully with Spotify, without it I need to choose between them and the logical choice is to go where my friends are.

The other feature I'm not satisfied with is the social aspects of Google music.   Right now I can only have a private playlist or a public playlist.    I want a playlist shared with just my friends, not the whole world.   I also want collaborative playlists.

Update: I found another nit.   Google play doesn't currently work with 3rd party music players like Sonos or Squeezebox.    Since I use these in my home it's a big problem for me.    I hope that Google releases a supported API so that the service is accessible.   

Conclusion
Great pricing, the ability to upload my own music and integrate with the rest of the streaming catalog.    Overall the feature set is good to very good.     The downside to all this is your friends are probably on Spotify now and you will lose your social features if they don't move over.     The industry is still pretty young so Google has a reasonable chance to gain big market share here.    Spotify still loses money every year, so they might not even be around in a few years.  I think I will keep my subscription for a few months and see how it goes, but right now I'm not canceling Spotify.   (And if anyone from Google reads this, please create an app or plugin that converts Spotify playlists!)

The reasoning behind Web Cookies

I get a  fair number of questions about cookies from individuals and the press.    I thought I would try and explain some of the motivation and history behind Web cookies as well as some of the design behind them.      Feel free to post more questions and I will try and expand this article with more details.

See my recent post on 3rd party cookies as well!


Motivation
HTTP is the networking language behind your browser.    HTTP, or Hyper Text Transport Protocol, was designed and introduced as part of the WWW project that started with Tim Berners-Lee at CERN and was expanded upon by several universities, corporations and private individuals.    The WWW and HTTP are open standards which are published and given into the public trust in order to foster interoperability and to create products that can become ubiquitous.

One of the problems faced in the early years of the web was how to create websites that had "memory" for individual users.    The uses of "memory" on a website are many: shopping carts for shopping, personalized content, logging in, and many other interactive features require memory.    The problem in 1994 was a lack of mechanisms to identify a user individually.   HTTP was designed to be fast and efficient and part of its design was to connect to a website, grab a document and then disconnect.   This freed up the website to serve other customers, but it also meant that there was no concept of a session.     Without a session, each time a user clicked to move to a different page they would become just another random user with no way to associate them with an action they had done just moments ago.     This is a bit like talking to someone with Alzheimer disease.    Each interaction would result in having to introduce yourself again, and again, and again.

History
By the summer 1994 the idea of adding some form of "memory" to HTTP had been kicking around the WWW design groups for a while.   I'm not sure exactly how long but I would guess for 1 or 2 years.    There were some interesting proposals, but one of the popular ones that kept coming up was to add a unique identifier to every web browser so that a web site could individually identify each user use that to build a session.    I was very much against this concept because the unique identifier could be used to track a user at every website.     Without another proposal the idea of adding sessions or memory remained dormant.
 
Sometime around July of 1994 I went to a meeting to talk about a shopping server that another group at Netscape was working on.     They needed to build shopping cart functionality into the server, but they didn't have a good way to make it work with the existing technology.    They explained what they wanted to do and I had to shake my head and agree with them and tell them that they could not do it in any reasonable way with what existed.     I promised to think about the problem and get back to them.    At some point during the next week the general concept of Web Cookies formed in my head.    The idea of allowing a single web site to send a session identifier to the browser that would get sent back only to the server appealed to me and prevented cross site tracking.    I wanted to create something that had more utility so that we could do a lot more than just shopping carts, so I extended the concept of the session identifier into a general payload that would get sent back to the server.   

Design
With a rough sketch in my head I worked on the general design of cookies.    The goal was to create a session identifier and general "memory" mechanism for websites that didn't allow for cross site tracking.    I discussed the design with other members of the engineering team and got useful feedback, especially with regard to denial of service attracts and other security concerns.   John Giannandrea was especially helpful with the design.   The end result was the cookie specification that still defines 95% of what cookies do today.     A Web server can return information to the browser that the browser should give back to the server each time it contacts the server in the future.     Within the Web Cookie, a server can embed a random session identifier, a username, a shopping cart, or anything it wants as long as it doesn't try to send something too big or set too many cookies.    There are also restrictions on how cookies can be transferred, either over secure connections or not and if the cookie should be erased when the user closes the browser or reboots their computer.  


The name

I had heard the term "magic cookie" from an operating systems course from college.     The term has a somewhat similar meaning to the way Web Cookies worked and I liked the term "cookies" for aesthetic reasons.     Cookies was the first thing I came up with and the name stuck.   


Usefulness

We released the Netscape browser in the fall of 1994 and within a year it was the most popular browser in the world.     We also released the Cookie specification so that other browsers could implement it and so that web sites would know how to use it.    With most of the world using a browser that supported Web Cookies, web sites started using cookies for many different things and in ways that we could not have predicted.      Most of those uses were fantastic, some of them were concerning.

By 1996 and 1997 the web had grown dramatically and was a big business.    Most websites offered their content for free and were advertising supported.     Advertisers were looking for ways to increase the effectiveness of their ads and they looked to Web Cookies to help them with that.    


An Unforeseen Problem
A problem that I missed during the Cookie design phase was an interaction between cookies and embedded content within a webpage.    Webpages start as a single HTML document on one server.   That one HTML document contains references to other resources that are loaded to display the site that the user sees.    Images, videos, more text, and plug-ins are all references within an HTML document and any of those resources can be loaded from anywhere in the world.     This referencing technique is one of the things that make the Web so amazingly powerful.     When Web Cookies are combined with embedded references that point to other websites they are called "3rd party cookies" and they represent a new way in which users can be tracked across multiple web sites.

Advertising uses
Big ad companies, in particular DoubleClick, started using 3rd party cookies to track a browser uniquely across all of the sites that used DoubleClick to serve ads.     The use of the tracking wasn't to actually identify the name of a user, but to make sure they didn't see the same ad every time or to track the number of unique users that saw a particular ad.     Eventually the ads were customized to reflect the browsing habits of the unique identifiers.    If a browser had looked at kayaks and racquetball racquets in the past then they were given ads for sporting goods and kayaks and racquets.    

Around 1996 ad tracking via Cookies became a hot topic.    Users were upset with the practice and asking for change.     Tracking across websites was certainly not what cookies were designed to do, they were designed with the opposite intention, but what could be done at that point to fix the problem?        One of the solutions we came up with was to add controls to give each user control over what cookies to accept and from whom.    The big question at the time was whether or not to disable 3rd party cookies completely or not.     The decision had wide ranging effects since advertising was paying for most of the web and disabling 3rd party cookies would also disable many legitimate uses for cookies on embedded resources.    

An uncomfortable decision
In the end the decision to disable 3rd party cookies or keep them on was left to me.     I agonized about the decision for weeks and in the end I chose to keep them.   My logic was two fold:

Any company that had the ability to track users across a large section of the web would need to be a large publicly visible company.   Cookies could be seen by users so a tracking company can't hide from the public.    In this way the public has a natural feedback mechanism to constrain those that would seek to track them.

If 3rd party cookies were disabled ad companies would use another mechanism to accomplish the same thing, and that mechanism would not have the same level of visibility and control as cookies.   We would be trading out one problem for another.

Today, I still believe that it was the correct decision.     Governments have an ability to regulate the collection of data by large visible companies and has shown a willingness to do so.    The public has a responsibility to keep pressure on both the companies that have the ability to track users and governments to enact reasonable privacy regulations and enforce them.      Most importantly, there are other mechanisms that can replace Web cookies for tracking if they are universally disabled, and those mechanisms would be much harder to observe and disable.

Durability
The design of Cookies have remained fairly unchanged for the last 19 years.   The biggest changes have been to how users can view and control their cookies.     Many people have proposed alternatives, but none have caught on.     Cookies are not perfect, but they have certainly proved good enough and much of the functionality of the web depends on them.   

Summary
Web Cookies find themselves in the midst of a very controversial area, and have gained a level of notoriety because of it.    Even though they were designed to protect privacy they have been used in ways that are sometimes infringing upon it.    Many efforts have been taken to protect the privacy of Web users and Cookie controls have been put into the hands of every user.    The future of Cookies and user privacy is surely to continue as an interesting news item for a very long time and rightfully so.    The nature of the advertising business is to collect as much information as it possibly can, so the public needs to push back when it goes too far. 

A short history of the "about:" URL

Here is a fairly recent musing on "about:" URLs that I did just a few months ago.   

Due to my lack of a real blog this may be the first time anyone will read it. 

http://www.montulli.org/lou/about_urls

In case you are wondering, about URL's are used by your web browser to do things that URL's really should not do.

There is more info on about URLs on Wikipedia of course:  http://en.wikipedia.org/wiki/About_URI_scheme