A drummers guide to using ABC Notation for sheet music

I have been passionate about drumming for a while now, and I have the need to create a fair amount of sheet music.   I originally learned to write it out by hand, but my hand written notation looks worse than my hand writing, which is pretty terrible.     I went on the internet to find a program to make sheet music, thinking that it would be as easy as typing, but I found very few programs and almost none that would support drummers well.

I eventually found ABC notation, a markup language that can be used to create sheet music. The advantage of ABC is that it has a standard text notation that can be easily shared and turned into beautiful sheet music with multiple programs.   This post will address the specific needs of drummers, but is not a full primer on ABC.

ABC notation makes it really easy to create music for melodic instruments, but the support for percussive instruments has been added in over time and is not as elegant.     Below is a simple example of a simple tune for a melodic instrument:

This is the ABC Notation:

X:1
K:G
GABc dedB|dedB dedB|c2ec B2dB|c2A2 A2BA|

This is the Rendering from the ABC Notation:

The notes in the ABC notation map very directly to the scale in the sheet music.    For this type of sheet music, ABC Notation could not be simpler.  For drummers, however, there is a different story.

In the beginning, ABC Notation was designed for folk music on melodic instruments and could not handle the special needs of drummers at all.   Over the years, ABC has grown into a very complete and detailed language with the ability to do complex markup and create very detailed output.  In order to create good looking drum kit sheet music we will need to use several of the advanced markup features.

For all of my examples I will be using ABC2SVG as my rendering engine.   ABC2PS should work exactly the same.     The advantage of ABC2SVG is perfectly scaled sheet music that renders really well in the browser.   (My images here are converted to GIFs for convenience)

For rendering and to experiment with the examples try the ABC2SVG online rendering tool.

First the simple stuff:

Use a percussion staff.  This command changes the staff symbol to a percussion symbol.

• K:C clef=perc

Use the following notes:

• Hi-Hat  - g
• Ride - f
• Crash - A'
• Kick - F
• Snare - c
• Rack Tom 1 - d
• Rack Tom 2 - e
• Floor Tom - G

Here is an example of all those together.

X:1
K:C clef=perc
g f A' F c d e G

Ok, the notes are in the right places, but they don't look like the right notes and the staff lines are not normal for drum kit notation.     Cymbals should use an "X" instead of a dot and drummers prefer all the staff lines up or down.

Cymbals with an X

To change the cymbals to use an "x" it gets a bit ugly.  Some rendering programs allow you to put a "^" in front of your notes to make it an "x".   ABC2SVG requires you to remap it with the "%%map" command.     In ABC2SVG we also need to define what the new symbol will look like, since we could potentially map it to any shape.   Using "^" and the remap command makes the markup compatible with multiple ABC Notation tools.

Here is the boilerplate code that you can use.   It goes near the top of the file and only needs to be defined once.    So set it and forget it, but use it every time.

We will also need to add a map command for the other cymbals and even the Hi-hat when played with the foot pedal.

This is the full boilerplate

%%beginsvg
 <defs>
 <use id="VoidWithX" xlink:href="#acc2"/>
 </defs>s
%%endsvg
%%map drum ^g    heads=VoidWithX   print=g   % Hi-Hat
%%map drum ^A'   heads=VoidWithX   print=A'  % Crash
%%map drum ^f    heads=VoidWithX   print=f   % Ride
%%map drum ^c    heads=VoidWithX   print=c   % Cross Stick
%%map drum ^d,   heads=VoidWithX   print=d,  % Foot Splash

You will also need to add a "voicemap" command before your actual notes, but after the "K:" line.

%%voicemap drum

Here is a full example:

X:1
M:4/4
%%beginsvg
 <defs>
 <use id="VoidWithX" xlink:href="#acc2"/>
 </defs>
%%endsvg
%%map drum ^g heads=VoidWithX print=g  % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A'  % Crash
%%map drum ^f heads=VoidWithX print=f  % Ride
%%map drum ^c heads=VoidWithX print=c  % Cross Stick
%%map drum ^d, heads=VoidWithX print=d,  % Foot Splash
K:C clef=perc
%%voicemap drum
^g ^f ^A' F c d e G

OK, getting better.    But it still has a ways to go.    Let's try to add a whole line of Hi-hats and link the flags as 8th notes.   ABC makes it easy to group notes; just put them together with no spaces.    When you add a space, it will break the bar and start creating a new one.   Also notice in this example I added "V: hands, stem=up".   This forces the note stems to go up.

X:1
M:4/4
%%beginsvg
 <defs>
 <use id="VoidWithX" xlink:href="#acc2"/>
 </defs>
%%endsvg
%%map drum ^g heads=VoidWithX print=g  % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A'  % Crash
%%map drum ^f heads=VoidWithX print=f  % Ride
%%map drum ^c heads=VoidWithX print=c  % Cross Stick
%%map drum ^d, heads=VoidWithX print=d,  % Foot Splash
K:C clef=perc
%%voicemap drum
V: hands, stem=up
^g^g ^g^g ^g^g ^g^g

This is starting to look like drum notation!  Now let's add kick and snare.  The secret to adding notes that vertically line up is to create a chord.   We do that by putting all the notes together in brackets "[" and "]".  The same rule about spaces and bar lines still applies.  We are going to add this to the example:

 [F^g]^g^g^g [c^g]^g^g^g [F^g]^g^g^g [c^g]^g^g^g

Here is the full example:

X:1
M:4/4
%%beginsvg
 <defs>
 <use id="VoidWithX" xlink:href="#acc2"/>
 </defs>
%%endsvg
%%map drum ^g heads=VoidWithX print=g  % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A'  % Crash
%%map drum ^f heads=VoidWithX print=f  % Ride
%%map drum ^c heads=VoidWithX print=c  % Cross Stick
%%map drum ^d, heads=VoidWithX print=d,  % Foot Splash
K:C clef=perc
%%voicemap drum
V: hands, stem=up
[F^g]^g [c^g]^g [F^g]^g [c^g]^g

Wow, look at that, a basic rock beat.    OK, now for the more complicated notes.  

Snare Ghost Notes

To properly enclose the snare note in parens to create a ghost note, we will use the text decoration feature.   "%deco"

The following header lines define what and where the "(." and ")." will render as.

%%deco (. 0 a 5 1 1 "@-8,-3("
%%deco ). 0 a 5 1 1 "@4,-3)"

Once we define how it will render, we add "!(.!!).!" to the front of the snare note "c" to create a ghosted snare note.  Here is the same example as above but with ghost notes:

X:1
M:4/4
%%deco (. 0 a 5 1 1 "@-8,-3("
%%deco ). 0 a 5 1 1 "@4,-3)"
%%beginsvg
 <defs>
 <use id="VoidWithX" xlink:href="#acc2"/>
 </defs>
%%endsvg
%%map drum ^g heads=VoidWithX print=g  % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A'  % Crash
%%map drum ^f heads=VoidWithX print=f  % Ride
%%map drum ^c heads=VoidWithX print=c  % Cross Stick
%%map drum ^d, heads=VoidWithX print=d,  % Foot Splash
K:C clef=perc
%%voicemap drum
V: hands, stem=up
[F^g]^g [!(.!!).!c^g]^g [F^g]^g [!(.!!).!c^g]^g

Accented notes

Accented notes are relatively easy, but they do have a gotcha.   To accent a note simply add "!accent!" in front of the note.   The gotcha is that the accent can only be added on the outside of a chord, so it must be outside of the "[" and "]" markers.

Accented Snare note example:   !accent!c
Accented Snare and Hi-hat:    !accent![c^g]

Open Hi-hat & Close Hi-hat

Open and close on the hi-hat works similar to the accent using "!open!" and "!plus!" keywords and has the same gotcha.   Make sure to put it outside of the chord.

Open Hi-hat example:    !open!^g
Close Hi-hat example:     !plus![c^g]

Snare Cross Stick (X-Stick)

We defined the cross stick to look like a cymbal above in the boiler plate.  So now just use "^c" instead of "c" and it will render as an "x".

Snare X-Stick example:   ^c

Here is a full example using accents, open, close and x-stick.  Note that I added a command to make the accents and other ornaments always be on top.  "%%ornament up"

X:1
M:4/4
%%ornament up
%%deco (. 0 a 5 1 1 "@-8,-3("
%%deco ). 0 a 5 1 1 "@4,-3)"
%%beginsvg
<defs>
<use id="VoidWithX" xlink:href="#acc2"/>
</defs>
%%endsvg
%%map drum ^g heads=VoidWithX print=g  % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A'  % Crash
%%map drum ^f heads=VoidWithX print=f  % Ride
%%map drum ^c heads=VoidWithX print=c  % Cross Stick
%%map drum ^d, heads=VoidWithX print=d,  % Foot Splash
K:C clef=perc
%%voicemap drum
V: hands, stem=up
!plus![F^g]^g !accent![c^g]^g [F^g]^g !accent![^c^g]!open!^g

Flams

To render a flam we will use a grace note called an acciaccatura.   To add the grace note to the snare we add a "{/c}" in front of the snare note.   This can't be added within a chord, so move it to the outside.

Example of just a flam:                      "{/c}c"

Example of flam and Hi-hat together:   "{/c}[c^g]"

Other items of note

• To make all the beams be even and flat along the top use: "%%flatbeams".
• Title uses the "T:" command:   "T: This is a title"
• Author uses the "C:" command:  "C: Test Author"
• Comment uses the "P:" command:  "P: Test Comment"
• You can add text above a note using the "^" command in front of a note:   "^text here^"
• Rest is: "z".  
• Hidden rest is: "x"

Now let's create a legend or key for all of our notes.   I'm using some hidden rests "x" to space out the notes so the descriptive text fits.

X:6
M:4/4
T: This is a title line
C: Test Author
%%flatbeams 1
%%ornament up
%%deco (. 0 a 5 1 1 "@-8,-3("
%%deco ). 0 a 5 1 1 "@4,-3)"
%%beginsvg
<defs>
<use id="VoidWithX" xlink:href="#acc2"/>
</defs>s
%%endsvg
%%map drum ^g heads=VoidWithX print=g % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A' % Crash
%%map drum ^f heads=VoidWithX print=f % Ride
%%map drum ^c heads=VoidWithX print=c % Cross Stick
%%map drum ^d, heads=VoidWithX print=d, % Foot Splash
%%staves (Stickings Hands)
P: Test Comment
%%musicspace 20px
K:C clef=perc
V:Stickings
x8 x8 x8 x8 x8 x8 x8 x8 ||
V:Hands stem=up
%%voicemap drum
"^Hi-Hat"^g x"^Open"!open!^g x "^Close"!plus!^g xx "^Accent"!accent!^g x "^Crash"^A' x "^Ride"^f x
"^Snare"c xx "^Accent"!accent!c x "^Cross"^c xx "^Ghost"!(.!!).!c xx "^Flam"{/c}c xx "^Rack Tom"d xx "^Floor Tom"a, xxxx
"^Kick"F xx "^Hi-Hat w/ foot"^d, xxxx "^Kick & Hi-Hat"[F^d,] ||

Now we have a full set of notes.    But how do we create real music?   Music is a combination of notes and rests of varying lengths.    ABC Notation allows any note to have any duration and is expressed by adding a number at the end of the note.    So far we have been dealing with 8th notes since they are the default.   

• To double the duration of the note add "2"
• To Quadruple add "4"
• To Half the duration of a note add "/2"
• To make the note 1.5 times a long (a dotted note) add "3/2"

If note "c" is an 8th note  

• "c2" would be a quarter note 
• "c4" would be a half note
• "c8" would be a whole  note.
• "c/2" is a 16th note
• "c/4" is a 32nd note
• "c3/2" is a dotted 8th note

Notes don't have to default to be 8th notes.   We can define the normal note to any length using the "L:" command.

• L:1/16  -- make the default note a 16th note
• L:1/32  -- make the default note a 32nd note

I generally use 16th notes or 32nd notes as the default since the patterns I write out are naturally 1/16 or 1/32 note patterns.

Here is an example of a 16th note pattern.   It has 8th notes on the hi-hat and ghost notes in between to make it a 16th note pattern.    This is a very common ghosted rock groove.

X:1
M:4/4
T:
L:1/16
%%flatbeams 1
%%ornament up
%%topspace 0px
%%deco (. 0 a 5 1 1 "@-8,-3("
%%deco ). 0 a 5 1 1 "@4,-3)"
%%beginsvg
 <defs>
 <use id="VoidWithX" xlink:href="#acc2"/>
 </defs>s
%%endsvg
%%map drum ^g heads=VoidWithX print=g  % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A'  % Crash
%%map drum ^f heads=VoidWithX print=f  % Ride
%%map drum ^c heads=VoidWithX print=c  % Cross Stick
%%map drum ^d, heads=VoidWithX print=d,  % Foot Splash
K:C clef=perc
V:Hands stem=up
%%voicemap drum
!plus![^g2F2]^g2 !accent![c2^g2]^g!(.!!).!c [^gF]!(.!!).!c^g2 !accent![c2^g2]!open!^g2 |

Triplets

If you are going to play jazz, blues or even many rock songs, you are going to need to use triplets.   The logic behind triplets is a little hard to follow sometimes.  Here is the documentation from the ABC Notation Wiki.   

Triplets are specified using the syntax "(p:q:r". Means 'put p notes into the time of q for the next r notes'.    In our case we will use "3 notes into the time of 2 or 4".

• To add 3 snares in a triplet:   "(3:2:3ccc"

If you are in 16th notes and want to do 8th notes triplets, put them into the time of 4.

• To add 3 snares in an 8th note triplet: "(3:4:3ccc"

X:6
M:4/4
T: Purdie Shuffle
L:1/16
%%flatbeams 1
%%ornament up
%%deco (. 0 a 5 1 1 "@-8,-3("
%%deco ). 0 a 5 1 1 "@4,-3)"
%%beginsvg
 <defs>
<use id="VoidWithX" xlink:href="#acc2"/>
 </defs>
%%endsvg
%%map drum ^g heads=VoidWithX print=g % Hi-Hat
%%map drum ^A' heads=VoidWithX print=A' % Crash
%%map drum ^f heads=VoidWithX print=f % Ride
%%map drum ^c heads=VoidWithX print=c % Cross Stick
%%map drum ^d, heads=VoidWithX print=d, % Foot Splash
K:C clef=perc
V:Hands stem=up
%%voicemap drum
(3:4:3[^g2F2]!(.!!).!c2^g2 \
(3:4:3^g2!(.!!).!c2[^g2F2] \
(3:4:3!accent![c2^g2]!(.!!).!c2^g2 \
(3:4:3^g2!(.!!).!c2[^g2F2] |

Hand stems up with kick stems down

Some drummers prefer to have the kick stems go down and the snares and hi-hats go up.   ABC notation supports this with the notion of voices.  You can have two voices on the same staff line, one for the hands and one for the feet.

Use the "%%staves" command to define two voices "Hands Feet".

• %%staves (Hands Feet)

In your ABC notation, separate out your foot part from the rest of the notes and move them to a separate line that follows the line:

• V:Feet stem=down

Example:  (The total note duration on each line needs to add to 8.   So doubling the duration of each kick drum makes for 4 quarter notes adding to 8)

%%staves (Hands Feet)
V:Hands stem=up
%%voicemap drum
^g^g [c^g]^g ^g^g [c^g]^g |
V:Feet stem=down
%%voicemap drum
F2 F2 F2 F2 |

Stickings

Stickings can be added above the staff using a separate voice and comments on the note.   Comments are just quoted strings in front of a note.  For example  "R"c   puts an 'R' above the snare.  In this example I'm using hidden rests in a separate voice to add the stickings.   It seems neater that way.

%%staves (Stickings Hands)
K:C clef=perc
V:Stickings

"R"x""x"R"x""x "L"x""x"L"x""x "R"x"R"x"L"x"L"x "R"x"R"x"L"x"L"x |
V:Hands stem=up
%%voicemap drum
c2c2 c2c2 cccc cccc |

The End

I wish I could say that it was easy to write drum kit sheet music using ABC.  To paraphrase Winston Churchill:   ABC Notation is the worst way to write drum sheet music, except for all the others.     The best I can say is that it works, and once you get the hang of it, it goes pretty fast, and the end result is very pretty.    In the mean time, I'm developing a tool to build it all automatically with a WYSIWYG UI.   More on that in a future post.

Online Drum Practice Log

I recently attended a weekend drum camp with Thomas Lang. where I learned a lot of things about drumming in general, but one thing very specific:  Practice time is everything.   This is also a motto of the amazing drum teacher Mike Johnston, who recently posted this simple truth video about practice.

My own modest experience as a struggling drum student certainly mirrors Thomas' and Mike's teachings and underscores the importance of persistent and consistent practice.   I think that I have been practicing a good amount, but I have been relying on my own memory for the number of hours practiced rather than a real log of my time.     At camp, Thomas spent a bunch of time teaching the importance of a detailed log and gave some great advice on what should be in the log and why.   I will attempt to summarize the teachings and then give details about an online tool for keeping a drum log practice log.     If you are not a drummer, keep reading, this tool could easily be adapted as a practice log for any instrument, or any skill that could benefit from a practice log.

What should be in a practice log?

• Duration - This one seems pretty obvious.   How much time was spent practicing.   I choose to log each time that I sit down at my kit rather than summarizing at the end of the day.
• Mood - Did the practice feel great or awful?  Some days are better than others, and for many people the time of day effects their mood and productivity.    By keeping track of the mood or general productivity in a detailed format, the log can help identify the best times or days to practice in the future.
• Time of day - The time of day is recorded so that the mood can be correlated.     With enough data it should be possible to see what time of day is the most productive time to practice so that future practice sessions can be optimized.
• Date - This allows for the correlation of mood as well as knowing how many minutes are practiced each day.
• What - What was practiced in general.    If consistent categories are used, the tool can tell how many minutes are practiced on a given category over time.
• BPM - "Beats Per Minute" track the speed of a given exercise so that progress can be tracked on rudiments, grooves and other exercises where building speed matters.   This may not apply to all practice sessions but is useful for many.  
• More Details - A place to record specific details about what was practiced, problems, successes and any other notes that might be useful

There is a lot of data to record and I am notoriously bad about keeping paper records, so I immediately started to look for digital versions of a practice log.

My wish list for a digital practice log are:

• All of the data points listed above should be easy to capture
• The tool must work on a variety of platforms including: PC, MAC, Android and IOS
• The data needs to be centrally hosted or synced among all the application instances so that I can record practice time from anywhere.    It definitely cannot be trapped on my phone only.
• The data should be exportable so that I can put it into a spreadsheet or another program to get better insight into the data.
• It shouldn't cost too much. :)

I was not able to find anything that met all of my criteria, so I started to investigate how difficult it would be to make my own.      One of my goals is that the data should be exportable to a spreadsheet, so I started my process by looking at my favorite collaborative spreadsheet program, Google Sheets. Google Sheets is part of Google Docs/Drive and is similar to MS Excel.     I knew that if I was able to get all the data into Google Sheets I would be able to meet most of my criteria since it is a cross platform centrally hosted program.   I knew that I would not be happy manually entering the data, so I started looking for ways to make data entry easy.       It turns out that Google Sheets has  a programming API that allows for mobile or web apps to query and enter data, but as I started working on a web app I realized that it was going to be a fairly involved effort.    I started researching some more and found that Google sheets has a very cool "forms" mode that I had never used before.    The more I played with it, the more I realized that it would be the simplest and most effective method to build the log tool with.  

Try out my solution

I ended up writing a new solution myself that works well for me.  Would you like to try it out?     The data entry form is here and the back end spread sheet is here. 

When you go to the data entry form you will notice that you do not need to enter the date or time of day, because that information will be automatically captured at the time you submit the form.    If you need to enter data for a prior day, simply log the information with the form and then go to the spreadsheet and change the date.    

The spreadsheet provides all the tools needed to graph practice time over days, months and years.    And since the data is in a standard spreadsheet program it is easy to export to any other program.    Since Google sheets is cross platform, the form and spreadsheet can be accessed from any device, mostly through a web browser, but also through the Google drive mobile app.

Set it up yourself

If you want to set this up for yourself, follow these steps.

• Make sure that you already have a Google Drive account set up.
• Go to the shared spreadsheet here
• In the "File" menu choose "Make a Copy..."
• Give the file a new name and save it.   It will now be in your Google Dive account.
• Select the sample data by clicking and dragging and use "edit->Delete Values" to clear it.
• Under the "form" menu, choose "Go to Live Form".   This is your entry sheet and can be used from any web browser on a computer, phone or tablet.    Save a bookmark to it.
• Start practicing and log each practice session using the "Live Form" to make it easy to use.
• Visit the spreadsheet to see all your sessions and graphs of your practice over time
• You may also want to edit the input form to customize it for your name.   Go to the "form" menu in the spreadsheet and choose "edit form".   Change "Music Practice Log" to "YOUR NAME's Practice Log"

Within the spreadsheet are several tabs along the bottom that contain summary graphs for different time periods.    They are made using pivot tables and summarize the data based on the following:

• Last 60 Days - Time summarized by day
• Days   (All days for the entire practice log)
• Months - Time summarized by month
• Years - Time summarized by year

All of the graphs are embedable in web pages in case you want to put up a page that shows your progress.

Advanced notes

If you are interested in using this tool for anything other than a drum log, or to expand the functionality, this section is for you.     You can easily change the names or headings of the data collected by using the form editor in Google Sheets.   Open the spreadsheet and look under the "form" menu.   The form editor will allow you to change the name of the items being collected and will allow you to add new items to the form.    If new items are added, a new column will be created in the spreadsheet automatically.    If you know how to use Excel or another similar spreadsheet program, you will be able to add to or analyze the data in almost any way you can imagine.    The benefit to the data being in a spreadsheet is that it is easy to manipulate and graph.

Comments

Feel free to post a comment below if you have any problems getting the spreadsheet to work, or if you have other questions about changing the spreadsheet to work for your needs.   I will try to reply as soon as I can.

Moto 360 Review

As I write this I have had the Moto 360 watch for 4 days, so I am not yet a seasoned pro, but I'm now starting to get used to it.

(Update after 2 weeks is below)

What is it?

In case you have not seen it, the Moto 360 is a smart watch that is paired with your phone.   It is far from the first smart watch, but it is the first watch that I thought was good enough to buy.   A big part of "good enough" is that it looks like a real watch, being round, and it looks great.

How I use it

Mostly I use it as a second display for my phone.   It is much easier to see a notification on my watch than it is to pull out my phone and see what just came in.   The most useful alerts are text messages.   I have also gotten a bunch of Facebook messages and call notifications.       There is a setting to ignore messages from certain apps that I have configured for games, LinkedIn and WiFi notifications.  
You can reject or answer calls from the watch.   If you answer the call and are on a Bluetooth headset you can start talking right away.   If your phone is needed, you still have to pull it out of your pocket. I have a Bluetooth speaker and mic in my ski helmet, so I think this will be a very useful feature while skiing.

Google Now provides alerts for many of my interests.    It will display the SF Giants score as it is happening (configurable for spoilers), tell me when I need to leave to drive to an appointment and other interesting news and event.

Other great uses

I have set up my watch to alert me if I leave my phone somewhere.     Whenever I walk more than 20-30 feet from my phone, my watch alerts me to it.    I'm using Wear Aware.  

My phone unlocks automatically when I am near it.   I set up my watch as a trusted device so that I can skip the lock screen or fingerprint reader.    The phone locks automatically if I leave it somewhere.     Try this or this.

Not so great

Technically, you can initiate actions using the watch.   You can say "Call Fred Phelps" and it should make a call.   Or you can make Google searches or a bunch of other things.   All of this must be done by speaking to the watch, which I find aggravating and error prone.  I find it to be much slower than just pulling out my phone.    I may try to make the call initiation work when I'm skiing, but I'm not really optimistic that it will work in a noisy outdoor environment.

Battery Life

This was the one area I was most concerned about, but fortunately I have not had a problem.    I charge it every night on the included wireless charging stand and I usually have between 40-50% battery life left at the end of the evening.    This is in the default configuration.    There is an "ambient mode" which keeps the screen on all the time that I would prefer to use, but I heard it will drain the battery to fast so I have not tried it yet.    I will try it sometime in the future and report back.

Summary

Overall I impressed and am liking the watch even more than I thought I would.    The display is beautiful, the UI works and the battery life is fine.    I have lots of great mechanical watches, so I'm not sure if the 360 can take over completely, but I should know in about a month if it is a "must keep" device or just a toy.

2 Week Update

I have had the watch for two weeks now and I have found the battery life to be way better than expected.   I was able to put the watch into "ambient" mode and never run out of battery.    Ambient mode makes the watch behave much more like a real watch, without any delay in updates or glance capability.    Lots of people are asking about the watch and really like it.

Another housing bubble in the Bay Area

I moved to the Bay Area in 1994.

Source: Zillow.com

I bought my first house in 1996.

I sold that house in 2000 for more than twice what I paid for it, and moved up to Truckee, CA.

I moved back to the Bay Area in 2006 during the last housing bubble and felt that the market was overvalued, so I decided to rent.    I found a high end rental house and have been renting for the last 7 years.  I have saved roughly $600K in housing costs during that time vs owning the same house.

Record growth this year
The market this year is red hot.    Year over year home prices are up about 31%.    Across the street from me a developer built a 4,000 sqft house on a 8,000 sqft lot and sold it for $4 million.    The most expensive home on the street previously had been only $2.5 million.   The owner of the house I have been leasing noticed and has wisely decided to cash in on the current market, so now I need to find a new place to live.   I have been making the tour of houses in Menlo Park for the last two months and I'm exasperated.  Very average houses are going for well above asking price and land values are crazy.    Complete tear downs are selling for close to $2 million.  How does this make sense?    I created a spread sheet to try and analyze the market and see if it makes any sense.

According to Zillow the average house in Menlo Park is currently $1.46 million dollars, that is up from $1.04 million in May of 2011.   Keep in mind that these are averages and include condos and distressed properties and houses literally right next to highway 101.    The real truth is that you can't buy a "normal" house for less than $1.9 million and that "normal" house will be about 1,600 sqft.   The average cost per square foot is at or above $1000 sq/ft.   Any house that you might consider "fancy" is 3 million or more.

The recommended guidelines for housing costs are about 28% of after tax income.   In California, people are used to spending more income on housing so doubling the figure is about 55%.   I built a spread sheet that calculates the full cost of owning a home and will calculate the income required for 28%, 55% and 75% of income going towards housing.   The spreadsheet takes into account the mortgage interest tax deduction and the property tax deductions.

If I run the numbers for $1.46 million and $1.9 million here are incomes necessary to have 28%, 55% and 75% of income going to housing

Necessary income for $1.4 Million house

28.00%	55.00%	75.00%
$688,198.86	$322,812.16	$221,766.95

Necessary income for $1.9 Million house

28.00%	55.00%	75.00%
$885,066.96	$421,091.92	$292,782.77

Now lets look at the average income in Menlo Park.    Average income can be misleading since it also includes people who are not home owners.   The published average household income for Menlo Park is about $110,000.    If I dig a little deeper and take just the most affluent area of Menlo Park we can find that the household income for West Menlo is about $220,000.  (The data source is city-data.com)

So we can see that at $1.4 million the average household would be spending 75% of income on housing.   The real reality of $1.9 million would find that average household would have to spend 94.5% of their income on housing.

I think that it is pretty clear from the data that the average person cannot afford the average house right now.    But what does that mean?

What is causing the spike in prices
If the average person living in an area could not afford to re-buy the house that they are living in, what does that mean?   

• The neighborhood could have a recent influx of new buyers that are more wealthy than the existing base.  (This is usually called gentrification)
• Supply is constrained and this is temporarily pushing up prices.
• Interest rates are about to rise and people are buying before they go up.
• The Facebook effect?
• The tulip craze is back.

Is it really Gentrification?
Gentrification is a possibility, but is it really happening?  How many people really make $885K a year?     Let's me more reasonable, lets allow for 55% of income for affordability, so the real question is how many people really make over $420K.    According to the IRS in 2010, 800,000 people in the US make $500K or more.   How many of them live in California?   I can't find a direct source for this, but we can estimate it.    According to the IRS in 2009 about 20% of Millionaires live in California.   If I generously estimate that 75% of them live in the Bay Area and not LA then we have about 15% of all the Millionaires living in the Bay Area.    That's 120,000 people (very approximately) in the Bay Area with household income above $500,000 per year.     So I guess this is  a possibility.  

Similar Neighborhoods in the penninsula:  (number of households)

• Menlo Park: (12,000)
• Palo Alto:   (25,000)
• Atherton:   (2,500)
• Los Altos:  (10,000)
• Hillsborough: (3,600)
• Los Gatos:  (12,000)
• Woodside:   (2,000)
• Total: (67,100)  peninsula only

Constrained Supply
This does seem to be a serious problem.   There just are not enough houses for all the people moving here.    Additionally, Prop 13 encourages people to stay in their homes and never move, so there is very little turn over in the market.   High prices combined with taxes hurt turnover as well.    If you are living in a house that is suddenly worth 2 million dollars it is very hard to sell because taxes will take half the proceeds.   Then you need someplace else to live and everything costs more than a million dollars.

Basic economics says that prices will go up if there is more demand than supply.    But how high and why now?    Supply has been a problem for the last 20 years.   Why is the market up 30% in one year?

Interest rates on the rise
I think the recent spike may be related to interest rates.   As interest rates climb the monthly cost of a home goes up substantially.    I did all my affordability calculations at 4% interest.   Here is a table of annual cost for a 1.9 million dollar house at different interest rates: 

4%	5%	6%	7%
$146,000	$160,000	$173,000	$188,000

That equates to more than a thousand dollars a month in difference for each percentage point.    At 7% you would spend $42,000 more per year on the same house than at 4%.    If interest rates return to normal levels of 7% over the next few years, it should cool the market considerably.


Facebook?
There is more than 75 Billion dollars worth of new value in the Menlo Park area.   The vast majority of that is owned by a small number of people, but there are at least a few hundred new Millionaires in the last year.

Tulip craze?
Are we in another temporary bubble?    I really don't know, but I hope so.    Nobody benefits from sky high real estate prices, except the real estate industry itself.    Surrounding neighborhoods are also experiencing record setting prices.    The Zillow Palo Alto index is over $1.7 Million and even towns like Sunnyvale and Mountain View, which are traditionally far less expensive, are approaching $1 million.     Local companies will not be able to attract talent to the area if people can't find an affordable place to live.   Keep in mind that the national average for a house is $158,300.    I would never expect the Bay Area market to be average, but I would expect it to be closer to 5x the norm rather than 10x.

What am I going to do?
I love living in the Bay Area and I love Menlo Park most of all.    The combination of an amazing climate, proximity to fantastic natural wonders like Tahoe, Marin and Napa and last but certainly not least the amazing people who live here.     If I could convince myself that the market won't fall again in the near future I would be happy to buy an overpriced home here.   Unfortunately I'm convinced of the opposite.   I think the market will correct downward at least 25% in the future, but I don't know how long it will take.

Some people called me crazy back in 2006 when I predicted a correction.   They said that real estate in California never goes down and even bet me that it wouldn't.  I won money on those bets, but I was polite enough not to collect since the folks who said it lost a lot in the down turn.    I was complacent in 2009-2010 and didn't buy then, so now I'm in a difficult spot, my mistake.

Barring a happy accident falling into my lap with an under priced gem, I will be renting again.    The rental market is just recently starting to correct closer to ownership prices, but it is still a much better value than owning, it also carries no risk of a massive loss if the market corrects back to normal levels again.

Here is my spreadsheet that has the calculations

Please leave a comment with your favorite theory and prediction for housing prices in the future.

Why great software artists steal rather than borrow

"Good artists borrow, great artists steal", Pablo Piccasso

It's not a perfect analogy for software, but it sure gets your attention.   A different way to say this is "Don't borrow the ideas of other products, steal the building blocks themselves instead of rebuilding them."

Software teams are able to get new products and updates to market much faster than they were 10 and 20 years ago.   Much of the productivity gains have come from "stolen" building blocks that have been made open source or integrated into the development stacks that we use.

When I helped build Netscape Navigator nearly 20 years ago we had to start at a very low level and build everything up.    The networking libraries started with TCP sockets and even then there was not a consistent API.   The UI was built up from fonts, pixels, primitive menu API's and drawing commands.    When we wanted to display a GIF or JPEG we wrote all the code ourselves.   Everything was built using plain old "C" and we had to build all of our support libraries from scratch.    Simple list structures, hashing functions, time functions, threads, better memory allocation to deal with 16 bit machines.  We eventually built a cross platform support infrastructure called the Netscape Portable Runtime (NSPR) that encapsulated all the support code into one library and let anyone build tools on top of it.   We also gave it away, so anyone could use it.     Thousands of people over the last 20 years have been building similar libraries and making them available through open source for others to build upon.

Today when we build an application we can code the UI in HTML and let the browser figure out how to render it.   We can build the business logic in a high level scripting language that handles threading, memory management and a whole host of other complex low level problems for us.   We have access to thousands of open source libraries that can do incredibly complex tasks.    On top of all that we can even host our applications in the cloud and let someone else build, maintain and monitor all our hardware for us.

New code is still hard
UI's are easier, infrastructure is easier, memory management is easier, but coding is still pretty much the same.     We have lots of new buzzwords for software development these days:   Scrum, Agile, Extreme, Waterfall.   It all sounds very exciting, but I can tell you that team programming is still pretty much the same.    I have been doing versions of Agile programming for more than 20 years and we have now codified the methodology significantly, but the nature of the game has not changed very much.    New code is still hard.   It takes longer that we estimate and getting it bug free is very hard.   Keeping it bug free while extending it is even harder.

Don't build anything you don't have to
Every single thing you build will take about 10x the time in maintenance than the time it takes to build.   The things we build tend not to be as fully featured or as well thought out as a successful open source library and likely won't have half the features of the components you could have "stolen".   

So when do you decide to build something?
Deciding when to build something new versus using something that exists is really tricky.   One of the hardest parts is knowing what is out there.    There are literally tens of thousands of open source libraries that we could grab that do virtually anything.    If you want to make smart decisions about when to "steal" a component you need to educate yourself and get to know what is available.     It may seem like a lot of time spent researching and reading, but imagine how much time you are going to save by not having to build a major component for your development project.    And how much time you will have later on by not have to fix the bugs and add features because the component you found is already feature complete and stable.    Get on the internet and ask people if searching doesn't yield results.   

Spend all your time wisely
In a start-up, and in any company, but especially a start-up, you should spend 100% of your time building unique value.    Time spent reinventing the wheel does nothing to generate unique value.    Investors won't care that you built the coolest new version of crontab to control your back end processes, they care how many users you have and how much money you can make.  

Use your components wisely
Don't try to use components in ways they were not designed to be used.   First, you will likely find bugs that no one else has encountered because they have never used that component in that particular way.   Second, it probably won't scale and perform the way you expect.   

Don't try to second guess the design unless you are an expert
Understand and follow the best practices for each component.   You might not understand all the reasons for the best practices, but they are there.    If you want to use the Zend PHP framework you are going to have to use a model-view-controller paradigm.   You may think that it's too complicated for the simple website that you are trying to get done right now and you don't want to spend the time to learn how Zend's M-V-C model works, but you need to do it.    There are really good reasons why you should do these things, but they really don't manifest themselves until you start maintaining the code or decide to change the UI later in the product life cycle.   Or you might decide that SQL is way too complicated and design an intermediate language to sit on top of it.   If you think that's a good idea and your not a bona fide SQL and relational DB expert you are going to get in trouble.     SQL got complicated for a reason, trying to second guess 30+ years of iteration is very dangerous.

Don't use components that suck
I have a whole separate blog article on this one.  "For software development tools it's the ecosystem that matters most".   Make sure all the components and tools you use are bulletproof.    Time spent fixing someone else's code takes away from your product.

Give back
On a final note, I would like to encourage you to give back when you can.   All of the code that we "steal" comes from somewhere and wouldn't be there without the hard work and generosity of those who put it there.   Recognize that and give back to the open source community by contributing new libraries and enhancements and bug fixes.    I have open sourced almost all the code I have written and it has been a highly rewarding experience.

For software development tools it's the ecosystem that matters most

In this article I will be exploring the different types of software development tools and how to choose the right ones for your next start-up or project.

In my career I have had the pleasure of working with several successful software teams on a variety of projects.  I have also had the opportunity to mentor or review more than a hundred software development teams. I have written programs for Windows, Mac and Unix, Android and a bunch of Web delivered apps. At Netscape I was part of a small cross platform development team that wrote Netscape Navigator which at that time was the most used application in the world.

In this article I would like to concentrate on platforms and tools. Platforms and tools are one of the least understood or appreciated since they technically focused and most engineers don't get the opportunity to choose their tools.  Whenever a new software project is created we make important choices about what platform and tools we are going to build with.  I'm defining the platform to be the hardware and operating system and the tools to be everything else above the operating system.

Platform
Platform choices used to be pretty easy:  Windows, Mac or Unix. When the world was 90% Windows, most software developers knew right away that they would be using Windows as their platform.  Today we have many choices. Win, Mac, Linux, Android, iOS, Blackberry, Windows Mobile or HTML.  Platform choices are usually driven by business reasons, but there are still technical choices to be made.  For instance should you have a native windows or Mac app?  Web applications are preferable for most since they solve the distribution and update problem, but they are not able to integrate seamlessly with the computer and can't handle long running background tasks.   Music players are a great example of this.    Lots of music companies have cloud players that run in the browser, but most choose to do a native app to support offline play and to have a fancy native UI.  Mobile app companies also have an important choice about what platforms to support. Many mobile start-ups I see are just concentrating on their mobile apps, and not creating a web based experience. Unless they are a gaming company this seems extremely short sighted.  Most of us live and work in a heterogeneous environment, so a web based experience allows us to work with a service no matter where we are. GoodReads and Google Maps are good examples of this.

Tools
I find that tools are the most difficult choices that we make at the beginning of a project. There are many different classes of development tools. These are just a few:

• Software language:  C++, Java, PHP, Ruby, ...
• Framework: LAMP, Zend, Rails, Tomcat, ...
• Compilers & IDEs: G++, MSVC, Eclipse, XCode, ...
• Debuggers: GDB
• Databases: Oracle, MySQL, Postgres, Mongo, Cassandra
• Bug tracking: Bugzilla, Jira, FogBugz, ...
• Revision Control: SVN, Git, CVS, Perforce, ...
• Integration: Jenkins, Bamboo, ...
• Unit Testing: CppUnit, Google Test, JUnit
• Static Analysis: Lint, Klocwork, Coverity, ...
• Hosting: Equinix, Amazon, RackSpace, Google, ...

Many software tools are chosen on the basis of what was used on the last project.    Familiarity is an important consideration, but should not be the only one.    If we simply go with what we know we are missing opportunities to choose a tool that has advanced or is simply better suited to the current project.  Here are some important criteria to consider:

• How suited is the tool to what I am building?
• Will the tool handle the scalability that I need? 
• Maturity
• Supporting tools
• Community

How suited is the tool to what I am building?
To a hammer the whole world looks like a nail. Perhaps you really need a wrench or a screwdriver?   If you are building a financial application, don't try to use a fancy new noSQL database.   Use a good old fashioned conservative SQL database.  If the type of app you are writing has a large portion of users using one tools set, you should probably choose that.   Why try and build an Android app with MSVC when everyone else does it with Eclipse? 

Scalability
If you are building a web application for a small group, performance probably doesn't matter.   If you are trying to be the next Facebook, you might want to think about how the tools you are using will scale up to millions of transactions and a big design team.  Facebook is experiencing this problem in spades with their choice of PHP as a language.  Their improvements to PHP are very impressive, but the effort could have been avoided by choosing a different language.  Scalability also applies to team programming.    Dynamically typed languages, such as PHP, Ruby and Python can be challenging for very large software projects since the functions are not explicitly typed.  This makes it hard to detect software problems at link time and requires engineers to document their code in ways that are not enforced by a compiler.

Maturity
Tool maturity is too often overlooked.  I think this one should be easy, but I see start-ups making the wrong choices over and over again.  If a tool isn't 99.999% stable, you probably should not use it.  All of your time should spent solving technical and business challenges.  If a start-up needs to spend time debugging their tools they are wasting valuable time and resources solving a problem that they didn't need to solve.    How should you evaluate maturity?   Past personal experience or a referral by someone you know and trust is best.   Otherwise, look to see who else is using it.    If you can't find a large list of other companies that are using the tool in a production environment, stay away.    Even the most promising tool can turn into a nightmare if you make it part of your product and it starts to crash or otherwise misbehave.  Building tools is hard, not just because they are hard to build, but because they are very hard to perfect. You need tools that work every time, not just most of the time.

Supporting tools
This is probably the most important tool trait. Every tool class listed above comes with an ecosystem of tools that support it.  With a database you need backup tools, profiling tools, monitoring tools, API libraries and documentation.  New tools often look promising until you look under the covers and find out that it is lacking basic support infrastructure.  Do you really want to find out after you go into production that your tool needs to be taken offline just to back it up?

Profiling and debugging tools are critically important and are often the last thing on a product road map.  If you have ever experienced a critical performance bottleneck and have been without a real profiling tool you will understand how important a profiler is.  How about finding a thread deadlock with a debugging tool that doesn't properly deal with threads?  Memory corruption detection without a dedicated memory debugger?  These are support tools that usually take many years if not decades to get right in a major tool class, but they are absolutely necessary.   

Community
Last but not least is the community of people that can support you for questions and problems.  The absence of a strong community should be a huge red flag. Poke around and ask questions, make sure that there is still active development of the tool that you are choosing and lots of active users.  Look for example code and places where people congregate to ask and answer questions.   You will save a lot of time if you can draw upon the community to figure out problems for you.  Open source libraries built for your tool can also make your job a lot easier.  Plugins to support your other tool choices and layer in great product features.

Looking at the whole stack
Now that you have a framework for evaluating individual tools, you now need to go up a level and look at your entire tool stack.  Almost all of the development tools you use will interact with each other so they need to be compatible.  Want to use a Java based messaging framework but you use C++ everywhere else?   Think again.  Love that C++ static analysis tool, but you use Python?  Sorry no dice.  Your choices on one tool effect every other tool you will use.   Make sure that you evaluate the whole stack before you commit to any one tool.  Otherwise your one choice will likely dictate and limit all your other tool choices.    

Current Experience Levels
Last but not least please be realistic about your team's experience relative to the tools you are choosing.  It might seem like a great idea to use Ruby on Rails for your next major web project, but if there is nobody on your team that has any experience with Ruby you are probably going to run into trouble.  If you already know PHP the benefits to moving to Ruby are probably overshadowed by all the mistakes that you will make learning a whole new framework.  Remember that the end product is more important than the language used to express it and the user isn't going to care if you are using the latest and coolest technology to pull it off.  Conversely, using a technology that very few people know, even if your team is very proficient with it, will limit your ability to hire new people in the future.  

Apply this same methodology to support libraries as well
All of the points above also apply to support libraries that you will use in your product.   Here are a few examples.

• Logging
• Messaging frameworks
• General Utility libraries such as boost for C++
• Caching and proxy libraries and servers
• XML & JSON libraries

Summary
Look at all your tools carefully before you start using them.   Make sure that they are all of the highest quality and maturity and that they come with the supporting infrastructure to handle the whole development process:    Coding - Testing - Debugging - Profiling - Monitoring
Doing this will save you and your team countless hours of frustration relative to less mature tools.

An HTML5 fancy user interface project

Sunburt graph

I recently finished a new project that uses HTML5 to display and graph file system data and I thought I would write a blog post about it.
 
History
HTML has come a long way from the early days when I was working on Lynx. The latest and greatest features are pushing graphics and user interfaces that are approaching native code levels of complexity and performance. I'm particularly fascinated by these features because they represent the realization of a vision that we had in the early days of Netscape where HTML could take over as a complete application programing interface.

WWW applications?
Using the Web just like a computer program seems pretty obvious these days, we use it that way on many if not most web sites, but in the beginning it was really just a document publishing platform. Hypertext, the basis for the web, was designed as a way to make documents more powerful by linking them together with references that could be accessed incredibly easily.  The idea of putting powerful application logic behind a hypertext interface was pretty far fetched in the early '90s.  I happened to have a need to do just that due to a need at my university. I came up with some ideas and put forth an idea to the other developers I knew that were working on the web. This idea soon sparked the addition of form elements to the web. Later while at Netscape, we took the next step by adding a scripting language, Javascript, to add even more client side capabilities and responsiveness.

Full GUI support
One of the visions we had at Netscape was to create a platform that had a fully complete and functional set of graphical elements so that other developers could create any application imaginable. The foundational elements are reasonably well known and are included in all modern operating systems and major graphics libraries.  The difficulty in delivering GUI primitives to the Web was figuring out a way to efficiently add them to HTML and to make them massively cross platform.    When I say "massively cross platform" I don't mean just Windows and Mac. We were thinking about operating systems, screen sizes, resolution, and a whole bunch of other things.  The web was designed from the ground up to try to be device neutral and that makes the implementation very hard.  CSS is one technology that supports multiple displays by providing different layout and styling for different displays as required.  Netscape as a company ran out of life before it was able to complete its roadmap for a full complement of GUI primitives, but it did make a good effort.  A technology called XUL (pronounced Zool) was introduced that allowed a large variety of GUI to be expressed, unfortunately, XUL didn't move forward fast enough and was essentially dead on arrival.     Simultaneously Microsoft smothered the market with Internet Explorer and tried its best to keep the Web from moving forward with HTML.

Tree graph

Enter HTML5
Fast forward several years from the Netscape era and the world has changed substantially.    Firefox, Chrome and a variety of Webkit based browsers have wrestled away the strangle hold that IE6 held on the market and began to make substantial improvements to the abilities of HTML.  HTML5 was soon born and with it a whole bunch of new graphical technologies that enable many new types of GUI applications.    SVG, advanced CSS, Canvas and Javascript now combine to create an ability to create very rich interfaces that are entirely driven within the Web browser.

My experiment  
A while back I came across a group that was creating a set of tools that harnessed the new power of HTML5.  The project is called D3 for Data Driven Documents.  I saw what D3 could do graphically and wanted to try it out and do something personally useful with the technology.   Since my current company Zetta.net is in the cloud storage space I decided to write a tool to visualize the size of the data on a computer hard drive.  One of the difficult issues with a tool to analyze disk space is that it needs access to all of the data on a drive and that data is sensitive and private.  I needed to figure out a way to scan the data and not violate a user's privacy. Standard HTML applications run in the cloud and couldn't do something like this without sending data to a server, but the new world of HTML5 can make everything happen locally in the browser.

Some complications
At this point in my project I had an idea of what I wanted to do, and a fancy new graphical library for graphing, but I still needed a way to collect the disk space data to be graphed.  Javascript has chosen to stay away from local file access to simplify the security model, but Java long ago added the ability to ask the user for local disk access.  I ended up writing a Java beans integrated module that spawns a Java application to scan the local disk and return the data to Javascript running in the browser for analysis.  The beauty of the Java/Javascript integration is that none of the data ever leaves the local computer that it is running on. The downside is that Java is often broken on user machines, or is disabled due to security concerns.  I hope at some point Javascript figures out a workable security model to allow some forms of local disk access.

Tree map graph

The finished project
Please check out the finished project if you are interested. I have made a video demo and a walk through to explain the use and features.   There is also sample data to play around with that skips the scanning process.  The code has been published as open source with a BSD license.    This project just might come in handy for you the next time you run out of disk space and need to know what is using it all up.

http://wheresmydiskspace.com/