Nigel McNie

Nigel McNie http://nigel.mcnie.name/ Website and blog of Nigel McNie, entrepreneur and founder of Shoptime (prior: Mahara and GeSHi). Nigel writes about webapps and being remarkable. en Wed, 25 Apr 2012 15:25 NZST Wed, 25 Apr 2012 15:25 NZST nigel@mcnie.name Veedub 5 Nigel McNie http://nigel.mcnie.name/static/images/nigel.jpg http://nigel.mcnie.name/ Why Do We Remember? http://nigel.mcnie.name/blog/why-do-we-remember <p>I don't think the dead wish to be sung.</p> <p>In fact, the dead don't wish for anything. When they were alive, they would have wished for things. Now, they are memories. Soon, they will be history. Later, maybe archeology.</p> <p>If they were anything like us, I know that there's one thing they wouldn't have wished for. They wouldn't have wanted to die for their country. Nobody is born wanting to die for their country. Were you?</p> <p>No, what happens is that we convince them that it's a good idea. That if they don't, something worse might happen. Of course, that wasn't a lie, but it <em>was</em> a shame. A damn, <em>damn</em> shame.</p> <p>Funerals are not for the dead. They don't watch us. They can't hear us. They're <em>about</em> them. They are for the living. We grieve, we remember.</p> <p>Time moves on, and our grief passes. ANZAC day is a funeral without grief. We just remember. But what do we remember? We remember that they died for us. Do we remember that we sent them?</p> <p>It sounds harsh, doesn't it. It wasn't our fault! We <em>wouldn't</em> have sent them, if there wasn't another way. And it was our ancestors who sent them, not us! How can we be responsible for something we didn't do?</p> <p>I fully agree. Our ancestors did the best they could with what they knew. If we knew what they did, and were in their situation, we would have done the same thing.</p> <p>We're also not responsible for their actions. The line of reasoning that concludes this is sheer madness. It's playing the victimisation card. "They did it, we're responsible" adds <em>nothing</em> to the conversation. It just clouds our judgement.</p> <p>However, we are responsible for <em>our</em> actions. And I think you would agree with me that you wouldn't want "sent others to die" against your name without something as powerful as "saved many more lives from death and despair" alongside it.</p> <p>In fact, you weren't born wanting those charges to your name at all, were you? They didn't want to die for their country. You don't want to have to send them. Or be them.</p> <p>That might sound obvious, but I wonder how many people have truly reflected on what it means. When we remember them, we do think "gosh, it would be awful to have been one of them". Some of us even ponder the bad luck of the ones who had to send them. But spare a thought for yourself, too. It's blind luck that you weren't one of them. You should do your best to make sure it never happens again.</p> <p>We can only control ourselves. So this is what I'm going to do. I am <em>never</em> going to become that man, the one who attempts genocide based on a lie, who wants to take the freedom of others. If I see someone heading in that direction, and I have the ability to influence or control them, I will do whatever I can to stop them. And I will attempt to influence those around me to do these things too.</p> <p>That's what ANZAC day means for me. It's awful that they died. It's terrible that we sent them. It will never happen because of me.</p> <p>How about you?</p> The 2012 Global Atheist Convention - A Guide to the Perplexed http://nigel.mcnie.name/blog/the-2012-global-atheist-convention-a-guide-to-the-perplexed <p>"Why would you want to go to a convention about <em>that</em>?"</p> <p>I've been hearing this question a lot recently. The event was the <a href="http://atheistconvention.org.au/">2012 Global Atheist Convention</a> and I'm an atheist, so one would think the question has a simple answer. However, it's clear that many people (perhaps including you?) are genuinely baffled as to why someone would care enough to go, and what on earth we'd talk about while we were there.</p> <p>This is my attempt to answer the questions I've received. Now that it's over, my answers are based on what actually happened. I've tried to keep them as factual as possible, reserving my opinions for the footnotes, so skip those if you don't want an argument :)</p> <div class="section" id="why-do-you-care-so-much-about-not-believing"> <h1>"Why do you care so much about not believing?"</h1> <img align="right" alt="Why so serious?" class="align-right" src="http://stuff.nigel.mcnie.name/protesters-atheistcon.jpg" /> <div style="float: right; clear: right; font-size: smaller; margin-top: 5px; width: 300px; text-align: center;">Some might say us caring about not believing is simply an equal and opposed reaction to these guys. Why so serious?</div><p>This was a common question. Many people didn't see what there is to be passionate about. After all, how can you feel so strongly about what you don't believe in?</p> <p>As it turns out, we do have something we believe. The convention tagline was "A Celebration of Reason". If you believe that the ability of humans to reason is a great thing, you would have been right at home <a class="footnote-reference" href="#id9" id="id1">[1]</a>. If nothing else, it was great to spend time with people who you <em>know</em> use their minds rationally, and that you <em>know</em> care about doing so <a class="footnote-reference" href="#id10" id="id2">[2]</a>.</p> <p>We didn't just gather for the camaraderie. The speaker lineup was so impressive that it had the attendees labelling the event "the Atheist Woodstock". I won't list them, as I'm assuming that a list of speaker names won't mean much to you - the only one you have probably heard of is Richard Dawkins. But for those of us who are "actively atheist", the lineup really was stellar.</p> <p>So, good company and good content - two fine reasons to go to any convention. But what issues were on the attendees' minds? What did we talk about? What, in other words, had made us currently "care so much about not believing?"</p> <div class="section" id="secularism"> <h2>Secularism</h2> <p>Loosely described as "the separation of church and state", <a href="http://en.wikipedia.org/wiki/Secularism">Secularism</a> was topical. "The separation of church and state" might sound a little confusing, so here's a simple explanation: It's the idea that the government of a country should not do anything that favours one religion over another. Note that it's <em>not</em> that the country should be free from religion - merely that anything the <em>government</em> does should not take sides.</p> <p>This idea is supported by many, including many religious moderates. It's a "live and let live" policy towards people's religious beliefs, and I thoroughly support it. You could be Christian, your neighbours could be Muslim, and a secular country would mean you could both practice your faiths without fear that the government will take sides with your taxpayer dollars.</p> <p>Secularism was discussed for two reasons. First, it's been noted for a while now that the first country in the world to have secularist principles embedded in its founding documents - USA - is highly religious. Various organisations fight a constant battle on such fronts as keeping prayer out of the classroom, or the ten commandments from being hung in government buildings <a class="footnote-reference" href="#id11" id="id3">[3]</a>.</p> <p>Secondly, as the convention was in Melbourne, the school chaplaincy programme in Australia came in for a lot of criticism. The government allows each school to include half an hour per week of religious instruction, which may be carried out by any "well meaning volunteer" - i.e., not a teacher. It's "opt-out" - and children who are opted out by their parents end up sitting in hallways or in front of the principals office, not understanding why they're missing out. It was disappointing to attendees, given the Australian constitution includes <a href="http://en.wikipedia.org/wiki/Section_116_of_the_Constitution_of_Australia">wording that suggests Australia is meant to be secular</a>. An aggravating factor is that Juila Gillard, an openly atheist prime minister, has apparently been bending over backwards to appease the religious by <em>expanding</em> the chaplaincy programme <a class="footnote-reference" href="#id12" id="id4">[4]</a>.</p> </div> <div class="section" id="morality"> <h2>Morality</h2> <img align="right" alt="Busy times at the convention" class="align-right" src="http://stuff.nigel.mcnie.name/busy-atheistcon.jpg" /> <div style="float: right; clear: right; font-size: smaller; margin-top: 5px; width: 300px; text-align: center;">With around 4,000 attendees, the place was full of happy, smiling heathens.</div><p>A common refrain from the religious is that "without god, how would you know what is right and wrong?". This question was hardly discussed by the attendees, and as we all seemed to get on like a house on fire without raping and pillaging, you would think the religious might be forced to conclude that they don't have as strong a question as they thought. A dry remark - "I know I haven't killed anyone for weeks" - by one of the speakers was the only coverage I noted of this question <a class="footnote-reference" href="#id13" id="id5">[5]</a>.</p> <p>However, morality was a large theme. For many atheists, morality is largely tied to the wellbeing (and suffering) of life. As such, the point was well made that women have even more reason to be enraged at religion than men, given the way they're treated by religions. In Islam in particular, women are still treated dreadfully, if you define morality the way atheists do.</p> <p>The Catholic Church came in for a heavy pasting over its position on homosexuality, its contribution to the spread of AIDS in Africa, attempts to lower the age at which children are Confirmed, and of course the disgusting scandal that is the child rapists in their midst, largely immune to the rule of law <a class="footnote-reference" href="#id14" id="id6">[6]</a>.</p> <p>Heavy criticism was also levelled at liberals, who were accused of failing to stand up for human rights and free speech. Ayaan Hirsi Ali alleged that through a combination of political correctness, guilt and a romanticised view of Islam, liberals were failing to help Christians and other minorities in the Arab world, who are badly suffering. I suspect fear is involved too. Islam is the most dangerous religion to be an enemy of these days, as they can and do use violence to get their way. Richard Dawkins offered a powerful response for anyone threatened by them. "I am backing down because you have threatened me, but do not by any means think that I respect you. On the contrary, I despise you!".</p> </div> <div class="section" id="something-from-nothing"> <h2>Something From Nothing</h2> <p>A smaller, but significant, theme was the rebuttal of the classic religious argument "you can't get something from nothing", used to justify the existence of god. I wouldn't do the science justice trying to describe it to you, but it turns out that our traditional notion of "nothing" doesn't appear to exist. Even in a total vacuum, particles are popping in and out of existence all the time, and it turns out that "nothing" actually weighs something! <a href="http://www.amazon.com/A-Universe-from-Nothing-ebook/dp/B004T4KQJS">Here's some reading material</a> if this idea interests you.</p> </div> <div class="section" id="why-people-believe"> <h2>Why People Believe</h2> <p>You gather a bunch of non-believers in a room. If they can't discuss what they believe, then what's the next question? Why other people believe, of course!</p> <p>One of the best talks of the convention was about death. Sam Harris suggested that the religious are united in their fear death - they can't accept that one day, their consciousness will end <a class="footnote-reference" href="#id15" id="id7">[7]</a>. Atheists however, accept this. Perhaps, he speculated, it's the one fundamental difference between us.</p> </div> </div> <div class="section" id="don-t-you-just-bash-god-for-a-few-days"> <h1>"Don't you just bash god for a few days?"</h1> <p>I hope you've seen by now that we discussed many things beyond the low hanging fruit of god. In fact, the god-bashing was largely left to the comedians on the first night. Perhaps that helped "get it out of our systems"? I know in my case, it was never "in" my system - I didn't want the convention to be all god-bashing, and it certainly wasn't.</p> <p>The comedians had a field day with the topic. The religious probably find this boring, but as the comedians themselves pointed out, comedy is about speaking truth to power, pointing out the absurd - and religion is a fertile topic for rational minds to pick over. Personally, I didn't feel the comedy was required, but I still enjoyed it <a class="footnote-reference" href="#id16" id="id8">[8]</a>.</p> </div> <div class="section" id="isn-t-it-just-like-church-but-for-atheists"> <h1>"Isn't it just like church but for atheists?"</h1> <img align="right" alt="Jin-oh Choi and I" class="align-right" src="http://stuff.nigel.mcnie.name/nigel-jinoh-atheistcon.jpg" /> <div style="float: right; clear: right; font-size: smaller; margin-top: 5px; width: 300px; text-align: center;">Jin-oh Choi and I. The convention was great for making new, <em>visible</em> friends.</div><p>I don't think so. Remember, it was a convention. The last one was two years ago, and it cost money to attend. Church is weekly, and free. The religious have conferences; it would be fairer to compare this event to one of those.</p> <p>It is true that we sat and listened to speakers, just like you'd do in a church. But then you'd do that at any other convention too. If this was just like church, then almost any other conference is just like church as well.</p> <hr class="docutils" /> <p>I hope you've not got a "feel" for what goes on at an atheist convention, what we talk about, and why we do it. It might not be your idea of a good time, but for a few days, four thousand of us had a blast, and I know I can't wait for the next one!</p> <p><em>If you have a question about the convention, please feel free to ask in the comments and I'll be happy to answer. Civility is mandatory, thanks. This is my website, not yours.</em></p> <table class="docutils footnote" frame="void" id="id9" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Note that I said "great". Most people readily say they feel the ability to reason is merely a "good" thing. Many at the convention, on the other hand, stronly believe that it's one of our finest attributes.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id10" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Surely, you might think, most people use their minds rationally? You'd be right - almost everybody does, in many areas of their lives. However, a significant portion of the population have large blind spots in certain areas. It can be frustrating to talk to otherwise rational, smart human beings, who then profess a belief in something completely nonsensical. It can quite easily derail a conversation, and I've had to train myself to ignore such comments in the interests of civility.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id11" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Their task is growing harder. The US supreme court is now <a href="http://www.factmonster.com/us/supreme-court/supreme-court-members.html">dominated by Catholics</a> who vote as a bloc.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id12" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>You might have expected for atheists to call for religious education to be expunged from the cirriculum entirely. In fact, many were calling for comparitave religious studies to be taught instead - i.e., teaching facts about all religions, done by trained teachers. I only partially agree. I think that such a topic would be a worthy part of Social Studies, but I don't see why it needs its own subject, unless it was strictly optional. I'd rather see critical thinking and ethics become subjects first...</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id13" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id5">[5]</a></td><td>It's widely accepted by atheists that our morality comes from within us, and has been shaped by our evolution. If we didn't have it, we certainly wouldn't have made it as far as Mount Sinai.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id14" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id6">[6]</a></td><td>Some will say, "bashing the Catholics again, how dull". We keep doing it because it <em>matters</em>. Africans are <em>still</em> dying, children are <em>still</em> being raped, and the Catholic Church is one of the richest, most powerful organisations in the world. If we don't fight for their children, who will? "Moderate" catholics should be complaining the loudest, but of course they have the most to lose...</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id15" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id7">[7]</a></td><td>You might not like this idea. To atheists, whether an idea is agreeable or not doesn't matter. We care more about what is true. While we don't know what it's like to be dead, we can speculate based on what we know. For example, as you damage a brain, it loses functionality - for example, you lose memories, or skills. We've seen no reason to believe, that if you damage a brain totally - killing it, in fact - that somehow part of you will depart your body, remembering your life and knowing how to speak English.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id16" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id8">[8]</a></td><td>It's the first time anyone has ever got me to laugh at a dead baby joke, which I won't repeat here. But I will assert that, if you have a rational mind, you would have laughed too, perhaps while cringing. Comedy does that sometimes.</td></tr> </tbody> </table> </div> Kony http://nigel.mcnie.name/blog/kony <p>Since the '80s, a delusional psychopath has raised an army of children, often by killing their families and neighbours so they have little choice but to fight for him.</p> <p>His crimes include murder, enslavement, sexual enslavement, rape, cruel treatment of civilians, intentionally directing an attack against a civilian population, pillaging, inducing rape, and forced enlistement of children. His goal: a theocratic state where he is little short of a god.</p> <p>He has evaded capture for over six years, hiding behind his army of children and the humanity of those who would put an end to his totalitarian ambitions. "Evil" barely begins to describe him. We should be offended that he still breathes the same air as us.</p> <p>And your outrage is directed at people who care enough to let you know?</p> <p>Grow a fucking moral compass.</p> A Beginner's Reflection on Kiwi Foo http://nigel.mcnie.name/blog/a-beginners-reflection-on-kiwifoo <p>I hope nobody feels I am being ungrateful by casting a critical eye over a gift as valuable as an invitation to <a href="http://baacamp.org">Kiwi Foo Camp</a>, a private gathering of some of the most interesting, driven, intelligent and friendly people you're ever likely to find in one place. It feels a little like being a guest in Auckland, and being asked "well, what do you think of our city?". You might venture platitudes about the nice weather, beach proximity and stunning <a href="http://www.giapo.com/blog/">gelato</a>, but my god the <em>traffic!</em> How do you stand the mind boggling <em>tedium</em>... apologies, I digress. On the contrary, I was most surprised and grateful for my invitation (thanks Nat!), and offer this honest feedback in the hope that other attendees - and future first-timers - will learn something from it.</p> <a class="reference external image-reference" href="http://www.flickr.com/photos/titine/6860841373/in/photostream/"><img align="right" alt="The Kiwi Foo Space Programme at work" class="align-right" src="http://stuff.nigel.mcnie.name/kiwifoo-space-programme.jpg" /></a> <div style="float: right; font-size: smaller; margin-top: 5px; width: 333px; text-align: center;">The Kiwifoo space programme at work. <em><a href="http://www.flickr.com/photos/titine/6860841373/in/photostream/">Photo</a> by Justine Sanderson</em></div><p>Luckily, I left the great Auckland why-the-hell-doesn't-somebody-fund-public-transport traffic jam behind as I road-tripped to Warkworth for a weekend of stimulation and good-natured mayhem. Foo starts only with the attendees - there are no "speakers" on Friday night when we arrive - but by Saturday morning, everyone had combined to fill the weekend with a broad array of sessions: poetry, software, law, science, even surviving the Zombie Apocalypse. Although I spent the morning running some 18 kilometers around the hills of Warkworth with a few others. That's the kind of gathering Foo is - if it sounds like fun, somebody makes it happen <a class="footnote-reference" href="#id4" id="id1">[1]</a>.</p> <p>Foo advertises itself as a place where the attendees "network, share their works in progress, show off the latest tech toys and hardware hacks, and find new partners for collaboration". A noble statement, which we shall return to later. For now, perhaps a sample of the experience... somebody brought a working Tesla coil; others were pitching startup ideas; a meterological balloon was seen all over the place as the "Kiwifoo space programme" spied on proceedings from on high. A debate on Saturday night had everyone in stitches while we dealt out some medieval mob justice, and sessions were held on on all manner of topics from drugs to designing a container house. "Energising" doesn't even begin to describe it. There were times when if the Average Joe had walked in on us, things might have ended with a Dotcom-esque visit from the police. But among this group, Tesla coils, poetry and Zombie Apocalypses are viewed as "cool!" and/or "interesting!", which is exactly as they should be.</p> <p>The sessions... well. This this the part where I tread carefully. Most of them really were fantastic, the calibre of the attendees saw to that. But I found that one or two of the discussion sessions were echo chambers, and in some, there was barely any listening. Someone would be talking, and others would be poised on the edge of their seats, busting for the current speaker to finish, so they could throw their opinion against the wall. In both cases, louder people tended to end up dominating the conversation, and I would come away from a session thinking that nobody had learned anything.</p> <p>Still, many of the sessions were run most admirably <a class="footnote-reference" href="#id5" id="id2">[2]</a>, and others were saved. In the session on drugs in New Zealand, it was obvious quite early on that we were all in favour of at <em>least</em> decriminalising drug usage, if not blanket legalisation. Against this backdrop, the reactions to the recent <a href="http://www.lawcom.govt.nz/news/2011/05/report-controlling-and-regulating-drugs-released">law commission report</a> were quite negative. "Drugs should be seen as a health issue..." yes, we all agree. It was only after someone started role-playing a rich, white, conservative parent that the room began to see that we should be more behind the report. It is a pathetically small step, but - we now realised - it's not an insignificant one. If its recommendations are adopted, we move the law in the right direction, and we can take up the conversation later from a higher position. After the session, I felt like we'd had a small success - now a room full of people were able to make a more informed judgement about something.</p> <p>However, on reflection I realised that the sessions were merely appetisers for a deeper reason to come to Foo - <strong>to discover people worth knowing</strong>. In this respect, they did not need to solve a problem, or even be remotely productive - they merely provided a topic and a list of people to seek out for further discussion. The real value comes in the conversations afterward <a class="footnote-reference" href="#id6" id="id3">[3]</a>.</p> <p>Returning to the Kiwi Foo advertisement, it's now possible to see why Foo is such a roaring success. What better way to find people worth knowing than to gather them together, give them permission to show and tell, let them talk about the things currently on their minds, and above all, keep them together? Providing all food at the venue is a masterstroke (and what excellent food it was), as is letting some of the attendees sleep on site. Conversations can go much further if you know your bed is just a door away.</p> <p>The whole experience was a blast, and if you should be lucky enough to receive an invite, my advice is to accept it greedily. My final thought is this: If Foo is about discovering people worth knowing, then it would make sense to maintain a strong intake of new faces each year. I have no idea how Nat, Jenine and Russell could balance this with keeping the camp small enough to manage, but I envy their dilemma. There ain't no party like a Foo party.</p> <a class="reference external image-reference" href="http://www.flickr.com/photos/titine/6860896691/in/photostream/"><div align="center" class="align-center"><img alt="The attendees of Kiwi Foo 2012" class="align-center" src="http://stuff.nigel.mcnie.name/kiwifoo12.jpg" /></div> </a> <div style="text-align: center; font-size: smaller;">A Foo party. <em><a href="http://www.flickr.com/photos/titine/6860896691/in/photostream/">Photo</a> by Justine Sanderson</em></div><hr class="docutils" /> <p>In response to a draft of this, John Hart pointed out a couple of ABC's that I neglected to mention, which first-timers will appreciate:</p>  <blockquote> <p>It's probably worth noting that an un-conference is nothing like any other traditional conference format - attendees should expect to be part of the content. It's not a place for passive consumers.</p> <p>Complete participation is the best way to get a 2nd invite. Throwing yourself into the spirit of sharing (and especially hosting a session) will ensure you get the most out of the experience.</p> </blockquote> <p><em>Thanks to <a href="http://twitter.com/FarmGeek">John</a>, <a href="http://twitter.com/rmi">Rob</a>, <a href="http://twitter.com/N_BenWilson_Z">Ben</a> and <a href="http://twitter.com/JoshForde">Josh</a> for reviewing this article.</em></p> <table class="docutils footnote" frame="void" id="id4" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>Getting up before 7am after a night of revelry, being driven miles down the road and sent into the bush to run back, knowing that you have just as full a day of mind-expanding discussion and merriment as everybody else still ahead of you, might not sound like fun to most people. I guess we gluttons for punishment will either die out or inherit the earth. Draw your own conclusions about my attendance at the Zombie Apocalypse session.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id5" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>One in particular, which was in my opinion a success against the odds, was the session on rebuilding Christchurch. There seemed to be more opinions than noses in the room, yet the discussion chairs listened, were engaged, and appeared to gain a lot from the participants.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id6" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Which makes me regret not suggesting a session on webapps: what's new, what's cool, where are they heading. Maybe another time.</td></tr> </tbody> </table> GYGO - A Post-mortem http://nigel.mcnie.name/blog/gygo-a-postmortem <p>After two years where I was working on <a href="http://www.getyourgameon.co.nz/">Get Your Game On</a> largely alone, Martyn finally finished up at Catalyst a few weeks ago to work on it with me. One week later, we decided to give up.</p> <p>On the surface, this sounds mad. Did Martyn throw in his job for nothing? Do we not have an ounce of intelligence between us? What happened to our big dream? As of a few months ago, we were still evangalising GYGO; we were even preparing to bring another co-founder on board. Now... it's over. The site lives on only to serve our customers while they decide what on earth to do, a conundrum I now share with them.</p> <p>Some background. Martyn and I have shared a journey for a while now: can we use our skills to fashion a future where we no longer exchange hours for dollars? To me, this is a powerful form of freedom; while you can live in a free society and learn to think for yourself, not having to worry about money multiplies your options. It is the oxygen of modern society, and it seemed (note the past tense, I'll return to that later) to me that having piles of it would leave you free to make whatever kind of difference you wanted in the world.</p> <p>GYGO failed not because the idea sucked, but because we didn't comprehend how hopeless the idea was for achieving this goal.</p> <p>It's not that it would never work. We have two paying customers who love it (and so do _their_ customers - players etc.), a bunch of interest from others who run sports competitions, an existing market to resegment and the positioning to do it (all online, easy/professional system - as opposed to the free, ad-sponsored or windows-centric competition). What we don't have is $500K, and quite frankly, if I want to pop my angel investee cherry and throw away five years, there had better damn well be a $100mil company at the other end. The market for GYGO is far too niche for that.</p> <p>Our mistake: we underestimated the work involved in beating Excel. Here's an example. In a kid's football league, a good team is scheduled to play a bad team, and the bad team defaults. This would normally count as a 3-0 victory to the good team, but they complain that "we would have beaten them by much more!". The solution? Count the game as a 7-0 win for the good team, but a 3-0 loss for the bad team. I know, I cried into my relational database too <a class="footnote-reference" href="#id3" id="id1">[1]</a>. Then I added support for it, because massaging results is rife in social competitions. It's the kind of thing you do in excel simply by changing the number in a cell. Excel matches how most people think about data - hack and kludge until it says what they want. This is precisely why it's so hard to beat.</p> <p>Perhaps if the real world was sane, we could have developed GYGO on the cheap, and turned it into a 4-5 person company doing $1mil turnover. That would have suited me fine - we could have automated it to the point where I was drawing a salary for doing nothing: mission accomplished. It seems obvious now that targetting SMBs would result in a massive support headache <a class="footnote-reference" href="#id4" id="id2">[2]</a>, which we couldn't spread over thousands of customers, and as you just heard, real world data laughs at your attempts to corral it. Large support costs, a small market and investment required are a fatal mix.</p> <p>So, what next? I have no idea. Between this and general life upheaval, I'm not even sure that my goal of being free from money is a good one. I was just at the most excellent kiwifoo, and it's got me thinking again about New Zealand and earthquakes, and whether we couldn't prepare to save some lives. If Wellington was flattened, I'd love to be able to tweet from beneath a crushed building, and know that volunteers would see it and alert first responders who could dig me out. No money in it, but frankly right now, I'm not sure I give a shit about that.</p> <hr class="docutils" /> <p><em>Postscript:</em> Reading through this again, it occurs to me that I didn't say anything positive about the experience. I don't regret a single day I spent doing it. I've learned a lot about business, relating to people, solving problems; I've joined twitter and absorbed heaps of great lessons from inspiring people via it; I felt what it was like to be wildly optimistic about the future; I gained heaps of life experience I never would have gained otherwise; made heaps of friends, and played a lot of football!</p> <p>Yes, I do recommend you start your own business, especially if you have a skill and want more out of life. Treat it as an investment in yourself, and you won't be disappointed.</p> <p><em>PPS:</em> My largest regret is letting our customers down. They're fantastic people, and they struggle with high workloads and sometimes angry people so that our kids and community can have fun. They both took the news gracefully, and one of them even gave me a jigsaw puzzle as thanks. If I can give you one piece of advice, it would be to to always tell them the truth and work your ass off for your customers. It's not just good business sense - when they're trusting you, it's the right thing to do.</p> <table class="docutils footnote" frame="void" id="id3" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>As Martyn dryly observed, we should have used MySQL.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id4" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>I envy Xero, they have an army of accountants with a vested interest in supporting their userbase.</td></tr> </tbody> </table> Realtime Conversations Made Easy (my latest job) http://nigel.mcnie.name/blog/realtime-conversations-made-easy-buzzumi <p>Why is it so hard - technically - to have a realtime conversation with people all over the world?</p> <p><em>"It's easy, just use Skype!"</em>. Not so fast. People need to install Skype - this is a task too hard for Grandma (unless you visit first and do it for her). And there's the whole contacts process. If you want to talk to me, we have to find each other and add contacts. After that, Skype will bother you when it's my birthday even though you only wanted to have one little chat. I could go on... <a class="footnote-reference" href="#id5" id="id1">[1]</a></p> <p><em>"You're a geek, use IRC!"</em>. Of course, IRC is basically unusable by many people whom you'd like to talk to - some co-workers, your family, certainly not Grandma. At least there's no contacts to manage. But there's no audio/video, and you can't "drop in and out" of an ongoing conversation very smoothly - if you lose your connection to the chat, you lose the messages sent while you were gone <a class="footnote-reference" href="#id6" id="id2">[2]</a>.</p> <p><em>"Use google chat/jabber!"</em>. Worst of both worlds. Low penetration, high complexity.</p> <p><em>OK then, just use... hmm</em>. Exactly.</p> <div class="section" id="ad-hoc-easy"> <h1>Ad-hoc & Easy</h1> <p>There are two problems that current solutions have. One is that our conversations are ad-hoc, sometimes with people you've never met before and only have a fleeting connection to. Skype's contacts are not ad-hoc.</p> <p>The other is that people need easy. Configuring IRC is hard. Heck, installing a damn program is hard for many people. Twitter and Facebook get close, but holding a decent conversation on Twitter is impossible, and Facebook is too autistic for people to use for ad-hoc communication. Besides, even <em>signing up</em> is often too hard. How many services do you not use because you would have needed to sign up first?</p> <p>When I worked on <a href="http://www.mahara.org/">Mahara</a>, we used IRC. A few enthusiastic types joined us, but in large, IRC remains to this day inaccessable to many people. Not all users of Mahara are willing, or capable, of using it.</p> <p>And when the Christchurch Earthquake hit this year, the response team that built eq.org.nz used Skype - which was by most accounts a terrible solution. Skype chat simply has too many bugs when you get to rooms of more than a few people - and we had over a hundred. Richard wrote a post-mortem that <a href="http://phirate.posterous.com/real-time-communication-in-a-crisis">lays out the issues for communicating in a crisis</a>.</p> </div> <div class="section" id="introducing-buzzumi"> <h1>Introducing buzzumi</h1> <p>So it's been my great privilege to have worked on a new service that addresses these problems. With <a href="http://phirate.posterous.com/">Richard</a> (a fantastic webapp developer who happens to be my cousin), we've been the tech team behind <a href="https://buzzumi.com/">buzzumi</a>, which just launched (he's lead, I'm wingman). It's a webapp that lets you create ad-hoc discussions - text chat, audio and video (A/V is completely optional). Your chat is accessible to anyone who can click on a link, <strong>without them logging in</strong> <a class="footnote-reference" href="#id7" id="id3">[3]</a>. It's simple, beautiful, and lightning fast.</p> <img alt="http://f.dollyfish.net.nz/5d7c4c" src="http://f.dollyfish.net.nz/5d7c4c" style="height: 290px;" /> <img alt="http://stuff.nigel.mcnie.name/buzz-video.png" src="http://stuff.nigel.mcnie.name/buzz-video.png" style="height: 290px;" /> <p>The chat host can set the background, and it changes for everyone in the chat. There's no limit on participants, and up to six people can use A/V at once <a class="footnote-reference" href="#id8" id="id4">[4]</a>. It's great for team meetings, one on one discussions, and even webinars with hundreds of guests watching a broadcast. There's nothing to install (except flash), and no barrier to entry. You don't have to know the other participants, and when you're done, you can close the chat and never see them again.</p> <p>Perhaps one of the coolest features is that you can make a chat have a fee for entry. buzzumi handles all of the payments for you. So you could hold a webinar on a topic you're an expert in, charge $50 to enter, and simply collect your profits when you're done (buzzumi takes a 10% cut).</p> </div> <div class="section" id="please-try-it-out"> <h1>Please Try It Out!</h1> <p>If you're in education, how could you use this for your school/university? If you're in disaster response, can we please ditch Skype for this? And if you know me from somewhere else, I'd love to hear your thoughts about it, technical or otherwise.</p> <p><a href="https://buzzumi.com/">Give it a try</a>, and let me know what you think! I have created <a href="https://buzzumi.com/nigel/BqI2Wq7lFX">a chat which I'll hang out in for a while</a> as well, come by and say hello if you have a second.</p> <p>ps: <a href="http://www.getyourgameon.co.nz/">Get Your Game On</a> is still rolling, we are busy running much of the summer football around Wellington.</p> <table class="docutils footnote" frame="void" id="id5" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>.. in this footnote :). Skype can <a class="reference external" href="http://en.wikipedia.org/wiki/Supernode_(networking)">suck bandwidth when you're not around</a>, the new UI <a href="http://blog.brizk.com/post/4508062266/on-the-skype-ui-disaster">is</a> <a href="http://community.skype.com/t5/Windows/Problems-with-Skype-UI-since-5-x-even-to-5-5/td-p/80402">crap</a>, it <a href="http://en.wikipedia.org/wiki/Skype_security#Flaws_and_potential_flaws">promises encryption but has flaws including rumoured government backdoors</a>, and it's now owned by Microsoft. Eew.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id6" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Geeks have ways to get around this, through hax that allow them to use IRC on internet connections more reliable than their own. This is a workaround, not a true solution.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id7" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>To create chats, you do have to sign up - but your guests do not.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id8" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Just quietly, the real limit is quite a bit higher, although we give no guarantees that it'll work properly beyond 6. I think the current record is 14. And note that this limit applies to people <em>publishing</em> their A/V, so you can have just one or two publishers with many more watching.</td></tr> </tbody> </table> </div> Continuous Deployment: Reprise http://nigel.mcnie.name/blog/continuous-deployment-reprise <p>Here are my <a href="http://www.slideshare.net/nigelmcnie/the-why-and-how-of-continuous-delivery/download">current slides</a> (which include my speaker notes) for the CD talk I've been doing around Wellington recently. Here it is embedded (CC-BY-SA):</p> <div style="width:425px; margin: 0 auto" id="__ss_9327958"> <strong style="display:block;margin:12px 0 4px"><a href="http://www.slideshare.net/nigelmcnie/the-why-and-how-of-continuous-delivery" title="The Why and How of Continuous Delivery" target="_blank">The Why and How of Continuous Delivery</a></strong> <iframe src="http://www.slideshare.net/slideshow/embed_code/9327958" width="425" height="355" frameborder="0" marginwidth="0" marginheight="0" scrolling="no"></iframe></div><p>There are some changes from <a href="/blog/continuous-deployment-a-better-software-deployment-strategy">the first time I gave it</a>. Sadly, I have failed to shorten it, but here are the main changes:</p> <ul class="simple"> <li>I've renamed it to Continuous <em>Delivery</em>. This is simply making the point that the aim is Continuous Deployment <strong>to production</strong>. If you're doing it up until staging/test, and not production, you won't be getting the full range of benefits - although getting as far as staging is a good start if you're converting an existing project to CD.</li> <li>I have backed down slightly regarding my position on Feature Flags. Previously, I believed they were essential, but <a href="http://scottchacon.com/2011/08/31/github-flow.html">Github clearly manage without them</a>. As such, I've downgraded them to a "strong recommendation". Feature Flags give you dark rollouts/kill switches and so on, which feature branches can never provide.</li> </ul> <p>I saw some feedback saying that there were not enough concrete examples. I agree, to some extent - however, I still find that many people haven't even <em>heard</em> of CD. As such, my presentation is designed as a conciousness-raising one - hopefully people who are interested will search around and find out more. If I had more time, it'd be great to walk through an example (IMVU springs to mind), but the hard reality is that I need to look for things to take <em>out</em>.</p> <p>There were also a couple of questions around how to convert an existing project to CD. So here's my suggested strategy for accomplishing that:</p> <div class="section" id="start-with-a-test-suite"> <h1>1. Start with a Test Suite</h1> <p>Remember the goal is to <strong>go fast with confidence</strong>. If you're just going fast, you're simply speeding, and will soon become another statistic.</p> <p>Okay, end of car analogy. However, you need to build confidence, which is what the test suite is for. Start there. You don't need to over-do it, but be smart. Start by ensuring it's easy to add tests, and then write relatively comprehensive tests for the critical flows in your application. What these are is entirely up to you - if you don't know them, maybe you had better start there :).</p> <p>You should also set up a CI server, and get the tests running on every commit. If nothing else, this should enhance your existing development process.</p> </div> <div class="section" id="rig-up-continuous-deployment-to-staging"> <h1>2. Rig up Continuous Deployment to Staging</h1> <p>Staging, test, UAT, whatever you call these environments that aren't production - get continuous deployment working at least that far. These environments can be broken with relative inconsequence, so you can work with confidence initially. Over time, you should find their uptime improves, especially as your test suite begins to show its strength.</p> <p>In particular, in this step, your aim is to put the test suite in the way of deployment. It should be impossible to deploy without a test run having completed successfully on the code being deployed.</p> <p>Note that the presentation suggests that you employ a "Commit triggers tests, tests passing triggers deployment" tactic. This is a simplification. In larger teams, it may make more sense to work more like this: "tests are run every 15 minutes, with success resulting in deployment". This allows many commits to be made in one window (common for large teams), with the knowledge that anything you commit will take at worst 30 minutes to make it to production (assuming the tests pass).</p> </div> <div class="section" id="the-management-gambit"> <h1>3. The Management Gambit</h1> <p>You will know you've done the first two steps when the tech team thinks "this is working nicely all the way to staging - why not to production?". There's one final obstacle in your way - the fear of everyone outside the tech team.</p> <p>On some level, management/clients are going to have to get comfortable with the idea of code being deployed without their approval, and often even without their knowledge. You can use presentations like mine to help here. Remember, if you can win over just one person, they can help sell the idea to everyone else.</p> <p>At a base level, they most likely will fear the lack of control they will have. To combat this, tell them that you'll implement the system, but they have the right to demand change freezes - periods of time where you won't be allowed to deploy. They probably already have this right, but re-affirm it anyway.</p> <p>This is false security for them. Martyn has a sign on his desk, which reads <em>"Change Freeze (n): A time in which more changes are made to production than usual"</em>. When there are urgent problems, change freezes be damned - changes are made, and we all know it. The next time they have a change freeze, they'll ask for a change, and instead of flailing about you'll be able to make it with a smile. They'll never have another freeze again.</p> <hr class="docutils" /> <p>I also mentioned during the talk a USB rocket launcher for retaliating against people who break the build. <a href="http://www.papercut.com/blog/chris/2011/08/19/who-broke-the-build/">Enjoy!</a></p> </div> Closing The Loop - The Canterbury Quakes, eq.org.nz And The SBTF http://nigel.mcnie.name/blog/closing-the-loop-the-canterbury-quakes-eq-org-nz-and-the-sbtf <p><em><a href="http://www.flickr.com/photos/greenfluoro/5472977899/in/photostream/">Image Source</a></em></p> <img align="right" alt="http://stuff.nigel.mcnie.name/ccnz.jpg" class="align-right" src="http://stuff.nigel.mcnie.name/ccnz.jpg" /> <p>It's been a year and a day since the first of a series of earthquakes hit Canterbury, which peaked in February this year with a <a href="http://en.wikipedia.org/wiki/February_2011_Christchurch_earthquake">6.3 magnitude quake that took over 180 lives</a>. While I was unaffected by the quakes themselves, I have since thought that I (along with everyone else in Wellington) have "gotten off lightly", due to the simple fact that <a href="http://db.nzsee.org.nz/2010/Paper23.pdf">Wellington straddles a known fault line with an 11% chance of rupture in the next 100 years</a> <a class="footnote-reference" href="#id5" id="id1">[1]</a>.</p> <p>The events following the quakes show that Wellingtonians, and indeed all New Zealanders, take this risk seriously. The <a href="http://eq.org.nz">eq.org.nz</a> project was a fantastic grass-roots effort that showed the "can-do" attitude in our country. People weren't happy just to donate or leave it to the government - instead they pitched in and did something that helped everyone.</p> <p>After eq.org.nz was shut down, there were some postmortems. Richard Clark wrote on <a href="http://phirate.posterous.com/real-time-communication-in-a-crisis">realtime communication in a crisis</a>, <a href="http://phirate.posterous.com/development-at-crisis-speed">development at crisis speed</a> and a look at <a href="http://phirate.posterous.com/many-minds-the-value-of-the-crowd-in-a-disast">how technology could automatically swing into effect in a disaster</a>. InternetNZ facilitated a <a href="http://internetnz.net.nz/news/blog/2011/Helping-Wellington-Get-Thru">discussion between the internet community and the Wellington City Council</a>, and <a href="http://www.youtube.com/watch?v=Zv7gEhKEMmw">a great video was made summarising the project</a>. As I was a part of the tech team for the site, I took one lesson away in particular - <strong>the tech was not as ready as it should have been</strong>.</p> <p>Yes, we managed to set up a site in a matter of hours after the quake. Yes, the site got significant use, and yes it was helpful to some. But I feel that we can do much better in the future, particularly with regards to the technology. The two areas we could improve on are <strong>being prepared</strong> and <strong>improving the tools</strong>.</p> <div class="section" id="being-prepared"> <h1>Being Prepared</h1> <p>From quake to functioning map took us a few hours - a commendable effort, made possible thanks to <a href="http://www.ushahidi.com/">Ushahidi</a>, their hosted <a href="http://www.crowdmap.com/">Crowdmap</a> product, and of course the efforts of everyone involved. However, I feel this was much too long.</p> <p>Firstly, we were working in an emotionally charged, chaotic environment - one in which mistakes would have been easy to make. Second, we were given a lot more luck than we may have otherwise expected, thanks to the Ushahidi team co-operating with us over the migration away from Crowdmap, and the help of <a href="http://crisiscommons.org/">CrisisCommons</a> in getting us started <a class="footnote-reference" href="#id6" id="id2">[2]</a>. And third, during the first few hours, stuff.co.nz and NZHerald (the two major online news sources in NZ) both set up their own maps, and had it not been for our connections and their willingness to help, three competing maps could easily have sprung up, none of which would have been much help at all.</p> <p>Spending hours setting up the site manually is the wrong way to do things, and we all know it. However, we can't predict disasters in advance. So it seems to me that we either need the ability to set up a site in minutes, or a "general" site prepared in advance, which can be adapted within minutes for whatever disaster is taking place.</p> <p>Let's explore both options briefly. The ability to set a site up in minutes requires some prior planning, however it makes no assumptions about the nature of the disaster, and allows us to set up and tear down sites as we see fit. Disaster hits two cities? Just deploy two sites. During down time, as long as a minimal amount of maintenance is done, we can keep the process well oiled. Furthermore, we can develop "template" sites - one for earthquakes, one for tornadoes etc., and deploy the right "type" of site during a disaster, providing great agility.</p> <p>Setting up a site (or sites) in advance has a different set of complications and benefits. A URL can be published in advance of any disaster, that everyone knows, so when a disaster hits more people know where to go (e.g. wellington.disaster.org.nz <a class="footnote-reference" href="#id7" id="id3">[3]</a>). However, it may take more work to be able to re-configure the site for different disasters, as opposed to simply deploying a fresh site. And naturally, the site will need maintaining during the time between disasters, which could be many years.</p> <p>The other aspect of being prepared, as alluded to here, is <strong>having data prepared</strong>. That doesn't just mean having categories prepared - that means having map data for everything of significance ready. Why should it take until a disaster for us to plot Wellington's ATM locations on a map? Why can't we prepare this data beforehand, ready for use the moment it's needed?</p> </div> <div class="section" id="improving-the-tools"> <h1>Improving the Tools</h1> <p>We made great strides towards improving Ushahidi for the task of disaster mapping on the eq.org.nz project. However, I fear that a lot of our good work hasn't made it as far as it could. <a href="https://github.com/ccnz/Ushahidi_Web">The code</a> sits (rots?) in a github repository, many of our changes having not made it anywhere near upstream - despite still being needed.</p> <p>I've helped out with a couple of disasters since the quakes, and in each case, I found myself doing the same setup work, then fixing the same bugs, as we did previously. Clearly, pushing some fixes upstream would be a good start. However, the issues with using Ushahidi for crisis mapping run deeper than just a few fixes.</p> <p>We made many changes that improved the usability of the map and homepage, <a href="fixing-website-performance-issues-whack-a-mole">the stability/scalability of the platform</a>, and to streamline the workflows of people approving reports. We didn't push these changes upstream, and I'm reminded of this with every disaster has occurred since then. The eq.org.nz site really did finish as a much better product, but it's a height we haven't attained since.</p> <p>To be clear, I'm not blaming anyone for this. People burned out, interest waned and we all had our lives to get back to. But I think it would be great if we could "close the loop". After all, you never know when it's you who will be needing the map...</p> </div> <div class="section" id="closing-the-loop"> <h1>Closing the Loop</h1> <p>I found out about the <a href="http://blog.standbytaskforce.com/about/">Standby Task Force</a> after the Christchurch Quake. They're a network of <strong>"adhoc groups of tech-savy mapping volunteers that emerge around crises into a flexible, trained and prepared network ready to deploy."</strong> In other words, they're the eq.org.nz team, except larger, more organised, and with a global focus. When disaster strikes, if the locals ask for help, the SBTF are ready to respond, providing a map, volunteers and expertise to get things rolling. Now I think of it, they're a little like Internation Rescue (from Thunderbirds), in its infancy. No rocket ships I'm afraid ;), but a group of people willing and able to help in times of emergency.</p> <p>Somehow, through a combination of my motivation to improve the tech for future disasters, and the prodding of <a href="http://twitter.com/georgechamales">George Chamales</a> and <a href="http://twitter.com/Jasper_Johns">Kirk Morris</a>, I've fallen into the position of SBTF Tech Team Leader. My plan is to, as part of this team, work on both of the technical aspects outlined above. The goal: that <strong>anyone will be able to deploy a map ready to handle a disaster within minutes of it occuring</strong>.</p> <p>I've donated some code from <a href="http://www.getyourgameon.co.nz/">Get Your Game On</a> to get started on a <a href="https://github.com/StandbyTaskForce/sbtf-ops">system for performing one-command</a>. We're also planning on maintaining a branch of Ushahidi <a class="footnote-reference" href="#id8" id="id4">[4]</a> optimised for crisis mapping, to which I hope we can apply many of our patches from eq.org.nz and other sources over time.</p> <p>With this, I hope we can truly close the loop - so that when Wellington, or anywhere else, is struck by disaster, our response will be as good as it can be.</p> <p><em>Final thoughts: I'm only focusing on the tech. There are clearly other issues we need to work through, not least the political ones. I hope next time to see a much closer relationship between government and such volunteer efforts, although it's not an issue I feel I can influence personally.</em></p> <table class="docutils footnote" frame="void" id="id5" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>In researching this I was glad to discover that the 11% figure is a 50% decrease in what was commonly believed before the "It's Our Fault" study. All the same, 11% is not a particularly comforting figure.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id6" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>In comparison, imagine what it would have been like if the disaster had struck 10 years ago. No CrisisCommons, no Ushahidi, barely any internet to speak of. We are truly lucky.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id7" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Naturally, something shorter would be better!</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id8" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>Naturally, we will try and push as much upstream as possible, however there's a simple reality that not all patches may be suitable.</td></tr> </tbody> </table> </div> Continuous Deployment - A Better Software Deployment Strategy http://nigel.mcnie.name/blog/continuous-deployment-a-better-software-deployment-strategy <p><em>Note: I have updated the presentation since first giving it - <a href="/blog/continuous-deployment-reprise">check out the new one</a>. The new post also includes tips for converting an existing software project to CD.</em></p> <p>I gave a talk on Continuous Deployment today. Here are the <a href="http://stuff.nigel.mcnie.name/continuous-deployment/">slides</a> and here are my <a href="https://docs.google.com/document/d/1S-rftz6BmPosEw2T5Qf2-JA88wcnyTIfGogvbUEJSjg/edit?hl=en_US">speakers notes</a> - which will probably be more interesting.</p> <p>My thinking on CD has advanced since last year, but the essentials remain the same. To do CD is to make a strategic decision to remove fear from the deployment process; to treat your test suite as an asset of the highest value; to truly value user feedback; to remove deployment as an obstacle to any other activity.</p> <p>What has changed? Over the last year, I've done CD on one project and worked on another using a fortnightly release schedule. I've been able to compare the two and observe first hand just how beneficial CD can be.</p> <div class="section" id="continuous-deployment-going-fast-with-confidence"> <h1>Continuous Deployment - Going Fast With Confidence</h1> <p>On the CD project, the complete lack of effort required to deploy changes has been a <em>huge</em> timesaver. I have never felt the need to wait before deploying one change, even if I was about to work on another. Little fixes, in other words, made it to production very quickly, pleasing my customers far more than assurances of "it'll be fixed next week" would have.</p> <p>I think the best moment was when I noticed a user trying and failing to complete a wizard due to a bug. I fixed the bug and deployed - allowing them, on their sixth try (and probably to their complete surprise), to complete it. If ever there was a moment where I appreciated the value of a robust, quick deployment process, this was it.</p> <p>Furthermore, this experience highlights one of the key benefits of CD. I could have hacked a fix on production - <strong>but it was easier to use the CD process, which included a full test suite run</strong>. There's simply no way any other process could have provided the same speed with the same level of assurance - hacking on production would have been the only faster option, and it would have been wildly dangerous. <a class="footnote-reference" href="#id5" id="id1">[1]</a></p> </div> <div class="section" id="fortnightly-deployment-the-lie"> <h1>Fortnightly Deployment - The Lie</h1> <p>On the fortnightly deploy project on the other hand, we encountered all the same issues that I'm so tired of.</p> <p>We'd do a release, then for the next two weeks, some fixes would be marked as so urgent that we had to do a deployment of just that fix, immediately. We'd made sure deployment was as close to a one-command process as possible. However, the process of patching and testing the stable branch was an annoying break in rhythm, given that we were doing most development on trunk <a class="footnote-reference" href="#id6" id="id2">[2]</a>.</p> <p>This was actually a point raised by <a href="http://www.chilts.org/">Andy Chilton</a> at the talk today. It seems that many project teams realise that there are some fixes that just have to make it out fast, and as a result they build a separate "hotpatch" channel to accomodate them.</p> <p>In my view this is madness, no matter how well tended the "hotpatch" process is. Do your hotpatches go through the test suite? They certainly should! And why create a "fast path", and then forbid its use in ways that would delight your customers?</p> <p>But I think my biggest objection is this: why have two processes when you could just have one? We coders know the evil that lies in needless duplication and complexity - which is exactly what a "hotpatch" system is. Duplication and complexity.</p> <p>The whole idea of having a separate deployment process exposes the "fortnightly" claim as a lie anyway. Who can honestly claim they deploy every fortnight, if they're hotpatching? <a class="footnote-reference" href="#id7" id="id3">[3]</a></p> </div> <div class="section" id="objections-to-cd"> <h1>Objections to CD</h1> <p>Perhaps the strongest objection that came up was that clients wouldn't tolerate the possibility of things breaking without them being aware of it. To me, this objection has a slight air of childishness about it - I'd give it more credit if clients ever bothered to hire a world-class QA team, but they never do, and they miss bugs slipping into production all the time even with their checking. I think there's just our old friend, the "Cover Your Ass" policy, at work here.</p> <p>Besides, nothing about CD precludes the possibility that they can still have a QA team checking things - with the able assistance of feature flags that limit features under development to just them. And I'd contend that the QA team would be just as delighted as the client themselves when told a bug they found half an hour ago was not only fixed on production and ready for them to check again, but that a test had been written to make sure it never happens again.</p> <p>Having said all of this, <a href="http://coffee.geek.nz/">Brenda Wallace</a> made the point that it all depends on the client, regardless of how good the idea sounds. Some simply won't change from what they know, and at the end of the day it's their project. Perhaps this is why CD is doing so well in the tech startup world - it's the startups themselves who are the clients <a class="footnote-reference" href="#id8" id="id4">[4]</a>.</p> </div> <div class="section" id="try-it-for-yourself-i-ll-help"> <h1>Try It For Yourself - I'll Help</h1> <p>All up, it was a great discussion, and it seemed like many there could at least see how CD could be better. If you count yourself among their number, I encourage you to try it out on the next project you do, and see how you go. I'm more than happy to chat with you about it and share experiences if you do, so feel free to contact me if you want to discuss anything about it.</p> <hr class="docutils" /> <p><em>As an aside, I do intend to continue my <a href="fixing-website-performance-issues-hardware">Web App Performance series</a>, I've just been focused on other things recently. Apart from business, I've joined the <a href="http://blog.standbytaskforce.com/">Standby Task Force</a> and am developing scripts to automatically deploy an <a href="http://www.ushahidi.com/">Ushahidi</a> within a few minutes of a disaster occuring. More on that in a future post.</em></p> <table class="docutils footnote" frame="void" id="id5" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>I'm the first to admit that this particular example was rather fortuitous, but I think it's even more relevant as your site gets busier. You'll see the errors occuring, diagnose and fix the problem, deploy - and it's inevitable that some customers will then begin to succeed at what they were doing. Contrast with hotpatching, where you could <em>break</em> the site for more people - or a slower deployment process where <em>more</em> people would encounter the problem.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id6" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>The CD example (the wizard fix) just goes to show how artificial this problem is. We were pushing back because our process made it harder than it should have been. Software development teams around the world do this all the time - lowering customer expectations about how long it takes to fix problems. I think we're doing our clients a disservice.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id7" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id3">[3]</a></td><td>Substitute "weekly", "monthly" etc. as appropriate. If you tell me you deploy weekly, I bet you do more than 52 deployments in a year.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id8" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id4">[4]</a></td><td>"Client" is defined here as "the organisation that uses the project for their benefit". For example, Fairfax uses Catalyst IT to develop stuff.co.nz. Fairfax is the client. In a tech startup, it's the startup themselves that gets the benefit from the project, so they're their own client.</td></tr> </tbody> </table> </div> NZ Red Cross Website Performance http://nigel.mcnie.name/blog/nz-red-cross-website-performance <p>I've just seen <a href="http://computerworld.co.nz/news.nsf/news/emergency-response-red-cross-it-and-the-quake-aftermath">this article in ComputerWorld</a> about how the NZ Red Cross scaled its website to handle the aftermath of the Cantebury earthquake on February 22.</p> <p>I don't know how much the Red Cross themselves knows about the details of how their site was fixed, but I know one thing for sure. It was <a href="http://twitter.com/arjenlentz">Arjen Lentz</a> and I who fixed it. Not Netspace.</p> <div class="section" id="the-story"> <h1>The Story</h1> <p>Here is what the story says happened:</p> <blockquote> <p>To cope with the spike in traffic Netspace Services Limited added a reverse proxy to handle the static content delivery, tuned PHP to include an op-code cache and also trimmed the PHP modules to the bare minimum. The Apache daemon was also tuned for high turnover of processes to prevent memory bloat. The web server was put onto a new 64-bit Debian operating system and the database was shifted to its own 64-bit hardware and operating system.</p> <p>The MySQL database was tuned, with indexing optimisations applied and alterations to cache settings to increase performance while reducing overall load. The presentation code was rewritten to push more work to the SQL database, to optimise queries, and to streamline processing. Within 36 hours of the site getting hit with a wall of traffic ten times larger than the solution was specified to handle, Netspace had reduced processing and memory load on the servers while being able to serve the higher connection demands from the world, and return to business as usual.</p> </blockquote> <p>The above has some grains of truth in it. However, it wasn't Netspace that eventually fixed the site. Read on for what really happened.</p> </div> <div class="section" id="the-facts"> <h1>The Facts</h1> <p>Immediately after the quake, the NZ IT community pooled together to begin work on what would eventually be <a href="http://eq.org.nz">eq.org.nz</a> - the Christchurch Recovery Map. I had previously worked on scaling a <a href="http://queenslandfloods.crowdmap.com/">map for the Queensland Floods</a> earlier in the year, so I became part of the team that worked on the site.</p> <p>We migrated the site from a temporary home on <a href="http://www.crowdmap.com/">Crowdmap</a> to dedicated servers, configured and tuned the stack supporting the site (PHP under fastcgi, MySQL, the <a href="http://www.ushahidi.com/">Ushahidi</a> software that powered the site), and monitored it while the site reached 100,000 visits in a week. A <a href="http://www.nzcs.org.nz/newsletter/article/94">more detailed list of what happened</a> was published in the NZCS Newsletter on the 18th of March.</p> <p>After the quake, the <a href="http://www.redcross.org.nz/">NZ Red Cross</a> site was very slow to respond, if it responded at all, to most requests. It was obviously under high load with much of the country wanting to donate, but it was clear that most people were not even getting through to the site.</p> <p>While we were working on eq.org.nz, we saw the Red Cross was having trouble, and the group made the decision to reach out to help. As a result, I was put in touch with Gerard Creamer at Netspace <a class="footnote-reference" href="#id3" id="id1">[1]</a>. He connected me to Fletch, who works for them as a sysadmin.</p> <p>The site was struggling. Fletch was preparing to move the site to a new server, having seemingly abandoned the existing one as a lost cause. I got access to it to see what was going on. Digging around, I found out why it was so slow. The apache powering the site was <em>woefully</em> misconfigured.</p> <p>The hardware the site was on was, I believed, more than capable of handling the load the site was under. Unfortunately, the software hadn't been configured in any kind of sensible fashion. I made the decision that time would be better spent fixing the software rather than moving to new hardware - which wouldn't have made a button of difference if the site was again misconfigured!</p> <p>After a pause while we had to reboot the server when it completely locked up, I got to work. I started by configuring the apache to allow many more connections and turned off keepalive, which was just an immediate fix to make the site a little more responsive.</p> <p>I then began work on putting nginx in front of the site. Nginx is the reverse proxy mentioned in the story. The idea is that it handles the connections between visitors' browsers and the website, and serves any static files (images/CSS/javascript), instead of the slower, more memory-hungry apache.</p> <p>However, while working on nginx, I noticed that at random intervals every 30 seconds to 10 minutes after apache was restarted, it would race to MaxChildren and the site would lock up. Very strange behaviour, which I initially put down to the high load. I figured nginx would fix it, so I kept an eye on it and restarted apache every time it locked up, while continuing to work on nginx.</p> <p>After a while, I got nginx in front of the apache. That made an immediate difference to the site performance, now that apache was only handling PHP requests <a class="footnote-reference" href="#id4" id="id2">[2]</a>. However, the random apache lockups continued.</p> <p>Arjen came on the scene somewhere around this point, and began looking at the database. Everything that Netspace claimed they did regarding database tuning, he did. Index optimisations, alterations to cache settings, etc. In fact, Arjen even <a href="http://www.brisbanephp.net/events/16760466/?eventId=16760466&action=detail">gave a talk about what he did at the Brisbane PHP & MySQL user group</a>.</p> <p>We began investigating the lockups in more detail. I've actually <a href="fixing-website-performance-issues-evidence">blogged about how they were caused and how we fixed them before</a> - "the case of the crashing website" was actually "the case of the crashing NZ Red Cross website". Once we fixed it, success! The site was running smoothly.</p> </div> <div class="section" id="the-story-vs-the-facts"> <h1>The Story vs The Facts</h1> <p>Any mention in the story of Netspace having tuned software is dubious at best. I honestly can't remember whether I installed an opcode cache, whether they did, or whether there even was one. However, the reverse proxy setup and apache configuration was me. The site was, according to Fletch, running on the existing hardware that it always used to run on, so I'm not convinced it was ever "put onto a new 64-bit Debian operating system" - and that wouldn't have made any difference if it had.</p> <p>Regarding the MySQL - apart from moving it to a separate machine (which may not have even happened - all I know is that it was on a separate machine when I got there) - all of the tuning was done by Arjen. He was MySQL employee #25, and now runs <a href="http://openquery.com/">OpenQuery</a>, a MySQL consultancy. There are tickets in OpenQuery's issue tracking system where he detailed all of the work he did.</p> <p>The funniest part about the whole story is this:</p> <blockquote> Within 36 hours of the site getting hit with a wall of traffic ten times larger than the solution was specified to handle, Netspace had reduced processing and memory load on the servers while being able to serve the higher connection demands from the world...</blockquote> <p>The solution was specified to handle? The solution was a horribly misconfigured apache, an unconfigured MySQL and no reverse proxy, as well as a broken search system (see <a href="fixing-website-performance-issues-evidence">the case of the crashing website</a>). The "wall" of traffic would only have had to have been 10 visitors at once to floor it.</p> <p>The solution Arjen and I put in place dealt with the wall with ease. When I logged out of the web server, load was 0.1, and it was using just 700M of ram - in comparison to the swapping-like-crazy, constantly-locking-up mess it was in (aka: load of 20+, all 8G of ram + all swap used).</p> <p>Gerard sent me an e-mail afterwards:</p> <pre class="literal-block"> Hi Nigel, Thank you so much for your help with Red Cross - we really appreciate it, and so do the folks at Red Cross. Your efforts have enabled Red Cross to accept over 7500 donations since noon yesterday - over $800,000 going directly to where it's needed. And that number will continue to climb. You seem to still have 10 open sessions on the redxprdww02 server - let me know if you still need them and I can kill them off if you're done. If there are any expenses or costs please don't hesitate to invoice them to us. Netspace Services Limited PO Box 404 Palmerston North New Zealand Thank you again for all of your help. Regards, Gerard </pre> </div> <div class="section" id="q-a"> <h1>Q & A</h1> <div class="section" id="q-nigel-isn-t-this-sour-grapes"> <h2>Q: Nigel, isn't this sour grapes?</h2> <p>I don't think so. There's no mention of us in the article. People will get the impression that it's Netspace that tuned the site if I don't say anything.</p> <p>I wouldn't have had a problem with it if the article had said that Netspace accepted the help of the team that worked on eq.org.nz, who did [insert all the configuration stuff here].</p> <p>ComputerWorld is a widely read publication, it doesn't seem right to let such inaccuracies go without correction.</p> </div> <div class="section" id="q-what-about-the-other-stuff-in-the-article"> <h2>Q: What about the other stuff in the article?</h2> <p>We only ever dealt with the website. Everything else wasn't us, and I know nothing about it.</p> <p>I never actually dealt with Charles Ranby, the Red Cross IT Manager. I wonder how much he knows.</p> </div> <div class="section" id="q-so-what-do-you-think-of-netspace"> <h2>Q: So what do you think of NetSpace?</h2> <p>I hope they simply forgot to mention they had so much help.</p> <p>Fletch was good to work with - responsive and helpful.</p> <table class="docutils footnote" frame="void" id="id3" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id1">[1]</a></td><td>I was actually connected to him at <a href="http://face.co.nz/">Face</a>, but Face and Netspace seem to be interchangeable - see how <a href="http://face.co.nz/our-work">Face claims Red Cross New Zealand as their client</a>, at the bottom of the page.</td></tr> </tbody> </table> <table class="docutils footnote" frame="void" id="id4" rules="none"> <colgroup><col class="label" /><col /></colgroup> <tbody valign="top"> <tr><td class="label"><a class="fn-backref" href="#id2">[2]</a></td><td>Unfortunately though, not as much as it could have, because the CMS they're using serves images through PHP scripts</td></tr> </tbody> </table> </div> </div>