Wednesday, February 4, 2009

Zen and the Art of Open Source Hardware - Day 2

Today was a long day! Lots of time to struggle and think... and so I'm starting the first of my "diary entries" by pulling together the complete state of my thinking so far. I'm going to focus especially hard to cluster all of the ideas and concepts I've collected in my notes into "topic areas." I want to try to identify a common language, and set of terms that I think might be helpful for me for the rest of the week... and unfortunately it means more problems than answers at this point. Here goes nothing...

If Open Source hardware means open economics, and open community and business models, then it also means transparency. Transparency of file formats, circuits, schematics, processes, community, communications, and even cost and profit structure.

Open source hardware involves real, tangible products, processes, and people. It's not just "imaginary", it's real stuff. Sometimes it feels great, like when someone other than yourself holds something you've made, and they say, "that's cool!" And sometimes it can feel bad, like when you get an email that says, "this doesn't work" or "I didn't get what I thought I paid real money for!"

Recently, for instance, Luca bought an Ethernet Shield from the store, but the store had been out of stock for a week and a half until I got more from the guys at My auto-emailer script on the site has been broken unbenownst to me, and so Luca wrote in and said, "what the heck, where's my stuff!" That simple little email contained so much of why open source hardware is different that open source software: real stuff comes with real money and real people and emotions.

Conceptual Preconditions of Open Source Hardware
So here's some groundwork on the "conceptual pre-conditions" of Open Source hardware. What is it that allows Open Source hardware to work or not? What are some of the major concepts at play?

1) Shared Infrastructure
Open Source businesses and hardware companies are reliant on shared infrastructure. For liquidware, that means shared "virtual services" (I'm going to ding myself every time I use stupid buzzwords) like Paypal, Google's adwords, even web servers and bandwidth. It's a lot like in Zen and the Art of Motorcycle Maintenance, you can be as free as you want to be on a motorcycle driving across the country, but at the end of the day, you're still reliant on the government because at some point, they paved the roads beneath you. And you need the roads to be free. Open Source hardware needs similar tangible shared infrastructure. But who's going to pay for this?

2) Real Time
An Open Source hardware community, project, club, group, or company takes real time. Real time for me means sneaking around the day job, in between meetings, when traveling on public transportation, pretty much any time I can. The funny thing is, so far, nothing I've ever done at Liquidware, Antipasto, Arduino, or anywhere else, has reimbursed the value of my time, and the curious thing is ... that's perfectly ok. In the book Predictably Irrational, Dan describes a phenomena by which people are willing to work or barter or trade huge amounts of things because they'd do it just to help someone else out. Reciprocity, perhaps. But as soon as you put a price tag on their time, they switch into "economic" mindset, and start thinking, "there's no way I'd do that, because my time is worth more than that." Well, Open Source hardware faces this problem every day, because things immediately cost money. Ever part bought in a catalog costs money, just as every hour assembling something "costs" labor. And every time someone's thinking "hmmm that feature would be soooo cool" they're also hit on the head with what I'm going to call the "doh! now I have to reach to grab my wallet and buy this part from Sparkfun or Digikey" factor.

3) Long tails
So there's this guy Chris Anderson, and Clay Shirky, and a handful of others, and they all have the same basic idea: how can we cram as many pretty charts and graphs into a book with lots of jargon, and confuse the heck out of a reader? :-) That too. They also had this other idea that some people like to point to iTunes for; that individual needs can be met in a "long tail" system, because someone can get precisely what they want, even if there are only 1 or 2 of them out there. Perhaps a benefit of Open Source hardware is that if you want a slight permutation (like a Type B mini rather than Type B big adapter on the Arduino), you can take the source code files, and quickly and easily change it. You could even make up your own adapter, and then build one or two boards, and you'd be the only one who had one, but it would do exactly what you wanted. But! That faces a few problems to follow...

4) Experience and scaling
There's this curious little phenomenon I've read about recently called Henderson's Law, or the "Experience Curve." It states that the more volume of something you make, the cheaper it is on a per-cost basis. It seems that everything tangible produced has fixed and variable costs (just like every piece of circuitry has fixed and variable costs too). As someone builds more and more of something, they learn how to make it better, and more efficiently (e.g. by building machines or equipment like pick-and-places or solder-flow ovens or by designing better processes like assembly lines or Mike and my Illuminato assembly station). Normally, this is a good thing. But for Open Source hardware, it's a very bad thing. That's because building something in the long tail at a quantity of 1 or 2 costs just as much as designing something in the "fat" tail (err... whatever you call the left hand side of those 1/x graphs)! The point is that when building and designing a new Arduino (e.g. Illuminato?) it costs literally just as much to build *the first one of two* you build, as it does to build *the first one of thousands* ($300 if you're lucky and get it right on the first try, to $2100 if you're like me and you miss a few traces each time, and reverse the pin headers on the second to last try even though you triple checked them). After you get the design right, the decision to build higher and higher quantity is completely independent, which leads to the next point below. But the more you build, the lower the cost, so there's a penalty to customization. If the "open source hardware scaling penalty" were an equation, it would look something like this: (Between $300-2100 sunk cost) + (difference in price you could have gotten if you could have found others who wanted your one-off customized product) * (volume you actually made). Believe me, this is a very real, measurable, and perceptible cost, and for the Illuminato project, the scale penalty was somewhere around $2000 (but I was willing to spend it because I figured I did it to learn electronics, and so it was worth it to me!).

5) Capital inventory and opportunity cost
I've never taken a "real" traditional economics course, but from what I understand of "opportunity cost" it goes something like this: if money's tied up doing one thing, it could have been doing something else, so pick whichever one makes the most money. Ok. There are two problems with this in the "community commons": 1) profit? who said anything about profit, and who's making it, and where do I meet them? :-) where does it go, how's it accounted for, and who pays taxes on this?, and 2) where does the initial cash come from to build the inventory? Holding aside the first one for now (more on this later though!), the second one is a big problem for open source hardware. In order to achieve lower per unit cost for everyone, especially by building higher inventory volumes, someone has to pay the manufacturers and component companies to buy large volumes of parts from them. So someone has to take a "risk" in order to build tons of something, in order to produce them at lower cost on a per-unit basis. What happens if I decided to build the Illuminato in a really ugly color and no one liked it? Then the 50 I built (on top of the ~50 my mentor bought) would have been a complete waste, and I'd be out a lot of money! Who benefited from my misfortune? The PCB manufacturer, and the parts suppliers! Because they got "guaranteed" sales when I bought my parts from them. That brings me to...

6) The open source "value chain"
I have a couple of business books that I bought from the business book store up in Boston. They're very dense and hard to read (err hard to stay awake reading, to be more precise), and if you're not careful (and completely attentive!) you'll find yourself thinking that "synergy" and "vertical integration" are acceptable terms that have real meaning... they don't! (and if you already thought electrical engineers don't have too many friends ...which is true to a dataset of n=0001b (me)..., a surefire way to alienate everyone is to be an electrical engineer that's spouting business jargon nonsense!) But I digress. The Open Source hardware value chain looks something like this: idea->schematic circuit->layout file->pcb manufacturing supplier company->components and parts company->assembly company->testing->delivery and shipping. In order for true, long-tail, scalable "open source" hardware to work, every step along the way has to be equivalently scalable. Combining the points in the previous paragraph (5) with the point in paragraph (4), the problem is that the component manufacturers don't pass scale savings on their parts unless you commit to buying large quantities. That forces prices in the "long tail" at low quantities artificially high. And that "kills" or certainly limits the long tail customization. Borrowing some concepts from a few biology books I've read recently, it would appear that "diverse speciation" comes at a high cost to the environment. Assuming for a second we believe in a God that created the animals (I don't, but that's another story), that means that it cost God more money to make more diverse species of animals, and so if God wants to keep the total cost of making earth low (after all, he needed to save some money for other worlds), he tries to minimize the number of species. Darwinians might say there's an "invisible force", Adam Smithians might say "invisible hand". As long as any one step in the "open source value chain" does not pass along at-cost economic scaling, there is an "environmental fitness function" trying to shorten the long tail. (PHEW! That was the brainiest thing I've written in years) If my mom had asked me to paraphrase, I'd say, "as long as there's a big company somewhere in the development process that's trying to make money, they are inadvertantly forcing the number of open source projects to a minimum."

6) Ownership and assets in the commons
Can assets be shared by a pool of people, geographically distributed, and virtually connected? I start to imagine fanciful big-wigs talking about "micro-finance" and "peer-to-peer loans" and my eyes glaze over, because I think those are just fancy terms for what could be simpler concepts. But assuming point (5) could be solved by having lots of people pool their money into a single order, and therefore reducing the cost to everyone (hmmm... another example is that t-shirt company, Threadless... but they're a little too trendy for my tastes), the result would be a shared opportunity cost. Depending on which economic jurisdiction you're occupying at any given time, large companies typically get tax shelters and protection in the form of "depreciable capital assets" when they build large stockpiles of shared, pooled assets. The assumption is that they built it, needed it to survive, and so the government encourages them to grow by encouraging institutions to build up assets. But there aren't really any corporate forms that protect distributed, commons-like, shared assets. Communism? Socialism? (ha - insert your favorite anti-capitalistic slogan here). The "common thread" (pun intended) is the virtual firm that the government protects, if not incents to share and stockpiled assets. Can this be done on the internet? Can this be done with distributed accounting? Who knows!

7) Time barrier thresholds

It doesn't really matter if an open source project is "open source" if it takes an infinite time to understand the files, an infinite time to edit the files, and another infinite chunk of time to compile and turn those files (e.g. schematics) into a real product. To give a perfectly real example of this, if Linux were written in assembly code instead of C, it would take me about 30x longer to understand, and a simple change to the core kernel would take me many hours to do (but I could still do it!). At some point, the "I have better ways to spend my time" principle would come into play, and prohibit any productive progress, and I'd probably just turn around from my desk chair and instead watch reruns of The Office. Taking it a step further, if Linux were written in machine code (and, technically, all software, including Microsoft Vista is machine code "open source"), it might take 300x longer to understand, and a simple change might take days if not weeks to implement. Well, the same is true in hardware. Obfuscation of schematics serves the same purpose. It's just like the old days of running HTML source code obfuscators on websites you didn't want people to rip off. People did it to increase or magnify time barriers, as a protective technique. This feels less "open source" than it could be in the "best of all possible open source worlds." Likewise, it seems that "open source hardware" should be conscious of, if not constantly minimizing the amount of time it takes an individual person (not an instituion!) to pick up something and tweak it.

8) Access

From a philosophical perspective, "open sourceness" is a function of the tools at your disposal. Someone with a brilliant reverse-engineering compiler sees a compiled binary as "open source" just as someone with a set of lock-picks sees locks as "open source." Likewise, if someone built the Illuminato with a commercial grade schematics program that only contains proprietary file formats, yet publishes those files in the public, does anyone really consider this "open"? Well, yes and no. By comparison, another company that could afford to pay $10,000's on tools would see these files as "open source". In a similar way, an extreme nerd (*cough* Chris *cough*) could consider a picture of a schematic to be sufficiently "open source", because he's so quick at reverse engineering and laying out parts, that a picture is just as easy for him to use as a raw schematic file... up to a point. At the hyperbolic extreme, I suppose one could say that someone with a scanning electron microscope and atomic realigner would theoretically view *everything* as open source hardware, because they could immediately reverse engineer any physical, tangible object, and manipulate individual atoms in that object to tweak, reconfigure, or alter it. At the risk of irritating my philsophy friends, I suppose someone could even go as far as to say open source was "socially relativistic."

9) Incrementalism
Academics, and professors like to say "a corollary of X is Y" as a way to obscure a more simple phrase like, "X is a lot like Y." Incrementalism is a corollary of time barrier thresholds. :) Imagining for a second that we want to make an Open Source hardware project that mimics Linux or Wikipedia, what would it take? We'd want some incredible changelog and revision systems, thousands of core contributors from around the world, and contributions ranging from functional enhancements (like faster virtual memory paging algorithms) to architectural enhancements (like the holy wars over whether to pre-compile or runtime load-modularize a driver). This last piece I'll call "incrementalism", because that's what it feels like when actually doing it. It feels like biting off a small piece of the project (e.g. Linux), playing around with it, and then "checking it in" again when you're done. A bit like a "github" for circuit files. But to promote this form of work-process, or "productivity", we'd need a distributed file format. I've talked to some of the guys at EagleCad, and even NI, and I'm greeted with the same response... "what's open source?", "who's git?", "where did you get my number?", "I'm on the national do not call list no I don't want to switch long distance carriers." So I suspect it's not going to originate there. But until there's a structural, and architectural schematic-writing file format that encourages "social incrementalism," open source hardware isn't going to see the same prolific productivity that makes traditional magazine and print editors excited...

10) Inclusion and completeness

Every list has to have ten items because humans have ten fingers, and have obsessed about it, and even went as far as creating their number system in base 10. This list ends at 10 because it's getting late and I have a long way to drive tomorrow. Right now, the "open source" hardware community is a distributed smattering of nomadic tribes, wandering the web, picking up fruits and berries ("Arduino" even sounds like it ought to be a small red berry that looks tasty, but is probably tart, and makes your mouth pucker and they put it in flower decorations at Christmas time). At some point, the community will need to coalesce and start to hunt in packs and tribes, and maybe even get ambitious and go after a woolly mammoth from time to time (do NOT watch the movie 10,000 BC the timeline is totally inaccurate). Who are the woolly mammoths and where will they come from? Will they be academic grants, nice guys with extra money to donate to good causes, companies with donation budgets, or others? And when will organized civilizations start? How will the system evolve?

Wrapping up
Well, I don't normally aspire to be terribly removed from practicality, but every once in a while it helps me organize my thoughts by conceptually structuring what it is I'm even thinking about. I'm sure I've missed plenty of ideas, but I'm hoping this will start me off in at least in a productive direction. And at the minimum, now that I've called out these concepts, I don't have to regularly refer to them from now on, I can just link to this post, and save everyone from here on out a ton of unnecessary and weighty conceptualism in future posts!

Oh, and I desperately need to include some pictures in my blog posts to illustrate my ideas better, lest I get too wordy! I'll try better next time...



Matt said...

hmmm... i can't sleep :) lots of miles to drive tomorrow, so that's bad news already...

Greg said...

As a musician turned web developer who started tinkering with Arduino a couple of years ago and is now gearing up to go to grad school in it, I'm not 100% sure about the arc of development you're picturing.

The PC revolution didn't make it easier for individual computer enthusiasts to gather together in order to create and maintain their own mainframes. It made a crappy, nearly featureless computer (the Altair) cheap enough that they could actually have their own.

Similarly, I don't think the outcome of this revolution is necessarily that the current practices of the high priests of electrical engineering gradually filter down to laymen. Instead, a new generation of tools will emerge which are crappy enough to be widely affordable and featureless enough to be widely usable. The result will be the rise of an enormous collaborative community that is wildly more diverse in their interests, abilities, and aims than the previous class of experts. That community will reinvent all the previous functions fulfilled by the priests using their new crappy but accessible technology. Just as spreadsheets replaced giant accounting programs and crappy Pagemaker layouts replaced professional typesetting.

What we have to do is figure out how to move physical computing from the Altair to the Mac. If the Arduino is the tool that finally makes this stuff cheap and accessible to the geeky hobbyist types like us (i.e. the Altair equivalent), what is the "killer app" that finally makes some kind of general purpose physical computing device desirable to the masses the way VisiCalc did for the Apple II and layout/paint programs did for the Mac?

All of that said, as a non-EE artists/web programmer type, I'd love it if it was MUCH easier to learn some of the more advanced stuff. I'm pretty far along on this stuff. I've figured out enough to program my Arduino clone to ring a bell when I commit to git: I've even written a framework that lets me program the Arduino in Ruby: However, getting to the next level with the hardware -- understanding I2C and EEPROMs and the Arduino internals and how to design and build a PCB, etc. -- these things are non-trivial and there's no information on the topic that doesn't assume I'm either a 16 year old kid "Maker" doing my 10th grade science project or a full fledged Electrical Engineer.

Matt said...

@Greg - that was awesome! I think you're right in many ways - there's definitely a perspective bias going on. The funny thing is that Matt (the other one) is actually a promusic guy (sound engineering/remixing) who got into programming a long time ago, and has slowly migrated into hardware and electrical engineering over time. Meanwhile, I was actually a part time "musician" a long time ago - I played violin, but got frustrated until I found an electric violin, and discovered how much fun it was plugging it into all kinds of amps and remixers :)

Matt said...

@Greg - forgot to add this point too: regarding killer apps, I've always personally thought it was saving me money buying tons of gadgets over and over again. Chris thinks it's learning programming, Mike I think always thought it was making open source software tangible. I guess that means there isn't a clear killer app just yet, but I think you're right. There's definitely an order of magnitude difference between the old PC's and the modern ones, which was almost over-night, and then it hasn't really continued. Maybe it's a step function. I wonder what the order of magnitude increase in electronic circuitry and physical computing might be?

Anonymous said...

Hi there, interesting stuff.
Although I think Open source hardware is a quest I think it inevitably becomes the winner. One of the reasons is that in the end, although it's about hardware, the design itself is still a digital file. Since software is getting better all the time, it will become easier to change and improve these digital files over time. Apart from that the sofware is free where it used to be very costly.

And let's not forget production, there is new generation of flexible production machines such as 3D printers coming up. These will change the way economnics work. More people will print and improve.

Go for it!