The Forgotten Mistake that Killed Japan’s Software Industry

Japanese softwarehas problems. By international standards, it’s just embarrassingly bad.

We all know this, but what’s interesting is that there are perfectly rational, if somewhat frustrating, reasons that things turned out this way. Today I’m going to lay it all that out for you in a way that will help you understand how we got here, and show you why I am optimistic about the future.

And no, this is not going to be just another rant about all the things I dislike about Japanese software.

I am not going to waste your time or mine cataloging and complaining about the many, many bad practices, user-hostile design decisions, mind-boggling complex workflows, and poor development process that afflict Japanese software.

If you want details and debate about exactly how Japanese software falls short, or if you are just in the mood for some good old-fashioned venting about being forced to use it, check out Reddit or maybe Hacker News. This topic comes up pretty often there.

No, for the sake of this podcast I’m going to assume that we are all in agreement that on average, Japanese software. is just … awful.

That way we can spend our time talking about something far more interesting. We are going to walk though the economic events and the political forces that made today’s poor quality of Japanese software almost inevitable,

And by the end, I think it will give you a completely new way of looking at the Japanese software industry.   

You see, the story of Japanese software is not really about software. No, this is the story of Japanese innovation itself. The story of the ongoing struggle between disruption and control. It’s a story that involves war, secret cartels, scrappy rebels, betrayal, rebirth, and perhaps redemption.

How This Mess Started

So let’s start at the beginning. The beginning is further back than you might expect.

To really understand how we got here, we need to go back, not just to the end of WWII, but to the years after the Meiji restoration, the late 1800s, back when the Japanese economy was dominated by the zaibatsu.

Now, “zaibatsu” is usually translated as “large corporate group” or “family controlled corporate group.” While that is accurate, it grossly understates the massive economic and political power these groups welded around the turn of the 20th century.   

Japan’s zaibatsu were not corporate conglomerates as we think of them today.

You see, although the Meiji government adopted a market-based economy and implemented a lot of capitalist reforms, it was the zaibatsu, with the full support of the government, that kept the economy running.

And the zaibatsu system was almost feudal in nature.

The national government could, and did, pass legislation regarding contract law, labor reforms, and property rights, but in practice these were more like suggestions. In reality, as long as the zaibatsu kept the factories running, the rail lines expanding, and the shipyards operating at capacity, the men in Tokyo didn’t trouble themselves too much with the details.

In practice, the zaibatsu families had almost complete dominion over the resources, land, and people under their control. They were the law.

At the turn of the previous century, there were four major zaibatsu (Sumitomo, Mitsui, Mitsubishi, and Yasuda). And each zaibatsu had its own bank, its own mining and chemical companies, its own heavy manufacturing company, etc. But it wasn’t just industry, each of these zaibatsu  groups had strong political and military alignments. For example, Mitsui had strong influence over the army, while Mitsubishi had a great deal of sway over the imperial navy.

At the start of WWII, the four zaibatsu families controlled over 50% of Japan’s economy. This fact, when combined with their political influence, quite understandably, made Japan’s military government very uncomfortable, and during the war, the military wrested away a bit of the zaibatsu’s power and nationalized some of their assets.

After Japan’s defeat, the American occupation forces considered the zaibatsu a serious economic and political risk to Japan becoming a liberal, democratic fully developed nation. They targeted 16 firms for complete dissolution and another 24 for major reorganizations.

Rising from Ashes

Now, that was supposed to be the end of the zaibatsu. I say “supposed to” because those of you who know Japanese history understand that it never really happened.

Of course, many things changed. Important political and social reforms were implemented, the legal system was greatly strengthened, a lot of zaibatsu assets were nationalized, and the zaibatsu themselves ceased to be.

At least, officially.

You see, the zaibatsu were quickly allowed to restructure in greatly weakened , but very familiar, forms, as keiretsu.  This was permitted for two main reasons.

First, as the cold war heated up in the 40s and 50s, America’s idealistic vision for a democratic and progressive Japan took a back seat to the more practical and pressing need to develop Japan into a bulwark against Communism. And that meant prioritizing economic growth over social reforms. With these new goals in mind, both the American occupation forces and the Japanese government, quite correctly, concluded that having something like the zaibatsu groups would lead to faster, more predictable growth than tearing everything down and rebuilding from scratch.

The second important, and kind of surprising, reason was that almost no one in Japan really wanted to see the zaibatsu broken up. Not the politicians, certainly not the leaders of the zaibatsu, not the public at large, and to the endless frustration and confusion of western labor organizers, not even the rank-and-file zaibatsu workers and employees. In fact, at one point 15,000 Matsushita union members signed a petition demanding that the Matsushita zaibatsu not be broken up.

So in the end, important changes were made. Labor rights and contract law were strengthened significantly, and even more zaibatsu assets were confiscated. The traditional family holding companies were dissolved, but they were replaced by cross-company shareholdings  and interlocking corporate boards that achieved much the same result, but in a much more transparent and manageable way.

And so, most of Japan’s zaibatsu were allowed to morph into the smaller, less threatening, and much more manageable keiretsu.

Japan as a Global Innovator

In the same way that the zaibatsu defined the economic miracle that was Japan’s Meiji-era expansion, the keiretsu would come to define the economic miracle that was Japan’s post war expansion.

Today there are six major and a couple dozen minor keiretsu groups, and during Japan’s economic expansion, as much as possible, they kept their business within the keiretsu family.

Projects were financed by the keiretsu bank, the materials and know-how were imported by the keiretsu trading company, and the final products would be assembled in the appropriate keiretsu brand’s factory. And supporting all of these flagship brands were, and still are, tens of thousands of very small, exclusive manufacturers that make up the keiretsu supply chain — and the bulk of the Japanese economy.

And with the exception of a tiny handful of true startup companies like Honda and Sony, all of Japan’s brands that were famous before the year 2000 or so, are keiretsu brands.

And for those of you who think big companies can’t innovate, let me remind you that from the 50s to the 70s, these keiretsu groups began innovating, disrupting, and dominating almost every industry on the planet; from cars, to cameras, to machine parts, to steel, to semiconductors, to watches, to home electronics, Japan’s keiretsu simply rewrote the rules.

But how did keiretsu do in the world of software development?  Well, pretty darn well, actually.

It’s important to remember, though, that the software industry in the 60s and 70s was very different than it is today. The software development process itself was actually rather similar. Fred Brooks wrote The Mythical Man Month about his experience during this era, and it remains as one of the best books on software engineering and project management today.

But the way software was bought and sold was completely different. In the 60s and 70s, software was written for specific and very expensive hardware, and the software requirements were negotiated as part of the overall purchase contract. Software was not viewed so much as a product, but more like a service, similar to integration, training, and ongoing support and maintenance. It was usually sold on a time-and-materials basis, and sometimes it was just thrown in for free to sweeten the deal. The real money was in the hardware.

Software in this time (both in Japan and globally) was written to meet the spec. It did not matter if it was creative, innovative, easy to use, or elegant, it just had to meet the spec. In fact, trying to build exceptional software in this era was considered a waste of resources. After all, the product had already been sold and the contracts had already been signed. The goal back then, just like many system integration projects today, was to build software that was just good enough to get the client to sign off on it as complete.

Software that met the customer’s spec was, by definition, good software.

Japan’s keiretsu did well in the age of big-iron. Although Fujitsu, NEC, and Hitachi never seriously challenged IBM and Univac’s global dominance in the 60s and 70s, they did pretty well in mini-computers and large office systems.

They were innovators.

Japan Turns its Back on a New Industry

However, when the PC revolution arrived in the late 1980s, Japanese industry as a whole was hopelessly unprepared, and not for the reasons you might think.

The reason Japanese software development stopped advancing in the 1980s had nothing to do with a lack of talented software developers. It was a result of Japan’s new economic structure as a whole, and the keiretsu in particular.

As a market, personal computers were something fundamentally new. Sure, the core technology and the hardware were direct continuations from the previous era, but this new market was completely different.

The PC market quickly coalesced around a small number of standardized operating systems and hardware architectures. Keiretsu did pretty well in the hardware side of this market, making some really impressive machines, particularly laptops.

But a market for non-spec or “shrink-wrap” software was something new to everyone. It required delighting the customer, and knowing what they wanted before they did. It was the kind of challenge that the keiretsu of the 60s and 70s would have thrown themselves into whole-heartedly, innovated aggressively, and then dominated.

But things in Japan had become very different in the 1980s.

Here was a chance to define and lead a new global industry. A chance for keiretsu to build a software industry from the ground up.

But, wait a minute. Why should they?

Sure, back in the 60s when Japan’s economy was small, survival required looking outwards, competing globally, making long-term investments, and innovating to make the best products in the world.

But this was the 80s! Japan was the second-largest economy on the planet and in the middle of the largest economic boom the world had ever seen. This was the era of Japan as Number 1, with economists predicting Japan’s GNP would be larger than America’s within a decade.

With such a lucrative, and pretty well protected, market right at their fingertips it made much more sense for the keiretsu to focus on the easy money rather than to take risky and expensive bets on an uncertain and emerging global market.

Each keiretsu group had their own technology firm who started selling PCs and software, some to consumers, but the big money was in corporate sales.  And since the keiretsu liked to keep the business in the family, these technology companies grew and profited by selling to their captive customers within their keiretsu group. And just like before, they made real money integration, and customization.

An unfortunate result of this is that the big Systems Integration companies or “SIs” emerged as powerful players, and Japan’s software firms never had to compete globally, or with each other.

Japan simply missed the opportunity to develop a globally relevant PC software industry.

The Beginning of the End of Innovation

Japan’s software industry in the 80s and 90s remained much like it was in the mainframe area. The software had to be just good enough for the client to sign off on it, and since they were largely captive clients unable to look outside their keiretsu group for support, that was a very low bar indeed.

But hey, as long as the economy was booming, no one minded spending lavishly to keep all the work in the keiretsu family, and all those little software defects could always be fixed in “phase two” of the project.

Software development was an exercise in box checking. You implemented a feature once the customer had asked for it and the contracts had been signed.

This not only caused Japan to miss out on the global software industry, but it marked the beginning of the collapse of innovation across Japanese industry.  Over the next 30 years, software would become a key driver of both innovation and efficiency. But by outsourcing their IT strategy to a single integrator, they had tied themselves to an anchor that would ensure almost every industry fell further and further behind the technology curve with each passing year.

Japan still has not recovered from this. Even today most enterprise systems are decades behind their global competitors. But, as we’ll see a bit later, things are happening now that could enable a quantum leap forward in Japan.

Life as a Developer in Japan’s Dot-com Bubble

So, what was it like to be a software developer in Japan in the 80s and 90s?

It was pretty bad. Software development was considered rather low-skill work. It didn’t pay well and was viewed as a kind of clerical work. The job was simply to write software that was close enough to whatever sales had promised the client while they were out drinking last week.

New hires with degrees in literature, business, or law, or whatever were rotated through software development for a few years to give them a sense of how different parts of the company worked. There was no real career path in software development. I mean, maybe you could move up into project management or over into sales, but if you were still actually writing code when you were 30, people kind of wondered what went wrong.

Of course there were some great, even visionary, software developers in Japan at that time. I knew some of them. People who wanted to make computers do new things. People who saw how technology could disrupt other industries, and developers who simply had a passion for making software that delighted users.

There were plenty of developers like that in the 80s and 90s. They were miserable.

Interestingly, hardware engineers were viewed very differently. Both then and now, hardware engineers are highly respected in Japan. Engineers are some of the most admired people at companies likeToyota, Mitsubishi, and Sony.

So, perhaps unsurprisingly, hardware innovation continued at a furious pace during the 80s and 90s. Products like the walkman and the Nintendo consoles achieved global success and the domestic market was filled with electronic diaries, dictionaries, and planners that were way ahead of what was available in the West.

And of course, eventually i-mode. Japanese consumers were sending email and browsing the web years before the Blackberry was released, and almost a decade before the iPhone.

Falling Down

But the rest of the word was moving in a different direction. The rest of the world was moving away from dedicated hardware and towards innovative software running on standard hardware platforms. As Marc Andreessen would later point out “Software was eating the world.”

As the dot-com bubble started to inflate, Japan began to realize they needed talented software developers, but without a software industry that actually valued software developers, companies had no idea where to find them. The best talent was usually unrecognized and trapped at the lower levels of the org chart. There was not much of a future pipeline. Since software engineering was not a respected or profitable career, few students opted to pursue it.

Some did of course, but these were the people that just loved programming. It was like becoming a musician or a manga artist. It was great that you are following your dreams. You might make it, but the odds were not in your favor.

When the foreign software companies started crashing into Japan in the 90s, the domestic industry could barely put up a fight.

The dot-com boom of the late 90s was the first wave of venture-funded, disruptive innovation in Japan, but it was not yet time for Japan’s software developers to step into the spotlight.

The successful founders of that era were mostly well-connected or incredibly scrappy businessmen. The general opinion of software developers had hardly changed at all. They just weren’t the kind of people you would trust to run a company.

I started my first Japanese startup during the dot-com boom, and at that time, I think the fact that I was a technical founder was even more unusual than the fact that I was a foreign founder.

Of course in one sense, the dot-com boom was an amazing time to start a software startup in Japan. You could call up almost any talented developer you knew, let them know hat they would be working on something important, that they would have meaningful input into product development, they’d be on a team that cared about code quality, and that their skills would be respected … and yeah, they’d want to come on board.

The Lost Decades Were Never Really Lost

The late 90s and the nights is often referred to as Japan’s “Lost Decades.” It was not a good time for the keiretsu companies. Not only did their power continue to weaken, but increased scrutiny of their cross-shareholdings and financials, and the merger of several banks across keiretsu lines meant that business as usual was over.  And pushing the knife in deeper, all of the implied economic and social guarantees that the keiretsu system was based on began to unravel.

In previous decades, Japan had focused on exporting domestically made goods, but now, not only was the domestic market attracting a greater focus, but Japanese industry began moving production out of Japan into cheaper overseas markets.

This was considered a betrayal by thousands of mom-and-pop manufactures who had spent their lives (or sometimes generations) as a highly integrated and specialized part of a single keiretsu’s supply chain, and who now found themselves suddenly cut off.

Japan’s famous lifetime employment system effectively ended during this period as well. It made sense for corporate groups to promise lifetime employment and predictable promotions when profits kept rising and labor was scarce, but now faced with mounting losses, corporate Japan began walking back all these implied promises.

This was a shock for Japan. It was a breach of the social contract that holds everything together. If hard work and loyalty would not be rewarded, then why dedicate your life to the company? To the distress of pundits and politicians, many young Japanese started saying they had no interest at all in joining the corporate world.

But the truth is, especially now that we can look back on it, these decades were not really lost decades at all. Growth slowed, and the changes were incredibly painful, but they were absolutely necessary to set the foundations for the coming wave of startup innovation and for Japan’s software developers to finally get the respect they so deeply deserve.

The Triumph of the Japanese Software Developer

I mark 2010 as the year Japan’s software developers finally started stepping into the spotlight, although things started moving a bit before that.

There were two triggers that led to this development. First, the emergence of cloud computing and second, the introduction of the smartphone. Although these were both technological developments, it was not the technology itself that led to the change.

Cloud computing drastically reduced the capital and time required to start a startup. In the dot-com era a decade before, starting an internet startup required purchasing racks of servers and paying system administrators to keep them running, but suddenly fully configured, maintained, and secure servers could be had for a few cents per minute — pay as you go.

Suddenly Japan’s software developers didn’t need to explain their idea to a VC and convince them that it would sell. They could just build things and get people to start using them and start paying for them. And that’s just what they did.

The other important development was the introduction of the iPhone in 2007 and Android a year later. Not just because of the technology, but because of how it changed the software business model.

Japan’s i-mode was years ahead of the West when it first came out, but getting your app on i-mode was largely a matter of lengthy negotiations with the telcos for one of the few highly-coveted slots on the menu. The smartphone ecosystems were different. Anyone who could develop an app of reasonable quality could deploy and sell it. There were no business connections, exclusive negotiations, or revenue commitments required.

2010 marked the beginning of the end of the software startup gatekeepers. As more and more talented developers realized how easy it was to start a startup, more and more started choosing startups over the traditionally low-status career path at large companies.

This, combined with a large dose of Silicon Valley glamour, has complexly transformed Japan’s image of the software developer. Software developers are valued and respected today. Unlike the dot-com days, both startups and enterprises compete aggressively to recruit and retain talented programs, even though there are a lot more of them today. Thankfully, people also talk a lot more about code quality.

Of course, this attitude shift was much broader than just developers. With the safety net of lifetime implement and guaranteed promotions removed, people have had to become less risk averse and more innovative. Those workers who had rejected corporate life, became freelancers and formed the core of Japan’s flexible startup workforce, and some of those tiny supply-chain companies began to rethink their business models.

This brings us to the start of the Disrupting Japan podcast about eight and a half years ago. We’ve talked to the innovators and followed the development of the startup ecosystem together during that time.

So, Where Do We Go From Here?

As we talk here together at the start of 2023, what does the future look like for Japanese software?

Japan has had a lot of catching up to do over the past fifteen years. After basically sitting out the global PC and dot-com revolutions, Japanese software developers have been making up for lost time and in the startup space. Japan is developing a competitive software market in some areas, but on average, there is still a long way to go.

Japan’s once dominant Systems Integrators will continue to see their power decline. Their customer lock-in is fading fast, and B2B SaaS software startups are letting Japanese enterprises leapfrog to modern IT systems for less than costs to maintain their SI-run legacy systems.

The SIs won’t disappear, of course. There will always be a need for good systems integrators, and the more forward thinking ones are already trying to reinvent themselves. However, the days when the SIs dictated their clients’ IT strategy are coming to a close. That is a very good thing for Japanese software, Japanese startups, and Japanese competitiveness as a whole.

The Kishida administration has made startups a national priority, and the importance of quality software and software startups in Japan has never been higher.

Even the old keiretsu firms have come around. They are increasingly looking to software startups to supplement internal R&D though both M&A and through long-term partnerships. In fact, last year Keidanren, Japan’s largest business federation, an organization that was one of the main architects and drivers of Japan’s post-war economic expansion, called on its member companies to greatly increase their startup investment and partnerships.

I am optimistic. As always, things will develop differently in Japan. In the same way the zaibatsu defined Japan’s Meja-era economic miracle, and the keiretsu came to define Japan’s post-war economic miracle, from some new combination of startups, enterprise, and academia will emerge something that will define the next economic miracle.

Today is a very good time to be developing software in Japan.