Top
Best
New

Posted by chmaynard 9 hours ago

What Sun got wrong(bcantrill.dtrace.org)
463 points | 259 commentspage 3
time4tea 2 hours ago|
First work computer was Sparc IPX, a brief dalliance with HP D and K series, then had various Ultra, and servers were all sorts including E10k then various SunFire, and the Fujitsu Prime Power.

They were quite fun.

But can confirm, buying Sun was PITA.

nielsbot 7 hours ago||
I worked there right before they sold to Oracle for a bit—my impression was a lot of brilliant engineers but no unified product story.
adamsb6 3 hours ago||
Wild that around that same time one of my fellow students at university managed to get a free trial of a Sun Thumper shipped to him. It arrived to our campus workplace on a pallet. Just to mess around with. No way was he going to actually purchase the roughly $70,000 machine.
matt_heimer 6 hours ago||
Sun was just never really setup to sell lots of small servers. You have different pipeline when your goal is to sell million dollar E10K servers.

I got to see a server installed via crane because they couldn't go up in that building's elevator.

chambers 2 hours ago||
Sun couldn’t handle the cold. Obviously in terms of the dotcom bubble burst, inobviously in terms of the company’s character. The things people loved about Sun (giving Java and NFS away, treating engineers as the point of the company, McNealy's loyalty to his people) were exactly the things that kept it from doing what it needed to survive. Very warm, very caring.

Oracle expressed the opposite. Within a year or two Larry killed OpenSolaris, let the MySQL, Hudson and OpenOffice communities fork away, sued Google over Java, and watched Gosling and most of the engineers who embodied Sun walk out.

The part that rankles the most, and undermines many a retrospective I think, is that Larry the lawnmower turned out to be a better steward of Java & MySQL than Sun was. Coldbloodedness is a survival trait.

justin66 9 hours ago||
Having read a lot of these things from ex-Sun employees, I dearly want to read one with the topic my department is the one that fucked up Sun. I don't think I have seen a topic like that yet.
bcantrill 8 hours ago||
I actually haven't read a ton of these, so would love to be pointed to what you have found. But yes, everyone will be biased, and I am too. I think that the organization that I was in fucked up plenty (when I joined the company in 1996, SunSoft was led by truly one of the most incompetent executives I have ever encountered, a bumbling oaf that Sun kept putting in charge of different business units, letting him asphyxiate them one by one), but the fact that everyone also sees something else wrong probably tells you a few things: everyone at Sun earnestly wanted the company to succeed; Sun did a bunch of stuff right; Sun did a bunch of other stuff wrong.

Ultimately, the leadership of the company needs to bear the responsibility for the outcome -- and I think it's pretty telling that Sun's executive leadership basically went on to do more or less nothing after Sun. I loved Sun -- but it was a breeding ground for engineers, not executive leadership.

justin66 8 hours ago||
I'm reluctant to call anyone out. I liked Sun a lot, as well as the people I've encountered who worked there. There was a thread between a couple of ex-Sun guys here on hn years ago - I can't find it right now - which made me feel there was a humorous and very human pattern in Sun postmortem comments: "man, that other team really killed Sun, we were doomed after that thing they did." I was pretty curious what the people on that other team (the SPARC microprocessor designers, in the case of that comment thread) might have said in their own defense, where they might have spread some of the blame, and so on.

(perhaps it doesn't do your rather full blog post justice to compare it to something considerably less complete like a comment thread)

> Ultimately, the leadership of the company needs to bear the responsibility for the outcome

I emphatically agree. Maybe it goes without saying but with Sun, multiple generations of leadership, I think.

jhbadger 8 hours ago|||
I seriously don't know what Sun could have done to survive - once Linux was stable, running Linux on commodity hardware was just a better solution than anything Sun could offer.
toast0 8 hours ago|||
I've never spent much time with Sun equipment, but the hardware and OS had features I haven't seen on commodity hardware with Linux. IIRC, Solaris could deal with an unrecoverable ECC error by killing the process that had the bad memory. And I think they had hot swap memory on some systems.

ZFS was pretty unrivaled for long term storage for a long time. Solaris zones were containers before they were cool. Dtrace is pretty neat. Etc.

Maybe they could have survived with their OpenSolaris push driving low end adoption and high end customers buying their hardware. Pretty sure they would have had to migrate to x86, but they were dabling in that.

Maybe make some low end commodity x86 servers. Sell enterprise a fully engineered big ass database server and a bunch of commodity web servers. Doesn't have to be exceptionally priced, just good enough to let customers have single vendor server purchasing instead of getting the big stuff from sun and the little stuff from dell or hp.

More systems consulting? Sun had great engineers, but you don't see too many war stories about them fixing stuff for customers (maybe they keep quiet though).

cyberax 3 hours ago|||
> IIRC, Solaris could deal with an unrecoverable ECC error by killing the process that had the bad memory. And I think they had hot swap memory on some systems.

Both of these are supported on Linux now. The kill-on-ECC is handled by hwpoison: https://www.kernel.org/doc/html/v5.1/vm/hwpoison.html and the hot-swappable RAM has been present for quite a bit longer.

toast0 1 hour ago||
I saw references to Linux support for hotswappable ram a long time ago, but I've yet to run into a commodity server that supports it. And honestly, I've yet to run a system that needed it nor did I have systems where are hard stop from an unrecoverable ECC error was a problem.

I'm not terribly surprised that Linux grew support for poisoning pages eventually (and I appreciate the link!), but I believe Solaris had memory retirement in Solaris 10 (2005), and Linux got page poisoning in 2009. If you were running workloads where it would be important, then Solaris was better for you at that time, unlike the claim from the poster above that

> once Linux was stable, running Linux on commodity hardware was just a better solution than anything Sun could offer.

I think only a curmudgeon would claim Linux wasn't stable in 2005. Although certainly the userland could use less churn. :P

cyberax 20 minutes ago||
Linux is not a monolith :)

There was (and still is) a bunch of niche Linux kernels. I remember reading about a team that used Xen to do triple-redundancy with voting for Linux. They virtualized _all_ the non-deterministic IO paths (including rdtsc) and observed the outgoing network packets, ensuring that they are completely identical. I think this was around the mid-2000-s timeframe.

But yeah, it was all super-niche. The consensus now is that you should design your systems to be fault-tolerant and self-recovering, rather than depending on perfect hardware functionality. And for everything else IBM still exists.

monknomo 5 hours ago|||
zfs + solaris zones were amazing
keeda 6 hours ago||||
Sun clearly suffered some from being a bit too early to what would become cloud computing —- didn’t help that their tagline was “the network is the computer”, which IMO was a bit too obtuse, because I didn’t really understand what it meant (sounded like a violation of the 7-layer OSI model lol) until I saw actual cloud computing come about.

But as some sibling comments indicate they already had a lot of the software infra for distributed computing the way it eventually came to be done today, with SOA/microservices, service discovery, or serverless lambdas etc. via frameworks like EJB or JXTA. They were admittedly pretty cumbersome, like the contemporary equivalents COM/DCOM, and made the fundamental mistake of trying to abstract away the fallacies of distributed computing, but also solved most of the common problems involved.

Potentially they could have pivoted to monetizing that software infra even if it got deployed on arbitrary hardware, which may have tided them over until they could spin up their own cloud service, even if it was a fast follow to AWS. However Sun seemed committed to making their expensive servers their primary business model and relegated the software to “loss leader” complements.

If they had just looked at how services were being developed on Linux + commodity servers, they’d have seen they already had offerings in that space that could be monetized.

bluGill 8 hours ago||||
They should have seen commodity hardware coming and built cheaper hardware themselves. They could have. Other companies were making clone Spark machines for much less so it would have been possible.
tkhattra 6 hours ago||
sure, they shoulda woulda coulda, but isn't that the innovator's dilemma?
bluGill 6 hours ago||
Yes. There were lots of other things they could have done.
icedchai 8 hours ago||||
Early Linux distros were very rough. Remember messing around with modelines in X11 configs?

Sun could've been the "Apple" of x86 Unix: it just works.

gjvc 7 hours ago||
I'm willing to bet they couldn't have done that because of the mind-boggling array of 1990s UNIX standards Solaris seemed to support. By the mid 1990s it was clear that the (platform-independent) GNU command line tools were generally superior to what was on Solaris, but presumably they had to keep all the old fossils around to satisfy their long-standing supply / support contracts with industry, defence, and government.

That was my guess back then, and it's my guess now. Someone please correct me if I'm wrong and you know for certain why.

icedchai 7 hours ago||
I'm not saying it would've been easy. They could've installed the GNU tools in a different path and left it up to the user to change their preference. They did something similar with putting the BSD tools in "/usr/ucb" (I may have that path wrong.)
gjvc 4 hours ago||
that would have required them to swallow their pride, and admit GNU tools were in demand, which didn't fit the marketing narrative.
icedchai 1 hour ago||
They could've also just built a real package manager, left it up to the community to add GNU tools. I remember the Solaris package situation being annoying, having to download crap off FTP and install manually. Or often build from source.

Solaris was pretty bare bones, out of the box. I spent days customizing it. Even early Linux distros, like Slackware, were in better shape.

justin66 5 hours ago||||
With the benefit of perfect hindsight, which is a bit silly?

Open source Solaris as early as possible.

Really compete with MIPS, ARM, Motorola, etc. in the embedded space. Don't even try to make it a huge financial win, just get the SPARC architecture and dev tools out there with good Linux support.

Consistently support Solaris x86 and frequently release new generations of Intel and AMD workstation-class hardware.

Partner with TSMC for SPARC production a lot earlier.

Since we're just having fun here: acquire Nvidia in 1999, at which time it was pretty obvious workstation graphics was going to be commodified, and Sun could have bought them with their pocket change.

emmelaich 7 hours ago|||
Exactly, I used desktop Sun machines and administered many very large Sun machines from 1997 through to 2010.

I saw the end coming for Sun when I had a dx2/66 at home running Linux in the late 90s. It was brilliant.

bluGill 8 hours ago||
That doesn't exist. No department is every completely at fault. No department has enough budget to sink a company alone. It is always a lot of departments that don't turn in expected results - often for reasons not in their control.

The fault is always upper management not doing their job. Even if one department really would be that bad (which again they are not), upper management allowed them to be that bad.

thomasfl 6 hours ago||
We had a similar company in Norway in the 1980ies. Norsk Data (literally means Norwegian Computers) made computers, a multi taskin operating, a word processor thar inspired MS Word, a relational database on the same level as Oracle, the firsr web server was made by Tim Berners-Lee on one of their computers. It all went down because the managers kept insisting that they were a hardware company. The software was supposed to help selling their hardware. They were not good at making hardware.
nickff 6 hours ago||
MS Word was a descendant of Bravo and Gypsy on the Xerox Alto, not ND-NOTIS. NOTIS is an interesting footnote in the history of software, but PARC's influence on early microcomputing is well-known and recognized.

[1] https://en.wikipedia.org/wiki/Xerox_Alto#Software

[2] https://en.wikipedia.org/wiki/Bravo_(editor)

[3] https://en.wikipedia.org/wiki/Gypsy_(software)

[4] https://en.wikipedia.org/wiki/ND-NOTIS

pasc1878 6 hours ago||
Berners-Lee made the first web server true but it was on a NeXT not Norsk Data
Exoristos 6 hours ago||
I'm perceiving a lot of enthusiasts of the race to the bottom here. Yes, Dell and Linux could offer prices and convenience that undercut frontier hardware companies like Sun and SGI. But now what do we have? The loss is palpable to those of us who used to work with varied hardware competing to awe us, not to mention filling niches much more satisfactorily than one-size-fits-all.
bigbuppo 7 hours ago||
As someone that buys hardware today, all the hardware companies are now in the "we don't want to do business with you" territory.
mianos 50 minutes ago|
It's full RRP for everything less than 1000 units. If you want a 100 unit refresh you are better off going to your local corner shop for price and service.
titzer 9 hours ago|
I interned for Sun and then worked as an MTS in Sun Labs for a couple of years. I miss Sun; I think they were very much an engineer's company that was spoiled by two things: big success in the early workstation market, and the dumb luck of the dot com boom. Sun hardware was solid (even though they fell behind the curve because of mismanagement of a couple of CPU lines), and its kernel (Solaris) was one of the well-engineered kernels ever. While any business makes some bad business decisions from time to time, one thing that was fatal was a crippling enterprise focus that failed to understand the huge growth opportunity that OSS and Linux represented. It was growth in a direction they not only didn't understand, but refused to understand, study, or adapt to.

Sun's heyday was in the boom. The fish were jumping in the boat. So naturally they were building bigger and bigger boats.

There was always the one-off ideas that might have gone here or there, but that core server cow just seemed like an endless source of cash. Thus they neglected the workstation market and ironically, the market for anything bigger than one single server rack. They didn't understand horizontal scaling, and the rise of cheap-but-replaceable consumer hardware like how Google did things was alien to them. They poo-poo'ed Linux, they killed and then unkilled Solaris x86, and a number of other things.

I do miss Sun though. As far as engineering goes, they were a very serious company full of serious engineers. Basically the polar opposite of 10X tech bros and startup culture.

bcantrill 8 hours ago||
Oof, the killing and subsequent unkilling of Solaris x86 in 2002 brings back very bad memories -- Sun management at its worst. It also highlighted the relative power of Sun's engineering in that everyone in Solaris knew it was the wrong decision, and we more or less disobeyed orders: during that period (January 2002 to August 2002, when it was reversed), we didn't change our treatment of x86. (And in particular, breaking x86 was grounds for a putback to be backed out.)

When Solaris x86 was unkilled (and then became increasingly important as x86 eclipsed SPARC), there was damage done that I don't think we ever fully repaired.

titzer 3 hours ago||
Agreed. When I was in the labs (2007 - 2010), my workstation was an AMD with Solaris on it, with a modern window manager and all, with IntelliJ and a great development environment. I almost didn't know it was Linux. But by then the ship had long sailed.
lokar 8 hours ago|||
I worked at sun (sunsoft) right out of school near the peak. It was great. I was very excited to have all of the sources for sunos easily available to read.

Later, in grad school I went to work at a startup… google. For me it recaptured everything I liked about Sun. And had the same CEO.

srean 8 hours ago||
It indeed was a fantastic place, doors nailed to the walls, trying out the newly fledged Mapreduce and Charlie's. Good times. Good times.
specialist 8 hours ago||
Great perspective to share, thanks.

> They didn't understand horizontal scaling

What of Jini, JavaSpaces / Linda, JXTA, "grid computing"?

Were they too early? Dismissed as (Bill Joy's) hobby projects?

At the time, I was very enthusiastic. Especiall grid computing. So much so that when cloud computing arrived, I couldn't recognize the sea change. Square peg, round hole.

We even POC'd some Jini support. Demo'd for hardware manufacturers, customers, our dealer channel. Customers loved the idea.

But like UPnP, Jini failed to get traction.

As for JXTA (peer-to-peer), although I didn't gave any skin in that gane, I really expected it to vecome the norm. eg My brother (for his ISP) POC'd video delivery in anticipation. But that got mooted pretty quick.

Arguably, AWS Lambda (et al) was is "dumb" grid (run anywhere in asandbox) and the OpenClaw agentic (madness) is the "evil" incarnation. (I'm aware of efforts like Emerald AI (et al) to make a "smart" grid.)

twoodfin 6 hours ago||
RE: JXTA, right around the turn of the century, it felt like there was an opportunity for a globe-spanning peer-to-peer application layer, based on content-addressable storage.

DHTs—Chord paper was 2001—started popping up. The big software vendors looked at the architectural (if not legal) success of Napster & the other P2P networks and were invested in finding legitimate applications. Open standards were seeing real adoption left and right.

But it didn’t happen. Maybe simultaneously too late in the Internet’s evolution and too early.

Too late because the big kids were involved. Especially Microsoft, who would never allow any standard they couldn’t EEE to emerge. Too early because the players like Google who later could muscle through core infrastructure standards weren’t there yet.

And now it’s likely too late, with mobile embedding a client-server model with the weight of 10B devices.

More comments...