Brad's Books and Organizations

Books

Books

Organizations

Organizations

Hi, I’m Brad Feld, a managing director at the Foundry Group who lives in Boulder, Colorado. I invest in software and Internet companies around the US, run marathons and read a lot.

« swipe left for tags/categories

swipe right to go back »

Just Make It Faster

Comments (78)

As a user, how often have you thought “I wish this web service was faster.”  As a CEO, how often have you said “just make it faster.”  Or, more simply, “why is this damn thing so slow?”

This is a not a new question.  I’ve been thinking about this since I first started writing code (APL) when I was 12 (ahem – 33 years ago) on a computer in the basement of a Frito-Lay data center in Dallas.

This morning, as part of my daily information routine, I came across a brilliant article by Carlos Bueno, an engineer at Facebook, titled “The Full Stack, Part 1.”  In it, he starts by defining a “full-stack programmer“:

“A “full-stack programmer” is a generalist, someone who can create a non-trivial application by themselves. People who develop broad skills also tend to develop a good mental model of how different layers of a system behave. This turns out to be especially valuable for performance & optimization work.”

He then dissects a simple SQL query (DELETE FROM some_table WHERE id = 1234;) and gives several quick reasons why performance could vary widely when this query is executed.

It reminded me of a client situation from my first company, Feld Technologies.  We were working on a logistics project with a management consulting firm for one of the largest retail companies in the world.  The folks from the management consulting firm did all the design and analysis; we wrote the code to work with the massive databases that supported this.  This was in the early 1990′s and we were working with Oracle on the PC (not a pretty thing, but required by this project for some reason.)  The database was coming from a mainframe and by PC-standards was enormous (although it would probably be considered tiny today.)

At this point Feld Technologies was about ten people and, while I still wrote some code, I wasn’t doing anything on this particular project other than helping at the management consulting level (e.g. I’d dress up in a suit and go with the management consultants to the client and participate in meetings.)  One of our software engineers wrote all the code.  He did a nice job of synthesizing the requirements, wrestling Oracle for the PC to the ground (on a Novell network), and getting all the PL/SQL stuff working.

We had one big problem.  It took 24 hours to run a single analysis.  Now, there was no real time requirement for this project – we might have gotten away with it if it took eight hours as we could just run them over night.  But it didn’t work for the management consultants or the client to hear “ok – we just pressed go – call us at this time tomorrow and we’ll tell you what happened.”  This was especially painful once we gave the system to the end client whose internal analyst would run the system, wait 24 hours, tell us the analysis didn’t look right, and bitch loudly to his boss who was a senior VP at the retailer and paid our bills.

I recall having a very stressful month.  After a week of this (where we probably got two analyses done because of the time it took to iterate on the changes requested by the client for the app) I decided to spend some time with our engineer who was working on it.  I didn’t know anything about Oracle as I’d never done anything with it as a developer, but I understood relational databases extremely well from my previous work with Btrieve and Dataflex.  And, looking back, I met the definition of a full-stack programmer all the way down to the hardware level (at the time I was the guy in our company that fixed the file servers when they crashed with our friendly neighborhood parity error or Netware device driver fail to load errors.)

Over the course of a few days, we managed to cut the run time down to under ten minutes.  My partner Dave Jilk, also a full-stack programmer (and a much better one than me), helped immensely as he completely grokked relational database theory.  When all was said and done, a faster hard drive, more memory, a few indexes that were missing, restructuring of several of the SELECT statements buried deep in the application, and a minor restructure of the database was all that was required to boost the performance by 100x.

When I reflect on all of this, I realize how important it is to have a few full-stack programmers on the team.  Sometimes it’s the CTO, sometimes it the VP of Engineering, sometimes it’s just someone in the guts of the engineering organization.  When I think of the companies I’ve worked with recently that are dealing with massive scale and have to be obsessed with performance, such as Zynga, Gist, Cloud Engines, and SendGrid I can identify the person early in the life of the company that played the key role. And, when I think of companies that did magic stuff like Postini and FeedBurner at massive scale, I know exactly who that full system programmer was.

If you are a CEO of a startup, do you know who the full-stack programmer on your team is?

  • http://twitter.com/mattamyers Matthew A Myers

    I just redid one of my websites because I wasn’t happy with its performance. I knew it could and needed to be faster – even though users are willing to wait for the load.

    Main page is about 4x faster now, and another 50% reduction will happen soon! Does it make me an uber geek that I’m so excited by this? Hehe.

    Mind you, I’m not at the point of massive traffic where speed with scale is a concern just yet – but I’m excited to tackle any issues when it (hopefully doesn’t) occur. :)

    • http://www.feld.com bfeld

      Excitement is good! Keep on optimizing.

    • http://www.feld.com bfeld

      Excitement is good! Keep on optimizing.

  • http://twitter.com/mattamyers Matthew A Myers

    I just redid one of my websites because I wasn’t happy with its performance. I knew it could and needed to be faster – even though users are willing to wait for the load.

    Main page is about 4x faster now, and another 50% reduction will happen soon! Does it make me an uber geek that I’m so excited by this? Hehe.

    Mind you, I’m not at the point of massive traffic where speed with scale is a concern just yet – but I’m excited to tackle any issues when it (hopefully doesn’t) occur. :)

  • http://freepository.com John Minnihan

    Cool story – I’ve a similar one.

    Customer was Juniper, and the project was the JUNOS repo conversion. It took 15 days <– yes, days, which meant that it would never actually get converted because the business wouldn't stop all development for 15 days.

    I invented a technique that collapsed the 15 days to 36.5 hours, allowing us to run the automation over a weekend. There were lots of smiles.

    • http://www.feld.com bfeld

      Great example.

    • http://www.feld.com bfeld

      Great example.

  • http://freepository.com John Minnihan

    Cool story – I’ve a similar one.

    Customer was Juniper, and the project was the JUNOS repo conversion. It took 15 days <– yes, days, which meant that it would never actually get converted because the business wouldn't stop all development for 15 days.

    I invented a technique that collapsed the 15 days to 36.5 hours, allowing us to run the automation over a weekend. There were lots of smiles.

  • http://freepository.com John Minnihan

    And since an underlying theme here is performance (as a setup for the ‘full-stack programmer’ concept), I’m going to plug two services that folks should check out:

    - New Relic. RoR, Java, PHP, .Net Lew invented perf management back in ’98 & I’m thrilled to have been an early team member at his first co. Wily. Using NR, co.s quickly locate & can resolve all sorts of perf issues.

    - Torbit – Josh & Jon are onto something VERY interesting here. Check them out *right now*.

    I’ve no stake in either of these cos., but they are my friends.

  • http://freepository.com John Minnihan

    And since an underlying theme here is performance (as a setup for the ‘full-stack programmer’ concept), I’m going to plug two services that folks should check out:

    - New Relic. RoR, Java, PHP, .Net Lew invented perf management back in ’98 & I’m thrilled to have been an early team member at his first co. Wily. Using NR, co.s quickly locate & can resolve all sorts of perf issues.

    - Torbit – Josh & Jon are onto something VERY interesting here. Check them out *right now*.

    I’ve no stake in either of these cos., but they are my friends.

  • http://twitter.com/aflandry André F. Landry

    Same principle could be applied to managers… A Full Stack Manager : someone who can understand finance, tech, marketing, user experience, etc., etc. and see the impact of any decision on all the organisation.

    • http://twitter.com/mattamyers Matthew A Myers

      aka Founder? :)

      • http://twitter.com/aflandry André F. Landry

        If you are or want to be a founder, you better be a full stack everything! ;)

      • http://twitter.com/aflandry André F. Landry

        If you are or want to be a founder, you better be a full stack everything! ;)

    • http://twitter.com/mattamyers Matthew A Myers

      aka Founder? :)

    • http://www.feld.com bfeld

      Excellent, excellent point. Also – “full stack entrepreneurs!”

      • http://carlos.bueno.org Carlos Bueno

        Man, do you really want to start a new buzzword? “Rockstar ninja” is bad enough. :D

      • http://carlos.bueno.org Carlos Bueno

        Man, do you really want to start a new buzzword? “Rockstar ninja” is bad enough. :D

    • http://www.feld.com bfeld

      Excellent, excellent point. Also – “full stack entrepreneurs!”

  • http://twitter.com/aflandry André F. Landry

    Same principle could be applied to managers… A Full Stack Manager : someone who can understand finance, tech, marketing, user experience, etc., etc. and see the impact of any decision on all the organisation.

  • DaveJ

    Also very important, in addition to the full-stack skill set, is an intuitive grasp of Amdahl’s law. Don’t expend energy optimizing things that only constitute a small fraction of the cost. Many engineers fall down on this, as they would prefer to spend their brain cycles on an analytically perfect optimization of a small portion of the problem, rather than an heuristic or even kludgey solution to the dominant issue.

    Incidentally, Amdahl’s law is highly applicable to business, and to life in general.

  • DaveJ

    Also very important, in addition to the full-stack skill set, is an intuitive grasp of Amdahl’s law. Don’t expend energy optimizing things that only constitute a small fraction of the cost. Many engineers fall down on this, as they would prefer to spend their brain cycles on an analytically perfect optimization of a small portion of the problem, rather than an heuristic or even kludgey solution to the dominant issue.

    Incidentally, Amdahl’s law is highly applicable to business, and to life in general.

  • http://www.justanentrrepreneur.com Philip Sugar

    Most succinct argument for why you can’t outsource development.

    You are right at every company has to have that go to person.

    One characteristic that you always find is they do as much development work at home on their own time as they do at work. They live, breathe, eat, sleep development. Whether is be structuring databases or coding for Lego Robots they want to know how it all works.

    • Anonymous

      Ah, don’t really need “to know how it all works”! There are alternatives! E.g., assume something is broken (yup, Virginia, it’s been known to happen!).

      Symptoms? Bad. Causes? Unknown! Users? Disappointed. Revenue? No more! Your best customer where one on one you worked out a special Christmas ad campaign? Torqued. Business? Headed for the toilet.

      So, we want to fix the problem, and for the alternatives “to know how it all works”, let’s have the envelope, please:

      (1) Yell and scream until hoarse.

      (2) Get a hammer and beat on each tower or rack mounted case.

      (3) Call customer support at HP, Dell, Asus, etc.

      (4) Call in an electrician to check the quality of the A/C power; i.e., blame it on the electric company!

      (5) Drive around the neighborhood and see if anyone is using electric arc welding and blame it on them! Yes, climb through all the bushes and look into all the windows!

      (6) Check sunspot activity!

      (7) Wait until evening and call the home phone number of the guy who was fired a month ago, blame it on him, and ask him to fix it!

      (8) Reinstall all the software on all the servers.

      (9) Regard the problem as getting ‘stuff’ to stick to the wall, and throw it against the wall overhand, left handed, with back spin, over left shoulder, etc. and hope that it sticks!

      (10) When some of the symptoms go away, declare victory and walk away!

      (11) And for the best alternative of all, call in 10 top psychics as consultants!

      Uh, it’s basically a machine that takes in stuff, crunches it, and puts out stuff. So, logically there is a ‘chain’ from the beginning to the end. If the chain is not holding, then it is broken in at least one place. To find the first break, start at the beginning of the chain and trace forward to the first break. Look closely there, check the relevant test data, look for what’s special or different, check ‘edge’ cases, etc., and see just why it broke. Fix it. Rinse and repeat.

      For all of this, it is from important up to crucial to have documentation of the whole chain and each link in it.

      And need to be able to trace the links. Or, no more complicated than a car, need real time instrumentation of oil pressure, engine RPM, alternator output, etc.

      Since software is ‘intangible’, need enough ‘instrumentation’ to make it tangible. For a 101 level lesson, for any on-line application, need log files, and in the code need to check for anything that looks strange and write a relevant note to a relevant log file.

      Want a ‘message level’ to control how much gets written to such log files. Eventually will want a little TCP/IP socket application to tweak the message level in real-time. Yes, I recently wrote some code to have really good ways to write to log files. I’ve had such message levels in my code from the first, and I intend to add a real time tweaking ability soon.

      Trust: In 99 44/100% of the cases, when all the links are solid, then the chain will hold.

    • Anonymous

      Ah, don’t really need “to know how it all works”! There are alternatives! E.g., assume something is broken (yup, Virginia, it’s been known to happen!).

      Symptoms? Bad. Causes? Unknown! Users? Disappointed. Revenue? No more! Your best customer where one on one you worked out a special Christmas ad campaign? Torqued. Business? Headed for the toilet.

      So, we want to fix the problem, and for the alternatives “to know how it all works”, let’s have the envelope, please:

      (1) Yell and scream until hoarse.

      (2) Get a hammer and beat on each tower or rack mounted case.

      (3) Call customer support at HP, Dell, Asus, etc.

      (4) Call in an electrician to check the quality of the A/C power; i.e., blame it on the electric company!

      (5) Drive around the neighborhood and see if anyone is using electric arc welding and blame it on them! Yes, climb through all the bushes and look into all the windows!

      (6) Check sunspot activity!

      (7) Wait until evening and call the home phone number of the guy who was fired a month ago, blame it on him, and ask him to fix it!

      (8) Reinstall all the software on all the servers.

      (9) Regard the problem as getting ‘stuff’ to stick to the wall, and throw it against the wall overhand, left handed, with back spin, over left shoulder, etc. and hope that it sticks!

      (10) When some of the symptoms go away, declare victory and walk away!

      (11) And for the best alternative of all, call in 10 top psychics as consultants!

      Uh, it’s basically a machine that takes in stuff, crunches it, and puts out stuff. So, logically there is a ‘chain’ from the beginning to the end. If the chain is not holding, then it is broken in at least one place. To find the first break, start at the beginning of the chain and trace forward to the first break. Look closely there, check the relevant test data, look for what’s special or different, check ‘edge’ cases, etc., and see just why it broke. Fix it. Rinse and repeat.

      For all of this, it is from important up to crucial to have documentation of the whole chain and each link in it.

      And need to be able to trace the links. Or, no more complicated than a car, need real time instrumentation of oil pressure, engine RPM, alternator output, etc.

      Since software is ‘intangible’, need enough ‘instrumentation’ to make it tangible. For a 101 level lesson, for any on-line application, need log files, and in the code need to check for anything that looks strange and write a relevant note to a relevant log file.

      Want a ‘message level’ to control how much gets written to such log files. Eventually will want a little TCP/IP socket application to tweak the message level in real-time. Yes, I recently wrote some code to have really good ways to write to log files. I’ve had such message levels in my code from the first, and I intend to add a real time tweaking ability soon.

      Trust: In 99 44/100% of the cases, when all the links are solid, then the chain will hold.

  • http://www.justanentrrepreneur.com Philip Sugar

    Most succinct argument for why you can’t outsource development.

    You are right at every company has to have that go to person.

    One characteristic that you always find is they do as much development work at home on their own time as they do at work. They live, breathe, eat, sleep development. Whether is be structuring databases or coding for Lego Robots they want to know how it all works.

  • http://michaelkimsal.com Michael Kimsal

    Never knew what to call myself, but I’ve fit in to that description in the past (probably still do, but trying to move in to other areas).

    Came in to a shop in 2004 that had a DB process which took anywhere between 8-10 hours. Working with one of the team I was able to to modify the process down to about 45 mins. He then worked with the sysadmin to change a few more things and got it down to about 20. 20 minutes. from 10 hours. Originally they said it couldn’t be done, but there was plenty of room for improvement.

    FTR this was a MySQL process – we changed the import process from million of records to the final table in to multiple steps. First chunk sets of 100k rows in to an in-memory table, process that table in to final table, clear out in-mem table, then start over again. Another proc ran after all data was finally in the target table, but most of the intermediate processing was being done on 100k chunks in memory rather than millions of rows on disk. Perhaps this will help someone else think different about their own data import issues. :)

    The annoying thing was no one seemed to care after we got it done. No ‘thank’ or ‘good job’ or anything.

    @bfeld – saw you in Durham last month – thanks for a fun evening!

    • http://www.feld.com bfeld

      “Good job!” Seriously – I know where you are coming from – performance improvements of this magnitude are heroic but for some reason the “magician in the system” is often not recognized.

    • http://www.feld.com bfeld

      “Good job!” Seriously – I know where you are coming from – performance improvements of this magnitude are heroic but for some reason the “magician in the system” is often not recognized.

    • Anonymous

      Right! I needed to load a lot of data into a table in SQL Server. So, I wrote the simple code to insert the rows one at a time and heard the disk rattle; not a good sound, and it was slow. Right; plenty understandable.

      Sooooo, I wrote code to read in a big chunk of the imported data into memory, sorted it in memory (sure, heap sort ‘by surrogate’), wrote that chunk to the DB in clustered key order, and put a loop around that code to handle all the imported chunks that way. It was faster, and the disk sounded much better (quieter).

      Good to see that I’m not the Lone Ranger here, that others have done the same thing!

      Sure, one more loop to read the data in clustered key order and write it that way to another table should be fast enough and get clustered key order still closer to actual physical disk order. Then for one more step to a little more performance, copy the new table to a recently ‘defragmented’ disk partition!

      Actually, for our high performance needs for the data we are keeping in SQL Server, we will just write that out to a file and have the specialized servers read it in, say, once a day or week. In the code in the specialized servers, the data will be just in an ‘array of objects’ (that code is running). Physically the data will be in virtual memory with a lot of locality of reference. Just noticed that 64 bit Windows can have 16 TB of virtual memory! Nice!

    • Anonymous

      Right! I needed to load a lot of data into a table in SQL Server. So, I wrote the simple code to insert the rows one at a time and heard the disk rattle; not a good sound, and it was slow. Right; plenty understandable.

      Sooooo, I wrote code to read in a big chunk of the imported data into memory, sorted it in memory (sure, heap sort ‘by surrogate’), wrote that chunk to the DB in clustered key order, and put a loop around that code to handle all the imported chunks that way. It was faster, and the disk sounded much better (quieter).

      Good to see that I’m not the Lone Ranger here, that others have done the same thing!

      Sure, one more loop to read the data in clustered key order and write it that way to another table should be fast enough and get clustered key order still closer to actual physical disk order. Then for one more step to a little more performance, copy the new table to a recently ‘defragmented’ disk partition!

      Actually, for our high performance needs for the data we are keeping in SQL Server, we will just write that out to a file and have the specialized servers read it in, say, once a day or week. In the code in the specialized servers, the data will be just in an ‘array of objects’ (that code is running). Physically the data will be in virtual memory with a lot of locality of reference. Just noticed that 64 bit Windows can have 16 TB of virtual memory! Nice!

  • http://michaelkimsal.com Michael Kimsal

    Never knew what to call myself, but I’ve fit in to that description in the past (probably still do, but trying to move in to other areas).

    Came in to a shop in 2004 that had a DB process which took anywhere between 8-10 hours. Working with one of the team I was able to to modify the process down to about 45 mins. He then worked with the sysadmin to change a few more things and got it down to about 20. 20 minutes. from 10 hours. Originally they said it couldn’t be done, but there was plenty of room for improvement.

    FTR this was a MySQL process – we changed the import process from million of records to the final table in to multiple steps. First chunk sets of 100k rows in to an in-memory table, process that table in to final table, clear out in-mem table, then start over again. Another proc ran after all data was finally in the target table, but most of the intermediate processing was being done on 100k chunks in memory rather than millions of rows on disk. Perhaps this will help someone else think different about their own data import issues. :)

    The annoying thing was no one seemed to care after we got it done. No ‘thank’ or ‘good job’ or anything.

    @bfeld – saw you in Durham last month – thanks for a fun evening!

  • http://blog.calbucci.com/ Marcelo Calbucci

    Brad, for what’s worth it, at Microsoft full-stack programmers are called Architects (although they might not have that word on their title). The good ones are capable of understanding and accounting for true throughput of 100MBps Ethernet network, latency between servers, disk seek time, database architecture, TCP/IP performance, DNS lookups, and a lot more, besides the code the team is writing.

    Very few startups have this kind of person when they get started. If the founder(s) is too young, they might be generalists, but lack the depth of knowledge in most of those areas. If the founder is an “expert” than he’s mostly aware of his side of the world.

    In general, I think most startups don’t need to and shouldn’t care about performance and scaling until they pass a certain point. Once they pass that point, it gets much easier to attract talent who are passionate about serious scaling issue, or they can throw money at the problem.

    The final myth is that cloud computing will make all performance and scaling issues go away. I heard so many entrepreneurs telling they don’t have to worry about performance and scale because they are running out of AWS. I’m pretty sure they will be pretty surprised when (if) they reach just a dozen simultaneous requests per minute and realize their system is taking 20 seconds to return a page.

    • http://www.feld.com bfeld

      I’ve met a number of the Microsoft architects. In my experience, most of the ones I know have a particular blindspot – they have knowledge of the full stack as long as it’s Microsoft’s full stack! I’ve had some very interested conversations with some of these folks about non-Microsoft tech – plenty of relevant / applicable knowledge but often very interesting blind spots.

      For most of the companies we invest in, we want this person to be part of the founding team and/or hired very early as building from the beginning for massive scale is important to get right in the initial architecture, at least in my opinion.

    • http://www.feld.com bfeld

      I’ve met a number of the Microsoft architects. In my experience, most of the ones I know have a particular blindspot – they have knowledge of the full stack as long as it’s Microsoft’s full stack! I’ve had some very interested conversations with some of these folks about non-Microsoft tech – plenty of relevant / applicable knowledge but often very interesting blind spots.

      For most of the companies we invest in, we want this person to be part of the founding team and/or hired very early as building from the beginning for massive scale is important to get right in the initial architecture, at least in my opinion.

    • http://twitter.com/StevenHB Steven Bergstein

      I disagree with your assertion that startups don’t need to worry about performance and scalability. Both are emergent characteristics of systems – meaning that you can’t bolt on the “performance module” or the “scalability module.” If you want a performant and/or scalable system, you must consider the impact of every architectural and design decision on the performance/scalability of the system.

      I was the architect for a system being developed for a state agency around 1988-1990 where my employer had a contracted commitment to deliver a system with one-second response time. With every decision we made, we considered the impact to response time. In the end, the system had about two-second response time, which was satisfied the business needs of the client. I’m sure we wouldn’t have been so close if we hadn’t considered the impact so consistently.
      I

    • http://twitter.com/StevenHB Steven Bergstein

      I disagree with your assertion that startups don’t need to worry about performance and scalability. Both are emergent characteristics of systems – meaning that you can’t bolt on the “performance module” or the “scalability module.” If you want a performant and/or scalable system, you must consider the impact of every architectural and design decision on the performance/scalability of the system.

      I was the architect for a system being developed for a state agency around 1988-1990 where my employer had a contracted commitment to deliver a system with one-second response time. With every decision we made, we considered the impact to response time. In the end, the system had about two-second response time, which was satisfied the business needs of the client. I’m sure we wouldn’t have been so close if we hadn’t considered the impact so consistently.
      I

  • http://blog.calbucci.com/ Marcelo Calbucci

    Brad, for what’s worth it, at Microsoft full-stack programmers are called Architects (although they might not have that word on their title). The good ones are capable of understanding and accounting for true throughput of 100MBps Ethernet network, latency between servers, disk seek time, database architecture, TCP/IP performance, DNS lookups, and a lot more, besides the code the team is writing.

    Very few startups have this kind of person when they get started. If the founder(s) is too young, they might be generalists, but lack the depth of knowledge in most of those areas. If the founder is an “expert” than he’s mostly aware of his side of the world.

    In general, I think most startups don’t need to and shouldn’t care about performance and scaling until they pass a certain point. Once they pass that point, it gets much easier to attract talent who are passionate about serious scaling issue, or they can throw money at the problem.

    The final myth is that cloud computing will make all performance and scaling issues go away. I heard so many entrepreneurs telling they don’t have to worry about performance and scale because they are running out of AWS. I’m pretty sure they will be pretty surprised when (if) they reach just a dozen simultaneous requests per minute and realize their system is taking 20 seconds to return a page.

  • http://twitter.com/annejohn Anne Johnson

    Off site back up used to have similar characteristics to – the database grew to the point where the nightly backup didn’t complete by the time business started again in the morning. We’d sell the networking equipment to improve the transfer rate – though I learned to request on site testing, to determine how much of the bottleneck was network induced, and how much of it was due to disc storage i/o.

    As an investor, have you checked that your CEOs have a full-stack programmer, or at least have access to someone who can review the technology required to go after the proposed market, and likely pivots ? (Brad does this, but it’s not uncommon to find people claiming performance results for which they couldn’t have afforded to test ..)

    • http://www.feld.com bfeld

      I’ll take your backup and raise you one. Back when I worked with 4GLs, index files (which were separate from the database in Dataflex and Clarion – two of the languages we wrote most of our software in), often became corrupt for a variety of reasons, including network issues, hardware issues, and software fragility. Reindexing happened on a client computer and the entire application and database were unavailable during the reindexing process. We had no control over reindexing – the only thing we could do was throw faster hardware at it. We’d occasionally have indexes that took > 12 hours to rebuild – now THAT was painful.

    • http://www.feld.com bfeld

      I’ll take your backup and raise you one. Back when I worked with 4GLs, index files (which were separate from the database in Dataflex and Clarion – two of the languages we wrote most of our software in), often became corrupt for a variety of reasons, including network issues, hardware issues, and software fragility. Reindexing happened on a client computer and the entire application and database were unavailable during the reindexing process. We had no control over reindexing – the only thing we could do was throw faster hardware at it. We’d occasionally have indexes that took > 12 hours to rebuild – now THAT was painful.

  • http://twitter.com/annejohn Anne Johnson

    Off site back up used to have similar characteristics to – the database grew to the point where the nightly backup didn’t complete by the time business started again in the morning. We’d sell the networking equipment to improve the transfer rate – though I learned to request on site testing, to determine how much of the bottleneck was network induced, and how much of it was due to disc storage i/o.

    As an investor, have you checked that your CEOs have a full-stack programmer, or at least have access to someone who can review the technology required to go after the proposed market, and likely pivots ? (Brad does this, but it’s not uncommon to find people claiming performance results for which they couldn’t have afforded to test ..)

  • AntS

    Experienced this from the other side, 15 years ago I had hacked together a predictive model for entire European car market by zipcode 5 years out into future. Work of genius but I was no programmer and it took 24-36 hours to run. Eventually asked a proper coder to look at it and first pass he got it down to 45 mins and from there to 10. Comments he added to source were wonderful “This is the bit where we repeat the same process 1000 times for no apparent reason”.

  • AntS

    Experienced this from the other side, 15 years ago I had hacked together a predictive model for entire European car market by zipcode 5 years out into future. Work of genius but I was no programmer and it took 24-36 hours to run. Eventually asked a proper coder to look at it and first pass he got it down to 45 mins and from there to 10. Comments he added to source were wonderful “This is the bit where we repeat the same process 1000 times for no apparent reason”.

  • http://jaxn.org Jackson Miller

    Brad, I am a full-stack guy and am also the founder. I am curious if you would have given up less technical control at Feld knowing what you know now. Theoretically the process would never have taken 24 hours if you were working on it the whole time. Is a full-stack programmer / founder more valuable as a programmer or a CEO?

    • http://www.feld.com bfeld

      In Feld Technologies’ case, I had no choice. As we scaled the business, I couldn’t be involved in everything. As the CEO, I ended up being front facing for much of our business, did a bunch of high end projects, and did most of the sales. Ultimately my partner and I decided that he couldn’t handle managing the fucked up projects so when things went off the rails (about 20% of the time), I ended up with some of them so he could focus on managing the 80% of the business that was working fine.

      I’m not sure a full-stack programmer / founder is more valuable as a programmer or CEO, but in this case I think I was more valuable to the business as CEO than if I had spent all my time coding.

    • http://www.feld.com bfeld

      In Feld Technologies’ case, I had no choice. As we scaled the business, I couldn’t be involved in everything. As the CEO, I ended up being front facing for much of our business, did a bunch of high end projects, and did most of the sales. Ultimately my partner and I decided that he couldn’t handle managing the fucked up projects so when things went off the rails (about 20% of the time), I ended up with some of them so he could focus on managing the 80% of the business that was working fine.

      I’m not sure a full-stack programmer / founder is more valuable as a programmer or CEO, but in this case I think I was more valuable to the business as CEO than if I had spent all my time coding.

  • http://jaxn.org Jackson Miller

    Brad, I am a full-stack guy and am also the founder. I am curious if you would have given up less technical control at Feld knowing what you know now. Theoretically the process would never have taken 24 hours if you were working on it the whole time. Is a full-stack programmer / founder more valuable as a programmer or a CEO?

  • http://www.facebook.com/flybrand Fred Lybrand

    This is a common problem outside of software too. For big industrial technologies you need someone on your side, or the customer’s side, who can see through the entire supply chain. In our textile business every individual plant manager and product guy can cite a dozen reasons a new process won’t fit (eg line speed, cost, implementation risk), but at some point an individual with a broader perspective sits back, takes a look at things and says, “We have to do this, and do it this way.”

    Often times the customer titles in the industrial world are identical to what you laid out here for IT – CTO, VP of Engineering, but it is surprising how often it winds up being someone on the sales team. I like Marcelo’s comment about the title ‘Architect’ being used at MSFT.

    I know we’ve laughed about cleantech VCs, but the reason it has gone so poorly for investors in that space (which is really just industrial technology 2.0), is that the interoperability of the industrial stack is more difficult to determine than it is on the IT side. The example of Oracle – PC – Novell network – PL/SQL is tough, but the nice thing about information technology is it is at least supposed to work together. When you start evaluating a materials science stack of polymer extraction – polymer formulation – fiber production – composite layering – final product build you have materials that are worse at interacting and cycle times that take years instead of weeks.

    Some of the winners who are starting to emerge in industrial technologies are those that are focused on cutting down their iteration cycles, and your post has helped clarify some of the issues here – thank you!

    • http://www.feld.com bfeld

      Great insight. While I’ve spent very little time paying attention to / thinking about cleantech, the few cleantech investments I’ve got in older portfolio’s have clearly demonstrated the power / challenge of this.

      • Anonymous

        “The power / challenge” sounds like a euphemism for “busted a gut and failed”!

        As we saw in

        Frederick P. Brooks, ‘The Mythical Man-Month: Essays on Software Engineering’, ISBN 0-201-00650-2.

        now in whole or part at

        http://javatroopers.com/Mythical_Man_Month.html

        “Many creative activities deal in intractable media; paint smears, wood splits, and electronic components fail. But computer programming, however, creates with an exceedingly tractable medium.”

        There are three ways computing is more difficult:

        (1) Computing is largely ‘intangible’. With wood, can pick it up and inspect it. With TCP/IP packets flowing at 100 Mbps, that’s more difficult!

        (2) Computing is potentially quite complex and is the source of by far the most complex systems we know how to build.

        (3) Due to (1) and (2), computing needs good documentation, and that’s in short supply.

      • Anonymous

        “The power / challenge” sounds like a euphemism for “busted a gut and failed”!

        As we saw in

        Frederick P. Brooks, ‘The Mythical Man-Month: Essays on Software Engineering’, ISBN 0-201-00650-2.

        now in whole or part at

        http://javatroopers.com/Mythical_Man_Month.html

        “Many creative activities deal in intractable media; paint smears, wood splits, and electronic components fail. But computer programming, however, creates with an exceedingly tractable medium.”

        There are three ways computing is more difficult:

        (1) Computing is largely ‘intangible’. With wood, can pick it up and inspect it. With TCP/IP packets flowing at 100 Mbps, that’s more difficult!

        (2) Computing is potentially quite complex and is the source of by far the most complex systems we know how to build.

        (3) Due to (1) and (2), computing needs good documentation, and that’s in short supply.

    • http://www.feld.com bfeld

      Great insight. While I’ve spent very little time paying attention to / thinking about cleantech, the few cleantech investments I’ve got in older portfolio’s have clearly demonstrated the power / challenge of this.

    • Anonymous

      So, by training, I am a telephone engineer and telephony… networking… provides a lot of examples of extremely complex systems. And that has allowed me to make some observations. I won’t bore everyone with all of them, but here are two related ones:

      1) Systems, whether it’s a single thing like the space shuttle or a network or a supply chain, can be viewed on a spectrum of complexity. At the low end, most everyone understands how the whole thing works and can therefore help optimize the performance of the whole. In the middle, the complexity is high enough that only a few people can hold all the constraints in their head long enough to think through the optimal solution. These people are the full-stack programmers, the architects, the best general managers. At the high end, the complexity is so high that no single person understands how everything in the system can/should work. At that point, design by committee is required… and the likelihood of reaching an optimal solution declines rapidly. I’d posit that many of the most important innovations happen at… even cause… the transition from that high-end complexity realm to the middle.

      2) The invention of abstractions and jargon is one of the most effective responses to complexity. 70 years ago, a “computer” was a thing that filled a room that only a few dozen people in the world could understand in its entirety. It simply could not get more complex and still have a chance of working. By 30 years ago, few people needed the detailed knowledge of how a processor works because the functionality had been included in a black box on the design diagram called a “microprocessor.” So, systems could be more complex because many details were encapsulated in that black box… with the benefits far outweighing any sub-optimalities created by drawing it. The same is true of layered software and layered protocols: for most systems using these stacks, the layering reduces the perceived complexity of the system into the low-complexity realm and therefore makes it possible to deploy large numbers of people to make things happen. And existence — the fact that something happens at all — is often good enough. Only some cases then remain that are both medium complexity and have the economic incentive to deploy a full-stack person to solve. The result is that the vast preponderance of systems are inelegant and suboptimal… and we architects hate that. :)

  • http://www.facebook.com/flybrand Fred Lybrand

    This is a common problem outside of software too. For big industrial technologies you need someone on your side, or the customer’s side, who can see through the entire supply chain. In our textile business every individual plant manager and product guy can cite a dozen reasons a new process won’t fit (eg line speed, cost, implementation risk), but at some point an individual with a broader perspective sits back, takes a look at things and says, “We have to do this, and do it this way.”

    Often times the customer titles in the industrial world are identical to what you laid out here for IT – CTO, VP of Engineering, but it is surprising how often it winds up being someone on the sales team. I like Marcelo’s comment about the title ‘Architect’ being used at MSFT.

    I know we’ve laughed about cleantech VCs, but the reason it has gone so poorly for investors in that space (which is really just industrial technology 2.0), is that the interoperability of the industrial stack is more difficult to determine than it is on the IT side. The example of Oracle – PC – Novell network – PL/SQL is tough, but the nice thing about information technology is it is at least supposed to work together. When you start evaluating a materials science stack of polymer extraction – polymer formulation – fiber production – composite layering – final product build you have materials that are worse at interacting and cycle times that take years instead of weeks.

    Some of the winners who are starting to emerge in industrial technologies are those that are focused on cutting down their iteration cycles, and your post has helped clarify some of the issues here – thank you!

  • http://danspinosa.com/ Dan Spinosa

    A full stack programmer can identify where optimization will provide the biggest bang for your buck, without forgetting that premature optimization is the root of all evil.

  • http://danspinosa.com/ Dan Spinosa

    A full stack programmer can identify where optimization will provide the biggest bang for your buck, without forgetting that premature optimization is the root of all evil.

  • http://www.hypedsound.com jonathanjaeger

    Sometimes as a non-programmer you can underestimate the work involved in making a site optimized for high levels of traffic. On a recent episode of TWiT, Kevin Rose explained how Digg’s engineers were spending all their time optimizing queries, which meant less time for product innovation and changes.

    • http://www.feld.com bfeld

      Yup – this is one of the big “playing catch up” issues when you have big scale ups but haven’t engineered correctly for this up front. Often your users start complaining that you aren’t adding any features when the vast majority of engineering time is being spent dealing with rearchitecting / rewriting stuff to deal with scale. I give you Twitter as a very notable example of this several years ago (2007/2008).

      • Anonymous

        So cruel, but another example, the death of Cuil: The newsies in their obits said that Cuil grew too fast and couldn’t keep up, that, thus, their search quality went into the tank, and users were disappointed, left, and didn’t come back. Likely Cuil got a lot of ‘negative virality’!

        Cuil was not short of interested early users, but in total Cuil ‘peaked too fast’ and ran into “You only get one chance to make a good first impression.”.

        Uh, before the big publicity campaign or the big ‘viral coefficient’, be sure can both serve the users and please them, or at least plug in more servers fast enough to do so. Typically for this, need an appropriate ‘architecture’ BEFORE the crucial software is written! Else may feel like rebuilding the airplane during its first flight! See the miracle: We take off in a biplane, rebuild during flight, and land in a 747! SURE you will, along with perpetual motion engines!

        One rule to success: When see a big pile of it ahead, go around it and certainly don’t step in it. Indeed, there will likely be some you don’t see and will step in, so be ready for those, too.

        Uh, aviation learned the hard way: Plan the flight. Be sure about payload weight and balance. Check the weather all along the route. Check current runway length and air temperature. Calculate fuel carefully. Have alternate destinations. Have reserve fuel. The alternatives lead to smoking holes.

        That Twitter was able to catch up speaks well for Twitter, but I have to suspect that the Board heard a LOT of engineering horror stories from management, and management kept asking the Board for big bucks, over and over. What was the bill for the all night coffee and carry in pizza? The office sleeping bags? What about the bill for the FedEx highest priority of computer parts? How to calm people down during fights after 48 hours without sleep?

        Uh, it’s an old saying: Vast projects with half-vast plans!

      • Anonymous

        So cruel, but another example, the death of Cuil: The newsies in their obits said that Cuil grew too fast and couldn’t keep up, that, thus, their search quality went into the tank, and users were disappointed, left, and didn’t come back. Likely Cuil got a lot of ‘negative virality’!

        Cuil was not short of interested early users, but in total Cuil ‘peaked too fast’ and ran into “You only get one chance to make a good first impression.”.

        Uh, before the big publicity campaign or the big ‘viral coefficient’, be sure can both serve the users and please them, or at least plug in more servers fast enough to do so. Typically for this, need an appropriate ‘architecture’ BEFORE the crucial software is written! Else may feel like rebuilding the airplane during its first flight! See the miracle: We take off in a biplane, rebuild during flight, and land in a 747! SURE you will, along with perpetual motion engines!

        One rule to success: When see a big pile of it ahead, go around it and certainly don’t step in it. Indeed, there will likely be some you don’t see and will step in, so be ready for those, too.

        Uh, aviation learned the hard way: Plan the flight. Be sure about payload weight and balance. Check the weather all along the route. Check current runway length and air temperature. Calculate fuel carefully. Have alternate destinations. Have reserve fuel. The alternatives lead to smoking holes.

        That Twitter was able to catch up speaks well for Twitter, but I have to suspect that the Board heard a LOT of engineering horror stories from management, and management kept asking the Board for big bucks, over and over. What was the bill for the all night coffee and carry in pizza? The office sleeping bags? What about the bill for the FedEx highest priority of computer parts? How to calm people down during fights after 48 hours without sleep?

        Uh, it’s an old saying: Vast projects with half-vast plans!

    • http://www.feld.com bfeld

      Yup – this is one of the big “playing catch up” issues when you have big scale ups but haven’t engineered correctly for this up front. Often your users start complaining that you aren’t adding any features when the vast majority of engineering time is being spent dealing with rearchitecting / rewriting stuff to deal with scale. I give you Twitter as a very notable example of this several years ago (2007/2008).

  • http://www.hypedsound.com jonathanjaeger

    Sometimes as a non-programmer you can underestimate the work involved in making a site optimized for high levels of traffic. On a recent episode of TWiT, Kevin Rose explained how Digg’s engineers were spending all their time optimizing queries, which meant less time for product innovation and changes.

  • Anonymous

    I put myself in the full-stack category, but it would be immodest to say why. But in my post-engineering business career, I’ve come across an interesting corollary (or perhaps anti-corollary) driven by the scarcity of full-stack programmers. Most applications are developed by folks who are not full-stack programmers. The reason for this is economic… aside from a few applications, the difference between “it works” and “it works well” is not sufficiently large to pay for, either in extra time to market or in the price of getting sufficient numbers of scarce full-stack programmers to work the project. This then has some rather negative impacts:

    1) There’s a great emphasis on capital efficiency these days and “fail fast” and even Agile. In software design, this translates to a lot of pressure to code just at the top-most layers without much understanding or any optimization of the lower levels. Code to the APIs and assume they work as advertised. And pray that the languages actually compile properly. The prototype looks pretty and all is good, until you have to scale. (I’m looking at you, RoR Twitter!) Or make sure it’s secure. (I’m looking at you, Gawker, who has now made me change N passwords over the past week.)

    2) The stack tends to lock in a given architecture. Yes, in theory, you can port the stack easily by recompiling, but this is infrequently true in practice and essentially never true for optimized code. I formerly worked with a processor that had significantly better performance than x86. Game designers, who “program down to the metal,” have enjoyed the performance boost. (They are some of the best full-stack programmers.) However, the cost to re-optimize a stack of dozens of libraries helped ensure that general programmers would never adopt it. It would be like asking them to adopt a Dvorak keyboard.

    So, yes, full-stack programmers are a great thing. They are the architects that understand a building is more than a facade. They need your support, because a lot of what they may say is short-term expensive but long-term crucial.

  • Anonymous

    I put myself in the full-stack category, but it would be immodest to say why. But in my post-engineering business career, I’ve come across an interesting corollary (or perhaps anti-corollary) driven by the scarcity of full-stack programmers. Most applications are developed by folks who are not full-stack programmers. The reason for this is economic… aside from a few applications, the difference between “it works” and “it works well” is not sufficiently large to pay for, either in extra time to market or in the price of getting sufficient numbers of scarce full-stack programmers to work the project. This then has some rather negative impacts:

    1) There’s a great emphasis on capital efficiency these days and “fail fast” and even Agile. In software design, this translates to a lot of pressure to code just at the top-most layers without much understanding or any optimization of the lower levels. Code to the APIs and assume they work as advertised. And pray that the languages actually compile properly. The prototype looks pretty and all is good, until you have to scale. (I’m looking at you, RoR Twitter!) Or make sure it’s secure. (I’m looking at you, Gawker, who has now made me change N passwords over the past week.)

    2) The stack tends to lock in a given architecture. Yes, in theory, you can port the stack easily by recompiling, but this is infrequently true in practice and essentially never true for optimized code. I formerly worked with a processor that had significantly better performance than x86. Game designers, who “program down to the metal,” have enjoyed the performance boost. (They are some of the best full-stack programmers.) However, the cost to re-optimize a stack of dozens of libraries helped ensure that general programmers would never adopt it. It would be like asking them to adopt a Dvorak keyboard.

    So, yes, full-stack programmers are a great thing. They are the architects that understand a building is more than a facade. They need your support, because a lot of what they may say is short-term expensive but long-term crucial.

  • http://viarob.com Rob Nagler

    The term “full stack programmer” seems to imply that coding is linear. When you solve a performance problem, it’s pretty much like any other problem: you use your breadth of experience to speed things up without breaking anything. You can speed up path performance through caching, but caching means state management, which is “across” the application, not down the stack: you want to avoid one part of the program using the cached value while another goes directly to the db, thus causing a state mismatch aka. a bug.

    My boss at Tandem in the 90s called these type of people “experienced generalists”. It’s fairly easy to find someone who can optimize code, queries, etc., but it’s a lot harder to find someone who can see the ramifications of said optimizations on the correctness of the code. Moreover, performance is usually the easiest class of problem an experienced generalist has to solve: it’s much harder, say, to codify IRS Pub 541, which is non-executable English created by bureaucrats and lawmakers. You solve a performance problem by running an sql analyzer, a code profiler, or even just watching the wall clock. It’s a repeatable process. I’ve found that anyone who can code Pub 541 actually enjoys performance problems, because they are fun to solve because you get immediate feedback.

    I also have a few questions: How do you reconcile full-stack programming with the idea that the best programmers in the world are in their 20s or even teens (viz. Peter Thiel’s stop out fellowships)? Once you become a full-stack programmer, do you stop coding and instead “architect” software? If so, doesn’t that contradict the concept of understanding how the stack works when a problem occurs? Or, is this part of a cycle where the young coders get handed the full-stack problems created by the ex-full-stack programmers/Architects who have lost touch with the code that’s actually in the stack?

    • Anonymous

      Some of your dilemma is due to a fallacy: “the idea that the best programmers in the world are in their 20s or even teens”. Absurd “idea”.

      Because people over 30 have brain disease? Not exactly! Because ‘programming’ makes such rapid progress? No it doesn’t! Because there’s little more to know beyond what a teenager can know? Nope. Instead, the idea is just absurd.

      You seem to be taking ‘programming’ in a sense that is far too narrow: Also crucial is design, architecture, algorithms, data structures, testing, integration, performance, reliability, and project management. Then beyond just the programming narrowly are many closely related topics from engineering and its applied math. E.g., what’s the big deal about using double precision when accumulating inner products? When might one want to use inner products? Hint: A big inner product as just one machine instruction is a favorite on ‘vector’ instruction sets.

      E.g., so, the company needs to schedule, say, airplanes, trucks, workers, machines, salesmen, etc. Uh, maybe they need to design the MPLS core of their IP network. The idea is to reduce cost or increase revenue. Running a cable of optical fibers a few hundred miles across the country could cost big bucks and should be not taken lightly!

      Here’s a really simple toy problem: There is an assembly line with ten ‘stations’, and here are 10 workers. For each worker, we ran some tests and found, for each station, how fast they could do that station. Now, how to assign the workers to the stations to make the line run as fast as possible?

      Show me a teenage programmer who would have even a clue how to attack this problem. Really, any CS graduate in their 20s. Any programmer. Really, any CS prof; a small fraction would be able to start.

      Actually, I ran into the problem once in how to assign signals to a satellite to minimize some signal interference with another satellite. I hope NASA was pleased.

      A programmer ran off and, for my solution, typed code into a general purpose 0-1 integer linear programming (ILP) package that finished with branch and bound. He returned to me saying that his ILP package was really good because it finished quickly. I responded, “Of course it finished quickly. It never had to do any branch and bound.” How’d I know this? Ask some teenage programmers!

      For the scheduling, sure, the CS profs right away will smell NP-complete which is their excuse to give up. Commonly their students actually seriously misunderstand NP-complete. Still the airplanes, trucks, etc. need to be scheduled. Okay, how about 0-1 integer linear programming set covering with Gilmore-Gomory column generation? How about some Lagrangian relaxation?

      Actually, for the ‘programming’ taken narrowly, it doesn’t change very fast. Indeed, Paul Graham is still pushing a language from 1960 or so, Lisp. As we move to parallelism and reliability, we will take renewed interest in some more quite old work. Uh, in a nutshell, ‘programming’ is (1) define data structures, (2) have some dynamic memory management to permit allocating and freeing them, (3) have the two main ‘control structures’ if-then-else and do-while, and (4) have functions to which pass arguments by reference. Tweak the data structures a little, permit pointers to functions, sprinkle over some syntactic sugar (the traditional way is with a pre-processor), and get ‘object oriented’ programming (OOP). The OOP idea is not so new: IBM had it in microcode in the early 1970s. Parallelism is harder, but again the main ideas are old, ‘tasks’, locks, semaphores, ‘actors’, etc.

      Computer security? The main ideas remain those of Kerberos, RSA, capabilities, and attribute control lists from MIT going back to Multics in 1970.

      Virtual machine is now a hot topic which, however, was rock solid in the IBM 360/67 with CP67/CMS of 1967.

      Relational data base? Of crucial importance but from the 1970s from M. Blasgen et al. at Yorktown Heights and W. Wong et al. at Berkeley.

      So, a programmer promoted to ‘architect’ should not have fallen so far behind that they have “lost touch with the code that’s actually in the stack”, and a teenage programmer will be far behind for a long time, especially if they don’t get some good courses.

    • Anonymous

      Some of your dilemma is due to a fallacy: “the idea that the best programmers in the world are in their 20s or even teens”. Absurd “idea”.

      Because people over 30 have brain disease? Not exactly! Because ‘programming’ makes such rapid progress? No it doesn’t! Because there’s little more to know beyond what a teenager can know? Nope. Instead, the idea is just absurd.

      You seem to be taking ‘programming’ in a sense that is far too narrow: Also crucial is design, architecture, algorithms, data structures, testing, integration, performance, reliability, and project management. Then beyond just the programming narrowly are many closely related topics from engineering and its applied math. E.g., what’s the big deal about using double precision when accumulating inner products? When might one want to use inner products? Hint: A big inner product as just one machine instruction is a favorite on ‘vector’ instruction sets.

      E.g., so, the company needs to schedule, say, airplanes, trucks, workers, machines, salesmen, etc. Uh, maybe they need to design the MPLS core of their IP network. The idea is to reduce cost or increase revenue. Running a cable of optical fibers a few hundred miles across the country could cost big bucks and should be not taken lightly!

      Here’s a really simple toy problem: There is an assembly line with ten ‘stations’, and here are 10 workers. For each worker, we ran some tests and found, for each station, how fast they could do that station. Now, how to assign the workers to the stations to make the line run as fast as possible?

      Show me a teenage programmer who would have even a clue how to attack this problem. Really, any CS graduate in their 20s. Any programmer. Really, any CS prof; a small fraction would be able to start.

      Actually, I ran into the problem once in how to assign signals to a satellite to minimize some signal interference with another satellite. I hope NASA was pleased.

      A programmer ran off and, for my solution, typed code into a general purpose 0-1 integer linear programming (ILP) package that finished with branch and bound. He returned to me saying that his ILP package was really good because it finished quickly. I responded, “Of course it finished quickly. It never had to do any branch and bound.” How’d I know this? Ask some teenage programmers!

      For the scheduling, sure, the CS profs right away will smell NP-complete which is their excuse to give up. Commonly their students actually seriously misunderstand NP-complete. Still the airplanes, trucks, etc. need to be scheduled. Okay, how about 0-1 integer linear programming set covering with Gilmore-Gomory column generation? How about some Lagrangian relaxation?

      Actually, for the ‘programming’ taken narrowly, it doesn’t change very fast. Indeed, Paul Graham is still pushing a language from 1960 or so, Lisp. As we move to parallelism and reliability, we will take renewed interest in some more quite old work. Uh, in a nutshell, ‘programming’ is (1) define data structures, (2) have some dynamic memory management to permit allocating and freeing them, (3) have the two main ‘control structures’ if-then-else and do-while, and (4) have functions to which pass arguments by reference. Tweak the data structures a little, permit pointers to functions, sprinkle over some syntactic sugar (the traditional way is with a pre-processor), and get ‘object oriented’ programming (OOP). The OOP idea is not so new: IBM had it in microcode in the early 1970s. Parallelism is harder, but again the main ideas are old, ‘tasks’, locks, semaphores, ‘actors’, etc.

      Computer security? The main ideas remain those of Kerberos, RSA, capabilities, and attribute control lists from MIT going back to Multics in 1970.

      Virtual machine is now a hot topic which, however, was rock solid in the IBM 360/67 with CP67/CMS of 1967.

      Relational data base? Of crucial importance but from the 1970s from M. Blasgen et al. at Yorktown Heights and W. Wong et al. at Berkeley.

      So, a programmer promoted to ‘architect’ should not have fallen so far behind that they have “lost touch with the code that’s actually in the stack”, and a teenage programmer will be far behind for a long time, especially if they don’t get some good courses.

  • http://viarob.com Rob Nagler

    The term “full stack programmer” seems to imply that coding is linear. When you solve a performance problem, it’s pretty much like any other problem: you use your breadth of experience to speed things up without breaking anything. You can speed up path performance through caching, but caching means state management, which is “across” the application, not down the stack: you want to avoid one part of the program using the cached value while another goes directly to the db, thus causing a state mismatch aka. a bug.

    My boss at Tandem in the 90s called these type of people “experienced generalists”. It’s fairly easy to find someone who can optimize code, queries, etc., but it’s a lot harder to find someone who can see the ramifications of said optimizations on the correctness of the code. Moreover, performance is usually the easiest class of problem an experienced generalist has to solve: it’s much harder, say, to codify IRS Pub 541, which is non-executable English created by bureaucrats and lawmakers. You solve a performance problem by running an sql analyzer, a code profiler, or even just watching the wall clock. It’s a repeatable process. I’ve found that anyone who can code Pub 541 actually enjoys performance problems, because they are fun to solve because you get immediate feedback.

    I also have a few questions: How do you reconcile full-stack programming with the idea that the best programmers in the world are in their 20s or even teens (viz. Peter Thiel’s stop out fellowships)? Once you become a full-stack programmer, do you stop coding and instead “architect” software? If so, doesn’t that contradict the concept of understanding how the stack works when a problem occurs? Or, is this part of a cycle where the young coders get handed the full-stack problems created by the ex-full-stack programmers/Architects who have lost touch with the code that’s actually in the stack?

  • http://flocial.com flocial

    Informative as always but just an enjoyable read as well. Would love to read more war stories like this.

  • http://flocial.com flocial

    Informative as always but just an enjoyable read as well. Would love to read more war stories like this.

  • Derek Scruggs

    A few weeks ago I met someone who’s still in college and learning programming for the first time using Ruby on Rail. I’m sure she has the skill set logically (physics major), are frameworks like Rails good for learning on? Rails eliminates 95% of sql. That’s nice when you’re trying to get code out the door, but it’s also really easy to create tables with no indexes in Rails, and a really simple looking method call might result in a four-table join under the covers.

    OTOH I’m sure a lot of Assembly programmers would sneer at my programming background.

  • Derek Scruggs

    A few weeks ago I met someone who’s still in college and learning programming for the first time using Ruby on Rail. I’m sure she has the skill set logically (physics major), are frameworks like Rails good for learning on? Rails eliminates 95% of sql. That’s nice when you’re trying to get code out the door, but it’s also really easy to create tables with no indexes in Rails, and a really simple looking method call might result in a four-table join under the covers.

    OTOH I’m sure a lot of Assembly programmers would sneer at my programming background.

  • http://www.onlineaspect.com Josh Fraser

    Great post. Obviously something near and dear to me. More great comments over on hacker news:
    http://news.ycombinator.com/item?id=2021782

  • http://www.onlineaspect.com Josh Fraser

    Great post. Obviously something near and dear to me. More great comments over on hacker news:
    http://news.ycombinator.com/item?id=2021782

  • Anonymous

    “If you are a CEO of a startup, do you know who the full-stack programmer on your team is?”

    Darned right I do! He’s the guy with the ‘business model’, the core ‘secret sauce’ work, and the scalable, reliable architecture who finds or invents algorithms as necessary, types in, times, tests, and documents the code, does the ‘system administration and management’, and fixes things that break. At shaving time, I see him in the mirror.

    Today he’s still recovering from being up all night two nights last week fixing such things. He got pooped, but they got fixed.

    Wish it were “faster”? The software to illustrate the stochastic optimal control math for my Ph.D. was going to run for 64 years, but I wanted to graduate! So, I saw where some of the calculations used only some of the state variables, did some pre-computing, got the time down to 120 seconds, and graduated on time!

    So, if have a big server farm and network, is it healthy? Let’s look only for problems never seen before. Okay, collect data, say, from some relevant ‘part’ of this monster, many times a second on each of several variables.

    Okay, now what to do with the data? Do a ‘statistical hypothesis test’, right, with known, adjustable false alarm rate. Need a multi-dimensional, distribution-free test, but could find none in E. Lehmann or elsewhere so invented one. The computing was going to run too long, so found a way to do ‘binary search’ but in several dimensions. Right: Reinvented k-D trees, k-dimensional binary search trees. Now the statistical test is fast enough!

    Want to look one at a time at maybe 100 million numbers and at the end have the, say, 50 largest. How to do that? Hmm! Borrow the heap data structure and ‘sift’ algorithm from heap sort! So for n = 100 million, get execution time proportional to (n)ln(50). If do that 100,000 times a second, will want something fast!

    How to do that 100,000 times a second? Right: ‘Architecture’. Use parallelism and partitioning, get that understood early on, and write the first code that way! Have a good direction, and don’t take the same ground twice!

    Have 50 million items sorted in ascending order and want to do a binary search once for each of 21 more items? Sure: Sort the 21 items and then pick the one in the middle of the 21, item 11, use binary search on it, partition the 50 million at that item, and wash and repeat with the top 10 items and the upper partition of the 50 million and the lower 10 in the lower partition. Actually needed that!

    SQL Server running 24 hours? I can understand that! Unless the database is really ‘small’ compared with the ‘size’ of the hardware, I trust relational database theory only to make running times way too long. So, I have only minimal on-line use of SQL Server, and so far in total have only a few tables, each really simple, have good ‘clustered keys’ and a few other important indices, and have no joins or stored procedures. Actually, each table could be partitioned and on its own collection of servers. We’re talking DIRT SIMPLE.

    Yes, my tables are still in ‘third normal form’!

    Yup, the usage of SQL Server is so simple likely could use Btrieve instead. Since both are mostly disk bound, and since SQL Server has some other advantages, I’ve gone with it. When my business is successful, Customer Service and the CFO’s group will be able to make good use of SQL Server as intended in relational database theory.

    “Wrestling Oracle for the PC to the ground (on a Novell network), and getting all the PL/SQL stuff working.” I can understand! Earlier this year, I had SQL Server and my first Web pages working fine and then got a virus, my first ever, that wiped out all my software installations and configurations. I still can’t believe that important software facing the Internet still has errors as dumb as buffer overflows, but it does.

    So had to redo all my software installs and configurations. Soon got another virus. While rebuilding, got another, this one from just one use of the Akamai Download Manager just to get a PDF on a motherboard.

    Gee, guys, we’ve known for a long time how to run any software at all, ‘malicious’ or not, with full safety. What’s with all these viruses that corrupt operating system installations? Angry? WAY beyond angry.

    So, implemented some much better virus defenses, and haven’t had anymore virus problems.

    For rebuilding a boot partition, took careful notes and got the time down by a factor of 50.

    To recover faster, set up NTBACKUP to save a bootable partition so that a restore would be bootable. On the first restore, the partition was not bootable. Bad documentation with some BNF Backus and Naur never heard of. So, ran experiments and tests and discovered how to get bootable restores. Lot of time I didn’t want to spend.

    SQL, it’s been fast, fun, and easy from Ullman’s book through own schema, T-SQL, and ADO.NET.

    Then in SQL Server tried to use the newer administration and management functionality around permissions, securables, roles, etc. and discovered that even simple T-SQL commands can make SQL Server get sick. Then it won’t SELECT, uninstall, repair, install, or reinstall, and for it Windows System Restore won’t. So a sick SQL Server means reformat the boot partition. Bummer.

    Work around: Install everything BUT SQL Server and use NTBACKUP to save the boot partition so that a restore will be bootable. THEN install SQL Server and if it gets sick restore the boot partition and then reinstall SQL Server.

    “Wrestling”? How ’bout ‘mud wrestling’?

    Viruses. Long boot partition rebuilding times. Awful NTBACKUP details. Sickly SQL Server. Got through it, but WHAT a MESS.

    Back to writing the last of the ASP.NET Web pages that access SQL Server, the computations, etc.

  • Anonymous

    “If you are a CEO of a startup, do you know who the full-stack programmer on your team is?”

    Darned right I do! He’s the guy with the ‘business model’, the core ‘secret sauce’ work, and the scalable, reliable architecture who finds or invents algorithms as necessary, types in, times, tests, and documents the code, does the ‘system administration and management’, and fixes things that break. At shaving time, I see him in the mirror.

    Today he’s still recovering from being up all night two nights last week fixing such things. He got pooped, but they got fixed.

    Wish it were “faster”? The software to illustrate the stochastic optimal control math for my Ph.D. was going to run for 64 years, but I wanted to graduate! So, I saw where some of the calculations used only some of the state variables, did some pre-computing, got the time down to 120 seconds, and graduated on time!

    So, if have a big server farm and network, is it healthy? Let’s look only for problems never seen before. Okay, collect data, say, from some relevant ‘part’ of this monster, many times a second on each of several variables.

    Okay, now what to do with the data? Do a ‘statistical hypothesis test’, right, with known, adjustable false alarm rate. Need a multi-dimensional, distribution-free test, but could find none in E. Lehmann or elsewhere so invented one. The computing was going to run too long, so found a way to do ‘binary search’ but in several dimensions. Right: Reinvented k-D trees, k-dimensional binary search trees. Now the statistical test is fast enough!

    Want to look one at a time at maybe 100 million numbers and at the end have the, say, 50 largest. How to do that? Hmm! Borrow the heap data structure and ‘sift’ algorithm from heap sort! So for n = 100 million, get execution time proportional to (n)ln(50). If do that 100,000 times a second, will want something fast!

    How to do that 100,000 times a second? Right: ‘Architecture’. Use parallelism and partitioning, get that understood early on, and write the first code that way! Have a good direction, and don’t take the same ground twice!

    Have 50 million items sorted in ascending order and want to do a binary search once for each of 21 more items? Sure: Sort the 21 items and then pick the one in the middle of the 21, item 11, use binary search on it, partition the 50 million at that item, and wash and repeat with the top 10 items and the upper partition of the 50 million and the lower 10 in the lower partition. Actually needed that!

    SQL Server running 24 hours? I can understand that! Unless the database is really ‘small’ compared with the ‘size’ of the hardware, I trust relational database theory only to make running times way too long. So, I have only minimal on-line use of SQL Server, and so far in total have only a few tables, each really simple, have good ‘clustered keys’ and a few other important indices, and have no joins or stored procedures. Actually, each table could be partitioned and on its own collection of servers. We’re talking DIRT SIMPLE.

    Yes, my tables are still in ‘third normal form’!

    Yup, the usage of SQL Server is so simple likely could use Btrieve instead. Since both are mostly disk bound, and since SQL Server has some other advantages, I’ve gone with it. When my business is successful, Customer Service and the CFO’s group will be able to make good use of SQL Server as intended in relational database theory.

    “Wrestling Oracle for the PC to the ground (on a Novell network), and getting all the PL/SQL stuff working.” I can understand! Earlier this year, I had SQL Server and my first Web pages working fine and then got a virus, my first ever, that wiped out all my software installations and configurations. I still can’t believe that important software facing the Internet still has errors as dumb as buffer overflows, but it does.

    So had to redo all my software installs and configurations. Soon got another virus. While rebuilding, got another, this one from just one use of the Akamai Download Manager just to get a PDF on a motherboard.

    Gee, guys, we’ve known for a long time how to run any software at all, ‘malicious’ or not, with full safety. What’s with all these viruses that corrupt operating system installations? Angry? WAY beyond angry.

    So, implemented some much better virus defenses, and haven’t had anymore virus problems.

    For rebuilding a boot partition, took careful notes and got the time down by a factor of 50.

    To recover faster, set up NTBACKUP to save a bootable partition so that a restore would be bootable. On the first restore, the partition was not bootable. Bad documentation with some BNF Backus and Naur never heard of. So, ran experiments and tests and discovered how to get bootable restores. Lot of time I didn’t want to spend.

    SQL, it’s been fast, fun, and easy from Ullman’s book through own schema, T-SQL, and ADO.NET.

    Then in SQL Server tried to use the newer administration and management functionality around permissions, securables, roles, etc. and discovered that even simple T-SQL commands can make SQL Server get sick. Then it won’t SELECT, uninstall, repair, install, or reinstall, and for it Windows System Restore won’t. So a sick SQL Server means reformat the boot partition. Bummer.

    Work around: Install everything BUT SQL Server and use NTBACKUP to save the boot partition so that a restore will be bootable. THEN install SQL Server and if it gets sick restore the boot partition and then reinstall SQL Server.

    “Wrestling”? How ’bout ‘mud wrestling’?

    Viruses. Long boot partition rebuilding times. Awful NTBACKUP details. Sickly SQL Server. Got through it, but WHAT a MESS.

    Back to writing the last of the ASP.NET Web pages that access SQL Server, the computations, etc.

  • Discopete

    I call ‘em the FULL ENCHILADA PROGRAMMERS.

Build something great with me