Why has Google open sourced TensorFlow?

I was sitting in a sun-warmed pizza restaurant in London last week talking about deep learning libraries. Everyone had their favourites. I was betting on TensorFlow, the new kid in town released by Google in late 2015. In response, a Torch fan pointed out that Google may invest in building up TensorFlow internally, but there’s no reason for them to invest in the shared, external version.

This got me thinking – why has Google open sourced TensorFlow?

Naively, I usually assume that companies keep their most crown jewels proprietary while open sourcing the periphery. In other words, keep your secret sauce close to your chest – but share the stuff that’s more generic, since it builds brand and goodwill, others may contribute helpfully, and you’re not straightforwardly giving a leg-up to your direct competitors.

Google’s approach to open source has been a little more strategic than this. Look at a handful of their major open source projects – Android, Chromium, Angular, Go, Dart, V8, Wave, WebM. The motivations behind them are various:

  • Android, Angular, Chromium, V8, Wave, WebM – creating a new version of an existing technology (free, better engineered, or faster) to disrupt an incumbent, or increase usage and thus drive revenue for Google’s core businesses.
  • Go, Dart and the long tail of minor projects are peripheral to their goals and serve less direct strategic interest.

For TensorFlow to make sense and be worthy of long-term support from Google, it needs to fall in the former category.

It is indeed a new version of an existing technology – it’s free, it’s better engineered, though not yet faster.

So, is it intended to either disrupt an incumbent, or to increase usage and thus drive revenue for core Google businesses? I can only think of two possibilities:

  1. TensorFlow is intended to be a major strategic benefit for Android. Machine learning is going to power a wave of new mobile applications, and many of them need to run locally rather than as a client-server app, whether for efficiency, responsiveness or bandwidth reasons. If TensorFlow makes it easier to develop cross-platform, efficient mobile machine learning solutions for Android but not for iOS, that could give the Android app market a major boost.
  2. TensorFlow is intended to be a major strategic benefit for Google’s platform/hosting, and to disrupt AWS. Right now, it’s pretty difficult and expensive to set up a cloud GPU instance. TensorFlow opens up the possibility of a granularly-scalable approach to machine learning that allows us to finally ignore the nitty-gritty of CUDA installations, Python dependencies, and multiple GPUs. Just specify the size of network you want, and TensorFlow allocates and spreads it across hardware as needed. This is why TensorBoard was part of the original implementation, and why AWS support was an afterthought. “Pay by the parameter”. If I had to guess, I’d say this is the major reason for open sourcing TensorFlow.

I want something like the above to be true, because I want there to be a strategic reason for Google to invest in TensorFlow, and I want it to get easier and easier to develop interesting and complex deep learning apps.

Blogging with WordPress and Emacs

When it comes to tools, I am a hedgehog rather than a fox. I like to have a small number of tools, and to know them well.

I recently resolved to start writing again. But I decided that I needed to sharpen my pencils first.

I have plans on how publishing and sharing should work. Grand plans. Too grand, perhaps.

So for now, I wrote something simple for myself. Now I can type away, press buttons… publish.

If you like Emacs, Python and WordPress, this might be interesting to you too. If not, it certainly won’t be.

wordpress-python-emacs GitHub repository

Most of the work is being done by this great Python/Wordpress library. Thank you.

I wrote some simple Python scripts. One grabs all my existing blog posts. One looks through their titles, and checks them against the filename to see if this is a new post.

And then there’s a very simple Emacs function that calls them to save/publish the current text file.

I could add more things: deleting posts, or a proper workflow for moving from draft to published. Maybe later.

I wrote this post, then hit M-x wordpress-publish-this-file.

PyEPL blog

PyEPL (the Python Experiment-Programming Library) is a versatile, high-level library for coding psychology experiments in Python. If you run human experiments and like coding, then I’d recommend it strongly.

PyEPL’s adoption and utility would both receive a tremendous boost if there were a wider array of snippets and tutorials floating around, so I created a collaborative PyEPL blog to provide an unofficial home for such tutorials, snippets and discussion. See the announcement post for more info.

N.B. I wouldn’t recommend PyEPL in its current form for low-level vision experiments, and it only runs on Mac and Linux. Both these concerns are slowly being addressed though, so things may have changed since the time of writing.

Non-parametric programming

Joel Spolsky (of Joel on Software fame) is in the midst of a World Tour, showing off the latest version of his company’s bug tracking and project management software, FogBugz 6.0.

I went along to see him when he visited Princeton, mainly out of curiosity. It turns out that FogBugz is pretty awesome. Let me highlight a couple of its most endearing features:

– the built-in wiki has the best WYSYWG editor I’ve seen

– utilizes backlinking between cases as a simple but handy aggregating related bugs

– really low barrier to entry for creating new bug reports

– beautifully integrated with email and discussion lists

Fine. But what really appealed to me was their emphasis on planning, and on tools that enabled more accurate estimations of shipping dates (‘evidence-based scheduling’). Firstly, they make it as easy as possible to log one’s activities (‘currently working on blah’), so that the system can keep track of how long is being spent on each case. Secondly, they require you to estimate up front how long you think that each case is going to take.

Now, in principle one’s ship date should now be determined by the some function of the number of cases left in your milestones list, how long each is projected to take, and the number of developers at your disposal. In reality, of course, these estimates are probably wrong. One can look at the history of an individual’s estimates, and compare them to how long things actually took. Perhaps I’m regularly off by a factor of 2 – so if I say something’s going to take four hours, it’s probably going to take the whole day. FogBugz builds a simple regression model, predicting actual duration from estimated duration for each developer, from which it can easily spot this factor-of-two trend.

If I’m reliably off by a factor of two, then that’s actually a kind of good news. It means that the system can be confident of its adjusted estimates. It means that the system can guess when the team is *actually* going to meet its milestones, rather than when they think they’re going to, by using the predicted durations from its regression model.

But things are probably more complicated. What if someone’s estimates are sometimes off by a factor of two, sometimes a factor of four, sometimes even over-compensating in the opposite direction? The variability in the goodness of this person’s estimates is more problematic for our projections. FogBugz’s solution is to use a kind of non-parametric permutation statistic to estimate 5%, 50% and 95% confidence intervals for shipping dates. I think the idea is to create a bajillion synthetic potential futures for each developer, each created by shuffling the evidence from their track record of estimations. We create a bajillion projected ship dates, each a combination of a set of synthetic developer-futures. Then, when the team manager wants to know the likelihood of shipping in time for Christmas, he or she can tell where it lies amidst the distribution of bajillion projected ship dates – if Christmas is earlier than all but a handful of the bajillion projected ship dates, then maybe there’s less than a 5% chance of shipping on time. If Christmas lies somewhere in the middle of the bajillion-strong distribution, then the system might estimate a 50% chance of shipping on time.

This idea of scrambling the past to generate a distribution is at the center of most non-parametric statistical tests, which avoid making assumptions about normal distributions, homescedasticity and other evil-sounding stuff that’s unlikely to be true in the real world. Anyway, props to Fog Creek for attempting to inject some low-level statistical intelligence in a way that might very plausibly work.

My take on emacsclient

Emacs is pretty lightweight relative to most modern editors, though by the time it loads all the modes and gets through all the uncompiled junk in my .emacs configuration, you wouldn’t know it.

Emacsclient is the solution – when you open a file with emacsclient, it doesn’t start up a whole new emacs – it just opens it in the running emacs, which is more or less instantaneous.

There are lots of webpages on this, so I won’t go into detail. Unfortunately, although you’d hope that calling ’emacsclient’ would work just like ’emacs’, this isn’t true:

  • I wanted to tell emacsclient to display an emacs frame, without feeding it any filename arguments to display. No dice.
  • If I don’t have a running emacs server (necessary for emacsclient to connect to), it just gives you an error message, rather than taking matters into its own hands and opening up a new emacs instance.
  • If I have emacs running on one computer, and I ssh into that computer, then I want to be able to type emacsclient and have a window show up. No dice. You have to explicitly feed it a display.

These issues were annoying enough that I nearly stopped using emacsclient. Various people have offered shell script hacks that handle some of these issues and more, but none of them solved the ones that bothered me. Plus, I like Python and I hate shell scripts. So I offer up a teeny python script, emf-on-display. If you alias ’emf-on-display’ to something handy like ‘e’, then it makes emacsclient behave in a way that I find much more consistent with emacs.

P.S. I have not dealt with the possibility that you’re running this in the terminal, and you don’t have an X11 display at all.

P.P.S. I suspect that it requires emacs 22 (or gnuclient) to work, since it relies on being able to pass ‘make-frame-on-display’ as elisp code to be evaluated.

Philosophy as debugging

Philosophical argument is kind of like debugging a program. You try and zero in on the source of the error, which is why you try and modularise the argument, provide test cases, see where you agree and disagree, and often it comes down to wrongful initialisation, a step accidentally added or omitted, or a failure to see the implications of some interaction you’d never fully considered.