# Paths to the Future: A Year at Google Brain

From 2017 through 2018, I was a software engineer on the Google Brain team. I started three months after receiving a Master’s degree in computer science, having just spent the summer working on a research project—a programming language for making optimal decisions, based on math—at Stanford, my alma mater. A part of me wanted to continue working with my advisor at Stanford, but another part of me was deeply curious about Google Brain. To see a famous artificial intelligence (AI) research lab from the inside would, at the very least, make for an interesting anthropological experience. So I joined Google in the autumn. My assignment was to work on TensorFlow, an open-source software library for building machine learning models.

My desk was just a few steps away from our kitchenette, or microkitchen, in Google-speak. Its name notwithstanding, the microkitchen was a comfortably large space, softly illuminated by sunlight that spilled through tall windows; the windows framed a view of Shoreline Park’s grassy knolls, a rustic gold during the dry season and a lush green when wet. The microkitchen was a place for snacking, caffeinating, informal conversations, and—for a new graduate like myself, not yet accustomed to the glamour of working at Brain—a place for people-watching. In that one kitchen, in a single afternoon, I saw Google’s CEO Sundar Pichai (who believes that AI is “more profound than electricity or fire”), its co-founder Sergey Brin, Turing award winners David Patterson and John Hennessey, and memeworthy engineer Jeff Dean (who, among other things, can make an espresso in an astonishingly short amount of time.)

The glamour wore off quickly. What I came to value most about Brain was that it provided me access to exceptional mentors. I worked with some of the original TensorFlow developers, including Derek Murray, Asim Shankar, Ashish Agarwal, and Alexandre Passos. These engineers treated me like a peer. They encouraged me to take ownership of important tasks, giving me guidance when I sought it and giving me more credit than I deserved. For example, at one point, Derek helped me redesign a small but important part of the TensorFlow runtime, making it possible to partition named units of computation across multiple devices. Even though much of the design was his own, he always introduced our work to others as “Akshay’s project.” Later, Asim, Ashish, and Alexandre let me take the lead in writing an academic paper about TensorFlow Eager, a technology that transformed TensorFlow from a declarative programming language to an imperative one, even though my contributions to the technology were smaller than theirs. The unreasonable amount of trust placed in me, and credit given to me, made me work harder than I would have otherwise.

In many ways, the culture of Google Brain reminded me of what I’ve read about Xerox PARC, perhaps the most influential of industrial computer science research labs. During the 1970s, researchers at PARC paved the way for the personal computer revolution by developing foundational technologies like graphical user interfaces and producing one of the earliest incarnations of a desktop computer. The culture of PARC is documented in The Power of the Context, an essay written by PARC researcher Alan Kay. Kay describes PARC as a place where senior employees treated less experienced ones as “world-class researchers who just haven’t earned their PhDs yet.” Kay goes on to say that researchers at PARC were self-motivated and capable “artists,” working to bring their “visions” into reality. Artists worked independently or in small teams towards similar visions, making for a productive environment that felt “out of control” at times:

A great vision acts like a magnetic field from the future that aligns all the little iron particle artists to point to “North” without having to see it. They then make their own paths to the future. Xerox often was shocked at the PARC process and declared it out of control, but they didn’t understand that the context was so powerful and compelling and the good will so abundant, that the artists worked happily at their version of the vision. The results were an enormous collection of breakthroughs, some of which we are celebrating today.

At Brain, as at PARC, researchers and engineers have an incredible amount of autonomy. They have bosses, to be sure, but they have an a lot of leeway in choosing what to work on — in finding “their own paths to the future.” I’ll give one example: a few years ago, many on the Google Brain team realized that machine learning tools were closer to programming languages than to libraries, and that redesigning their tools with this fact in mind would unlock greater productivity. Management didn’t command engineers to work on a particular solution to this problem. Instead, several small teams formed organically, each approaching the problem in its own way—TensorFlow 2.0, Swift for TensorFlow, JAX, Dex, Tangent, Autograph, and MLIR are all different angles on the same vision. Some are in direct tension with each other, since they’re solving the same problem. Still, each is improved by the existence of the other; we’d re-use each other’s solutions when possible, and we’d share notes often. It’s totally possible that many of these tools might not become anything more than promising experiments, but it’s also possible that at least one will be a breakthrough.

Unlike Xerox, which struggled to find applications of PARC’s research, Google makes use of the tools and research incubated by its AI research lab. For the past few years, Pichai has emphasized that Google is an “AI-first” company, with the company seeking to implement “machine learning techniques in nearly everything [they] do;” Google Photos, Translate, and the Assistant are maybe the most salient examples of AI-powered products.

I would guess that the PARC-like context that Brain operates in was instrumental in bringing about the creation of TensorFlow, the machine learning tool on which I worked. In late 2015, Google open-sourced TensorFlow, making it freely available to the entire world. At the time, TensorFlow was the only open-source framework for deep learning that was actively developed by industry. TensorFlow lowered the barriers to training machine learning models on very large datasets and subsequently using the trained models in real-world applications, accelerating research in machine learning as well as industrial applications of it.

TensorFlow quickly became enormously popular. Instructors at Stanford and other universities used it in their curricula (my friend Chip Huyen, for example, created a Stanford course called TensorFlow for Deep Learning Research), researchers across the globe used it to run experiments, and AI companies used it to train and deploy models in the real world. Today, TensorFlow is the fifth most popular project on Github out of the many millions of public software repositories available on it, as measured by the number of “stars”, or likes, per repository.

At least for TensorFlow, however, Google Brain’s hyper-creative, hyper-productive, and “out of control” culture has been a double-edged sword. In the process of making their own paths to a shared future, TensorFlow engineers released many features sharing a similar purpose, like constructing neural networks. Many of these features were subsequently deemphasized in favor of more promising ones. While this process might have selected for good features, it frustrated and exhausted our users, who struggled to keep up. On the other hand, many of the surviving features did substantially improve TensorFlow—for example, tf.data simplified data processing pipelines, and eager execution has made TensorFlow more expressive and pleasant to use.

These days, in addition to machine learning, I’m interested in in convex optimization, a branch of computational mathematics concerned with making optimal choices. Convex optimization has many real-world applications—SpaceX uses it to land rockets, self-driving cars use it to track trajectories, and financial companies use it to design investment portfolios. While well-studied from a theoretical perspective, as a technology, convex optimization is still young and niche. I suspect that convex optimization has the potential to become a powerful, widely-used technology. I’m interested in doing the work—a bit of mathematics, and a bit of computer science—to realize its potential. My advisor at Stanford, Stephen Boyd, is perhaps the world’s leading expert on applications of convex optimization, and I simply could not pass up an opportunity to do useful research under his guidance. (In fact, most of my mentors at Brain encouraged me to enroll in the PhD program. Only one researcher strongly discouraged me from pursuing a PhD, comparing the experience to “psychological torture.” I was so shocked by his dark warning that I didn’t ask any follow-up questions, he didn’t elaborate, and our meeting ended shortly afterwards.)

It’s been just over a year since I left Google and started my PhD. Since then, I’ve collaborated with my lab to publish five papers, including one that makes it possible to automatically learn the structure of convex optimization problems, bridging the gap between convex optimization and deep learning. There are many things about Google Brain that I miss, my coworkers—the most skilled engineers I know—most of all. But now, at Stanford, I get to collaborate with and learn from an intellectually diverse group of extremely smart and passionate individuals, ranging from pure mathematicians, electrical and chemical engineers, physicists, biologists, and computer scientists. (Sometimes, when I look at the layers of equations, diagrams, and curves scrawled on the whiteboards in our lab, I get the strange sense that I’m inhabiting a scene from the film Good Will Hunting or A Beautiful Mind, and I can’t tell if life is imitating art or art is imitating life.) I’m now one of three developers of CVXPY, an open-source library for convex optimization, and I have total creative control over my research and engineering projects. I’m not sure what I’ll do once I graduate, but for now, I’m having a lot of fun—and learning a ton—doing a bit of math, writing real software, and exploring several lines of research in parallel. If I’m very lucky, one of them might even be a breakthrough.

# A Primer on TensorFlow 2.0

This post is also available as a Python notebook.

From September 2017 to October 2018, I worked on TensorFlow 2.0 alongside many engineers. In this post, I’ll explain what TensorFlow 2.0 is and how it differs from TensorFlow 1.x. Towards the end, I’ll briefly compare TensorFlow 2.0 to PyTorch 1.0. This post represents my own views; it does not represent the views of Google, my former employer.

TensorFlow (TF) 2.0 is a significant, backwards-incompatible update to TF’s execution model and API.

Execution model. In TF 2.0, all operations execute imperatively by default. Graphs and the graph runtime are both abstracted away by a just-in-time tracer that translates Python functions executing TF operations into executable graph functions. This means in TF 2.0, there is no Session, and no global graph state. The tracer is exposed as a Python decorator, tf.function. This decorator is for advanced users. Using it is completely optional.

API. TF 2.0 makes tf.keras the high-level API for constructing and training neural networks. But you don’t have to use Keras if you don’t want to. You can instead use lower-level operations and automatic differentiation directly.

To follow along with the code examples in this post, install the TF 2.0 alpha.

pip install tensorflow==2.0.0-alpha0
import tensorflow as tf
tf.__version__
'2.0.0-alpha0'

Contents

# Maine and Potatoes: Approaching Life Like Steinbeck

Per my sister’s recommendation, I recently picked up Travels with Charley, Steinbeck’s account1 of a cross-country road trip he took one summer with his beloved Poodle in tow.

Steinbeck’s favorite kind of journey is a meandering one. By his own admission, he’s “going somewhere” but “doesn’t greatly care whether” he arrives2. Reflecting upon a leisurely detour through Maine’s potato farms, he writes,

everything in the world must have design or the human mind rejects it. But in addition it must have purpose or the human conscience shies away from it. Maine was my design, potatoes my purpose.

It’s tempting to interrogate whether your pursuits are meaningful, be they hobbies or careers3. A degree of such interrogation can be constructive: living with intention necessitates a design and a purpose. But indulge too much and you risk descending into a Hamlet-esque, nihilistic spiral that will inevitably derail your pursuit. The last thing you (and certainly I) want is to end up as Camus’ strawman, the individual who cannot cope with his discovery that life is without meaning. That Steinbeck’s design was Maine and his purpose potatoes is a gentle reminder that our own designs and purposes need not be grand. All that we require of them is to exist.

Footnotes
[1] The introduction to the book’s 50th anniversary edition cautions readers against taking Steinbeck’s story too literally, for he was “a novelist at heart.” But the book reads truthfully enough and, just as important, entertainingly enough. As author and writing instructor John McPhee joked in an interview with The New Yorker’s David Remnick, 94 percent accuracy is good enough for creative non-fiction.
[2] Approaching our actions with such a sentiment is precisely the Bhagavad Gita’s prescription for attaining the Good Life. For that matter, it is also the prescription of Kierkegaard’s Fear and Trembling. Both recommend we resign ourselves to the frustration of our desires, but that we do so happily so that we may pursue them nonetheless. If this sounds difficult to you, you’re not alone; Kierkegaard’s narrator describes this process as something he cannot hope to understand, though he spends the entire text describing it.
[3] Academics at MIT’s Sloan School of Management recently asked 135 people what made their work meaningful. For many, meaningful work is simultaneously “intensely personal” and bigger than themselves.

# Gaussian Random Walks: The N-step Distribution

This past evening, my former quiz bowl teammate turned co-worker (not to mention long-time friend) Michael Whittaker tossed a number of math problems my way. One in particular caught my interest:

Say you’ve got a normal random variable with mean zero and variance one. Draw a sample from that distribution. Now take that point you’ve just sampled and spawn a unit-variance normal distribution centered around it; draw a sample from this distribution. Continue this process $N$ times for arbitrary $N$. What is the distribution of the $N$th sample?

Michael told me that he and his friends at Cornell thought up this problem during a lull in a dinner-time conversation. The solution is after the break. A hint: It’s either simpler than it appears or, if you’ve done a bit of math, just as simple as it appears. read more »

# Learning about Learning: Educational Data Mining

Earlier this summer, I crossed the Atlantic and traveled to Madrid to give a talk at the 8th International Conference on Educational Data Mining. I presented a prototype, built by myself and my colleagues at Stanford, that stages intelligent interventions in the discussion forums of Massive Open Online Courses. Our pipeline, dubbed YouEDU, detects confusion in forum posts and recommends instructional video snippets to their presumably confused authors.

The Educational Data Mining Conference took place in Madrid this year. Pictured above the Retiro Pond in Buen Retiro Park. It has nothing to do with EDM. But I enjoyed the park so please enjoy the picture.

No, not that kind of EDM
Educational Data Mining — affectionately collapsed to EDM — might sound opaque. From the society’s website, EDM is the science and practice of

developing methods for exploring the unique and increasingly large-scale data that come from educational settings, and using those methods to better understand students, and the settings which they learn in.

Any educational setting that generates data is a candidate for EDM research. So really any educational setting is a candidate, full stop. In practice, EDM-ers often find themselves focusing their efforts on computer-mediated settings, like tutoring systems, educational games, and MOOCs, perhaps because it’s easy to instrument these systems to leave behind trails of data.

Popular methods applied to these educational settings include student modeling, affect detection, and interventions. Student models attempt to approximate the knowledge that a student possesses about a particular subject, just as a teacher might assess her student, while affect detectors classify the behavior and emotional states of students. Interventions attempt to improve the experience of students at critical times. My own work marries affect detectors with interventions in an attempt to improve MOOC discussion forums.

Making discussion forums smarter
I became interested in augmenting online education with artificial intelligence a couple of years ago, after listening to a talk at Google and speaking with Peter Norvig. That interest lay dormant for a year, until I began working as a teaching assistant for a Stanford MOOC. I spent a lot of time answering questions in the discussion forum, questions asked by thousands of students. Helping these students was fulfilling work, to be sure. But slogging through a single, unorganized stream of questions and manually identifying urgent ones wasn’t particularly fun. I would have loved an automatically organized inbox of questions.

The YouEDU architecture. Posts are fed to a classifier that screens posts for confusion, and our recommender then fetches clips relevant to the confused posts.

That these discussion forums were still “dumb”, so to speak, surprised me. I reached out to the platforms team of Stanford Online Learning, who in turn sent me to Andreas Paepcke, a senior research scientist (and, I should add, an incredibly supportive and kind mentor). It turned out that I wasn’t the only one who wished for a more intelligent discussion forum. I paired up with a student of Andreas’ to tackle the problem of automatically classifying posts by the affect or sentiment they expressed.

Our initial efforts at affect detection were circumscribed by the data available to us. Machine learning tasks like ours need human-tagged data — in our case, we needed a dataset of forum posts in which each post was tagged with information about the affect expressed in it. At the time, no such dataset existed. So we created one: the Stanford MOOCPosts dataset, available to researchers upon request.

The dataset powered the rest of our work. It enabled us to build a model to predict whether or not a post expressed confusion, as well as a pipeline to recommend relevant clips from instructional videos to the author of that confused post.

YouEDU was not meant to replace teaching assistants in MOOCs. Videos are notoriously difficult to search through (they’re not indexed, like books are), and YouEDU simply helps confused students find content relevant to the topic they’re confused about. Our affect classifiers can also be used outside of YouEDU — for example, they could be used to highlight urgent posts for the instructors, or even for other students in the forum.

If you’d like to learn more about our work, you’re welcome to look at the publication, my slide deck, or the edxclassify repository.

Data mining is not nefarious
My experience at EDM was a great one. I learned lots from learned people, made lasting friends and memories, and so on. I could talk at length about interesting talks and papers — like Streeter’s mixture modeling of learning curves, or MacLellan’s slip-aware bounded logistic regression. But I won’t. You can skim the proceedings on your own time.

The EDM community is tightly knit, or at least more tightly knit that that of ACM’s Learning @ Scale, the only other education conference I’ve attended. And though no raves were attended, EDM-ers did close the conference by dancing the night away in a bar, after dining, drinking, and singing upon the roof of the Reina Victoria.

Festivities aside, a shared sense of urgency pulsed through the conference. As of late, the public has grown increasingly suspicious of those who collect and analyze data en masse. We see it in popular culture: Ex Machina, for example, with its damning rendition of a Google-like Big Brother who recklessly and dangerously abuses data, captures the sentiment well. The public’s suspicion is certainly justified, but its non-discriminating nature becomes problematic for EDM-ers. The public fears that those analyzing student data are, like Ex Machina’s tragic genius, either greedy, hoping to manipulate education in order to monetize it, or careless, liable to botch students’ education altogether. For the record, neither is true. EDM researchers are both well-intentioned and competent.

What’s an EDM-er to do? Some at the conference casually floated the idea of rebranding — for example, perhaps they should call themselves educational data scientists, not miners. Perhaps, too, they should write to legislators to convince them that their particular data mining tasks are not nefarious. In a rare example of representative government working as intended, Senator Vitter of Louisiana recently introduced a bill that threatens to cripple EDM efforts. The Student Privacy Protection Act, a proposed amendment to FERPA, would make it illegal for researchers to, among other things, assess or model psychological states, behaviors, or beliefs.

Were Vitter’s bill to go into effect as law, it would potentially wipe out the entire field of affect modeling. What’s more, the bill would ultimately harm the experience of students enrolled in online courses — as I hope YouEDU shows, students’ online learning experiences can be significantly improved by intelligent systems.

Now, that said, I understand why folks might fear a computational system that could predict behavior. I could imagine a scenario in which an educator mapped predicted affect to different curricula; students who appeared confused would be placed in a slow curricula, while those who appeared knowledgeable would be placed in a faster one. Such tracking would likely fulfill the prophecies of the predictor, creating an artificial and unfortunate gap between the “confused” and “knowledgeable” students. In this scenario, however, the predictive model isn’t inherently harmful to the student’s education. The problem instead lies with the misguided educator. Indeed, consider the following paper-and-pencil equivalent of this situation. Our educational system puts too much stock in tests, a type of predictive tool. Perform poorly on a single math test in the fifth grade and you might be placed onto a slow track, making it even less likely you’ll end up mathematically inclined. Does that mean we should ban tests outright? Probably not. It just means that we should think more carefully about the policies we design around tests. And so it is for the virtual: It is the human abuse of predictive modeling, rather than predictive modeling in and of itself, that we should guard against.

# Machines that Learn: Making Distributed Storage Smarter

Equipped with shiny machine learning tools, computer scientists these days are optimizing lots of previously manual tasks. The idea is that AI can make certain procedures smarter — we can capitalize on a system’s predictability and implicit structure to automate at least part of the task at hand.

For all the progress we’ve made recently in soulmate-searching pipelines and essay-grading tools, I haven’t seen too many applications of AI to computer infrastructure. AI could solve interesting infrastructure problems, particularly when it comes to distributed systems — in a reflexive sort of way, machines can and should use machine learning to learn more about themselves.

Being smart about it: The case for intelligent storage systems
Distributed systems cover a lot of ground; to stop myself from rambling too much, I’ll focus on distributed storage systems here. In these systems, lots of machines work together to provide a transparent storage solution to some number of clients. Different machines often see different workloads — for example, some machines might store particularly hot (i.e., frequently accessed) data, while others might be home to colder data. The variability in workloads matters because particular workloads play better with particular types of storage media.

Manually optimizing for these workloads isn’t feasible. There are just too many files and independent workloads for humans to make good, case-by-case decisions about where files should be stored.

The ideal, then, is a smart storage system. A smart system would automatically adapt to whatever workload we threw at it. By analyzing file system metadata, it would make predictions about files’ eventual usage characteristics and decide where to store them accordingly. If a file looked like it would be hot or short-lived, the smart system could cache it in RAM or flash; otherwise, it could put it on disk. Creating policies with predictive policy would not only minimize IT administrators’ work, but would also boost performance, lowering latency and increasing throughput on average.

From the past, a view into the future: Self-* storage systems
To my surprise, there doesn’t seem to be a whole lot of work in making storage systems smarter. The largest effort I came across was the self-* storage initiative, undertaken by a few faculty over at CMU back in 2003. From their white paper,

‘self-* storage systems’ [are] self-configuring, self-organizing, self-tuning, self-healing, self-managing systems of storage bricks …, each consisting of CPU(s), RAM, and a number of disks. Designing self-*-ness in from the start allows construction of high-performance, high-reliability storage infrastructures from weaker, less reliable base units …

There’s a wealth of interesting content to be found in the self-* papers. In particular, in Attribute-Based File Prediction, the authors propose ways to exploit metadata and information latent in filenames to bucket files into binary classes related to their sizes, access permissions, and lifespans.

Predictions were made using decision trees, which were constructed using the ID3 algorithm. With the root node corresponding to the entire feature space, ID3 splits the tree into two sub-trees corresponding to the feature that seems like the best predictor (the metric used here is typically information gain, but the self-* project used the chi-squared statistic). The algorithm then recursively builds a tree whose leaf nodes correspond to classes. As an aside, it turns out that ID3 tends to overfit training data — these lecture notes discuss ways to prune decision trees in an attempt to increase their predictive power.

Diagram from “File Classification in Self-* Storage Systems”, by Ganger, et. al.

The features used were coarse. For example, files’ basenames were broken into three chunks: prefixes (characters preceding the first period), extensions (characters preceding the last period), and middles (everything in between); directories were disregarded. These simple heuristics proved fairly effective; prediction accuracy didn’t fall below 70 percent.

It’s not clear how a decision tree trained using these same features would perform if more granular predictions were desired, or if the observed filenames were less structured (what if they lacked delimiters?). I could imagine a much richer feature set for filenames; possible features might include the number of directories, the ratio of numbers to characters, TTLs, etc.

From research to reality: Picking up where self-* left off
The self-* project was an ambitious one — the researchers planned to launch a large scale implementation of it called Ursa Major, which would offer 100s of terabytes of automatically tuned storage to CMU researchers.

I recently corresponded with CMU professor Greg Ganger, who led the self-* project. It turns out that Ursa Major never fully materialized, though significant and practical progress in smart storage systems was made nonetheless. That the self-* project lives no longer doesn’t mean that idea of smart storage systems should die, too. The onus lies with us to pick up the torch, and to continue where the folks at CMU left off.

# A Small Glass Box

I took a trip up to San Francisco’s Exploratorium, some two weeks past. Though recently relocated, the Exploratorium is comfortably familiar. It’s still packed with exhibits that span the spectrum from mystically enchanting (one station lets museum-goers create delicate purple auroras that warp and spiral in a glass tube) to delightfully curious (another rapidly spins dozens of Lego Batmen and dolphins, making them dance to the tune of the Caped Crusader’s catchy theme song).

Exhibits at this unconventional museum are designed to stir your curiosity. It’s hard to resist playing with them, but of course there’s no need to — almost everything is hands-on. Photo by Sara Yang.

I meandered through the museum, all the while searching for a particular treasure. Just before the closing bells rung, I stumbled upon it: the cloud chamber, a large, humming, refrigerated box with a sky-facing window that allows for the observation of cosmic radiation. Cosmic rays hail from the beyond the solar system. They collide in the earth’s atmosphere, and minuscule particles rain torrentially upon us in the aftermath. The cloud chamber makes an otherwise imperceptible and invisible downpour from the heavens palpably visible, if only for a fleeting moment.

Our homemade cloud chamber consists of a small box with a lid lined with black felt. In order to nudge muons into uncloaking themselves, we douse the felt with isopropanol and heat it from above with my desk lamp.

The sight brought me back four years, to the first time I saw muons zip hither and thither through the same chamber. I had spent the better part of that year in my garage, tinkering with a friend of mine by the name of Hemanth on our own chamber for a science project.

On a nostalgic whim, I called up Hemanth the next day. We decided to fire up the chamber once again, for old times’ sake. We scrounged the necessary components, lugged them to Hemanth’s garage, and got started. Pulverizing dry ice, we began working to the sound of snow crunching underfoot and the sight of fumes eddying about.

With thick gloves and sturdy hammers, we first crush the dry ice into a coarse powder and pack it tightly into a Styrofoam base, on top of which the chamber sits. The one-two punch of a cooling source and a heating source forces the alcohol into a supersaturated, supercooled state. Muons streaking through the chamber rip electrons off the vapor, causing water molecules to visibly condense around their paths.

I followed our procedure as if on autopilot; my mind wandered and let bittersweet memories leak. We packed the dry ice into a foam base (days colored by failed prototype runs), doused the chamber with isopropanol (afternoons brightened by faint flashes of muons), and positioned my lamp atop the box (nights illuminated by the bluish glow of computer monitors).

Our small glass box held us rapt, as we saw the ghosts of muons pass through it. Unfortunately, the streaks are difficult to capture on camera.

We left the chamber to run for some time. When we returned, muons were streaking visibly through it. Spellbound, we lingered by the chamber for over half an hour. Four years ago, an anxious desire to create something novel and a preoccupation with results left little room for wonder. Now, we could stare into the cloud chamber for but the simple sake of doing so. The muons that passed through it, falling like delicate strands of spider web, were, paradoxically, both otherworldly and earthly. Our small glass box, glued together by a mom-and-pop craft shop, had become a window into the universe’s secrets. The sight was as humbling as it was beautiful.

# A Breakdown of IGN’s Game Rating Distribution

If you’d like to take a look at the data referenced in this post and the script I wrote to gather them, feel free to head over to my Github.

I’ll admit it: I take stock in video game reviews. I don’t read them religiously, nor am I insulted when a site rates a favorite of mine a bit too low. I read reviews to find out which games I might like. With so many games and so little time, reviews and ratings are, for me, a much-needed filtering process.

But video game ratings come with problems. On a 10 point scale, what’s the difference between a 7.3 and a 7.5? Even a 7 and an 8? And why does it seem like reviewers hardly dish out scores from the bottom half of their scales? How can reviewers package their opinions, highly subjective and finicky things that they are, into definitive scores? Numbers feel a lot more objective than words. It might be this perceived objectivity that makes people protest the seemingly arbitrary scores that reviewers select. (I wouldn’t call these ratings arbitrary — they’re backed by opinion. Subjective? Yes.)

Present-day game rating aggregators add to these problems. Averaging across multiple sites, each with their own distributions, makes things messy. A score of 7 from one site very likely might be equivalent to a 5 from another. And what about those troublesome letter-grade ratings? Metacritic converts Cs to 50s — but last time I checked, a C mapped to a 70 percent or so in school. Metacritic claims that they generate their aggregate scores after running their data through a weighted, proprietary algorithm, and I’m sure they do. But I question their algorithm’s efficacy. Pick a few games on their website and do an old-fashioned average of the scores they list. The score you come out with will likely closely resemble their aggregate score.

But I digress.

I decided to do some investigation into the nature of game ratings. I’m currently in the process of building a dataset of game ratings from different sites. I started with IGN. After a bit of web scraping, I collected review data on the 75,005 games that IGN kept track of as of July 13, 2013. Of those, only 17,027 had ratings in IGN’s index (others were not rated, marked as “NR”). I coded a python script to collect the data and used BeautifulSoup for HTML parsing. I analyzed the data with an evaluation copy of Wizard, a suite of statistical tools.

Without further ado, here are some summary statistics of IGN’s rating distribution, peppered with some tidbits of information that I found interesting.

Summary statistics
Data retrieval date: July 13, 2013
Population size: 17,027 rated games

A distribution of IGN’s 17,027 rated games. The distribution has a mean of 6.877 with a standard deviation of 1.762, and a median of 7.2 with an IQR of 2.1. All but one of the 51 0s are data collection errors that I didn’t prune out.

Mean score: Approximately 6.9
Standard deviation: Approximately 1.7
Skew: left

Interquartile range: 2.1
Lower quartile: 6.0
Median score: 7.2
Upper quartile 8.1

IGN’s index erroneously assigns a number of games 0s in their index (at least one game, though, actually did manage to earn a zero). I didn’t manually prune these entries from my data, so the above summary statistics are slightly off.

The highs and the lows
Only 314 games, or about two percent of rated games, received a score of 9.5 or above. IGN decorated 38 games with 10s. Classic games (the Zelda, Mario, and Pokemon franchises) dominated here. Some newer games did manage to break into the Hall of 10s, notably Naughty Dog’s third Uncharted entry and their recent The Last of Us. Rockstar’s GTA games and Red Dead DLC also carved out spots for themselves, as did a couple of Kojima’s Metal Gear Solid games. Some 10s you might not have heard of include Checkered Flag, Joust, and Shanghai (all Atari Lynx games), and Tornado Mania, a mobile game. Infinity Blade II was the sole representer of the iPhone.

2,913 games managed to be bad enough to nab a score of 5 or less, accounting for about 17 percent of total games. With a zero, Olympic Hockey Nagano ’98 boasts the lowest score of all games. Looney Tunes: Back in Action: Zany Race followed closely with a 0.5. Two other games, Extreme PaintBrawl and Action Girlz Racing, joined the less-than-one ranks.

Of the major home entertainment platforms (PC, Xbox 360, PS3, and the WiiU), the Wii had the lowest median score.

Of the major modern consoles, the Nintendo Wii had the lowest median rating (a 6.8). The Wii U, Xbox 360, PS3, and PC have median scores of 7.5.

A closing remark
With a median of 7.2 and an IQR of 2.1 (i.e., 50 percent of scores lie between 6.0 and 8.1), it does look like IGN awards higher scores more often than not. This does not mean that they’re doing anything illicit (this might seem obvious, but you’d be surprised at some of the shoddy “journalism” out there that sensationally misinterprets data). Perhaps IGN thinks that most games just aren’t that bad.

I’ve got a hunch that other review sites’ distributions won’t match IGN’s, just as they likely won’t match each other. So I’ll gather more data — maybe I’ll be able to do something with them.

Update, July 28, 2013: This post now links to my Github repository containing the source code and data referenced here.

# Shaving off Seconds: Making your Website Faster

Does your website load too slowly? Try using a content delivery network and caching static content. Read on for details and a quick-and-easy three-step guide.

I’ve never been satisfied with Debug Mind’s speed. If you ever visited this site before, you probably drummed your fingers impatiently while your browser slowly painted the web page down the screen. Having done more than my fair share of finger-drumming, I decided to make this site faster: Debug Mind now uses both CloudFlare, a content delivery network and optimizing service, and W3 Total Cache, a WordPress caching plugin.

Below you’ll find an outline of the steps I took to speed up my website. While the third step is WordPress-specific, the first two should be applicable to most websites.

Step one: Diagnose the bottlenecks
Before making your website faster, you might want to know why it’s slow. If you don’t care for the “why” and simply want results, skip ahead to step two. For those of you curious about why your site loads slowly, I recommend completing step one.

Analyze HTTP requests with Chrome Developer Tools
The Chrome browser packages a handy set of tools that you can use to analyze websites. If for some reason you don’t want Chrome on your machine, alternatives to Developer Tools exist for other browsers and a Google search will likely find them.

Chrome Developer Tools can help you diagnose speed bottlenecks. Often times uncached images and poorly written plugins slow your website down.

Assuming that you’re using Chrome, navigate to the browser menu’s tools section and click on Developer Tools. Click on the Network tab and then navigate over to the website you want to analyze. Once the website loads, you’ll see a list of requests that your browser made while fetching your site. The timeline tells you how long each request took. Isolate requests with the longest streaks in the timeline. These are the things that are slowing your website down. Chances are that the bottlenecks are image loads and other static media. Other things — perhaps plugins, if you’re on WordPress or something comparable — might also contribute to the lag. read more »

# Lesson Learned: Back up Your Website

I was careless — in over two years, I never made a backup of this site. You can imagine how I felt when, two days ago, I accidentally deleted every single comment posted here. Fortunately, it turns out that Hostmonster creates courtesy backups for its customers.

I chatted Hostmonster’s tech support and asked them to restore my site for me. They went right to it, and an hour later the restore finished. Or so they claimed — my database was left untouched. I called them up and asked them specifically to restore the database; déjà vu ensued. Since Hostmonster gives me access to these backups, I did a manual restore and lo and behold! The comments had returned.

I lucked out — my comments very well could have disappeared forever. You can bet I’ll religiously backup my site from now (well, actually yesterday) on.

So if you ever find yourself in desperate need of a backup, contact your web hosting service — you might luck out like I did. If your luck runs out with technical support, you can do a manual database restore through (e.g.) phpMyAdmin.

If you don’t already back up your website, I strongly encourage you to start. You’ll probably want to make backups of your database and all your files. An FTP client like Filezilla, coupled with phpMyAdmin, makes backing up your site really simple.

Have any questions about creating and / or restoring backups? I’m happy to help! Just leave a comment and I’ll get back to you.