Author: Zachary C. Lipton

Zachary Chase Lipton is an assistant professor at Carnegie Mellon University. He is interested in both core machine learning methodology and applications to healthcare and dialogue systems. He is also a visiting scientist at Amazon AI, and has worked with Amazon Core Machine Learning, Microsoft Research Redmond, & Microsoft Research Bangalore.

Is This a Paper Review?

With paper submissions rocketing and the pool of experienced researchers stagnant, machine learning conferences, backs to the wall, have made the inevitable choice to inflate the ranks of peer reviewers, in the hopes that a fortified pool might handle the onslaught.

Infographic depicting NIPS submissions over time. The red bar plots fabricated data from the future.

With nearly every professor and senior grad student already reviewing at capacity, conference organizers have gotten creative, finding reviewers in unlikely places. Reached for comment, ICLR’s program chairs declined to reveal their strategy for scouting out untapped reviewing talent, indicating that these trade secrets might be exploited by rivals NeurIPS and ICML. Fortunately, on condition of anonymity, several (less senior) ICLR officials agreed to discuss a few unusual sources they’ve tapped:

  1. All of /r/machinelearning
  2. Twitter users who follow @ylecun
  3. Holders of registered .ai & .ml domains
  4. Commenters from ML articles posted to Hacker News
  5. YouTube commenters on Siraj Raval deep learning rap videos
  6. Employees of entities registered as owners of .ai & .ml domains
  7. Everyone camped within 4° of Andrej Karpathy at Burning Man
  8. GitHub handles forking TensorFlow, Pytorch, or MXNet in last 6 mos.
  9. A joint venture with Udacity to make reviewing for ICLR a course project for their Intro to Deep Learning class

With so many new referees, perhaps it’s not surprising to see, sprinkled among the stronger, more traditional reviews, a number of unusual ones: some oddly short, some oddly … net-speak (“imho, srs paper 4 real”), and some that challenge assumptions about what shared knowledge is pre-requisite for membership in this community (“who are you to say this matrix is degenerate?”) .

However, these reviews, which to a casual onlooker might signify incompetence, belie the earnest efforts of a cadre of new reviewers to rise to the occasion. I know this because, fortuitously, many of these new reviewers are avid readers of Approximately Correct, and for the last few weeks my email box has been overflowing with earnest questions from well-meaning neophytes.

If you’ve ever taught a course before, it might not come as a surprise  that their questions overlapped substantially.  So while we don’t normally post QA-type articles on Approximately Correct, an exception here seems both appropriate and efficient. I’ve compiled a few exemplar questions and provided succinct answers here in a rare QA piece that we’ll call “Is This a Paper Review”?

Henry in Pasadena writes:

Dear Approximately Correct,

I was assigned to review a paper. I read the abstract and formed an opinion tangentially related to the topic of the paper. Then I wrote a paragraph partly expressing my opinion and partly arguing with one of the anonymous commenters on an unrelated topic. This is standard practice on Hacker News, where I have earned over 2000 upvotes for similar reviews, which together form the basis of my qualifications to review for ICLR. Is this a paper review?

AC: No, that  is not a paper review.

Pandit in Mysore writes:

Dear Approximately Correct,

I began to read a paper about the convergence of gradient descent. Once they said “limit” I got lost, so I skipped to the back of the paper, where I noticed that they did not do any experiments on ImageNet. I wrote a one-line review. The title said “Not an expert but WTF?” and the body said “No experiments on ImageNet?” Is this a paper review?

AC: No, that  is not a paper review.

Xiao in Shanghai writes:

Dear Approximately Correct,

I trained an LSTM on past ICLR reviews. Then I ran it with the softmax temperature set to .01. The output was “Not novel [EOS].” I entered this in OpenReview. Is this a paper review?

AC: No, that  is not a paper review.

Jordan in Boulder writes:

Dear Approximately Correct,

When reviewing this paper, I noticed that it was vaguely similar in certain ways to an idea that I had in 1987. While I like the idea (as you might imagine), I assigned the paper a middling score, with one half of the review a solid discussion of the technical work and the other half devoted exclusively to enumerating my own papers and demanding that the author cite them. Is this a paper review?

AC: This sounds like a problematic paper review. But it could be a good review if you increase your score to what you think it might be were you dispassionate, tone down the stuff about your own papers, and send a thoughtful note to the metareviewer indicating a minor conflict of interest.

Rachel in New Jersey writes:

I read the paper. In the first 2 pages, there were 10 mathematical mistakes, including some that made the entire contribution of the paper obviously wrong. I stopped reading to conserve my time and wrote a short one-paragraph review that indicated the mistakes and said “not suitable for publication at ICLR.” Is this a paper review?

AC: While ordinarily so short a review might not be appropriate, this is a clear exception. Excellent review!

This piece was loosely inspired by Jason Feifer’s Is This a Selfie? 

Related Posts

Troubling Trends in Machine Learning Scholarship

By Zachary C. Lipton* & Jacob Steinhardt*
*equal authorship

Originally presented at ICML 2018: Machine Learning Debates
[arXiv link]

1   Introduction

Collectively, machine learning (ML) researchers are engaged in the creation and dissemination of knowledge about data-driven algorithms. In a given paper, researchers might aspire to any subset of the following goals, among others: to theoretically characterize what is learnable, to obtain understanding through empirically rigorous experiments, or to build a working system that has high predictive accuracy. While determining which knowledge warrants inquiry may be subjective, once the topic is fixed, papers are most valuable to the community when they act in service of the reader, creating foundational knowledge and communicating as clearly as possible.

What sort of papers best serve their readers? We can enumerate desirable characteristics: these papers should (i) provide intuition to aid the reader’s understanding, but clearly distinguish it from stronger conclusions supported by evidence; (ii) describe empirical investigations that consider and rule out alternative hypotheses [62]; (iii) make clear the relationship between theoretical analysis and intuitive or empirical claims [64]; and (iv) use language to empower the reader, choosing terminology to avoid misleading or unproven connotations, collisions with other definitions, or conflation with other related but distinct concepts [56].

Recent progress in machine learning comes despite frequent departures from these ideals. In this paper, we focus on the following four patterns that appear to us to be trending in ML scholarship:

  1. Failure to distinguish between explanation and speculation.
  2. Failure to identify the sources of empirical gains, e.g. emphasizing unnecessary modifications to neural architectures when gains actually stem from hyper-parameter tuning.
  3. Mathiness: the use of mathematics that obfuscates or impresses rather than clarifies, e.g. by confusing technical and non-technical concepts.
  4. Misuse of language, e.g. by choosing terms of art with colloquial connotations or by overloading established technical terms.

Continue reading “Troubling Trends in Machine Learning Scholarship”

The Blockchain Bubble will Pop, What Next?

Last week, I flew from London to Tel Aviv. The man sitting to my right was a road warrior, just this side of a late-night bender in London. He was rocking an ostentatious pair of headphones and a pair of pants ripped wide apart at both knees. Perhaps a D.J.? At some point, circumstances emerged for us to commiserate over the experience of flying on Easyjet (not the easiest). Soon after, we stumbled through the obligatory airplane smalltalk: Where are you going? What do you do?

Turns out I was flying next to the CEO of an AI+Blockchain startup.

This image ran in an article in the Express discussing conspiracy theories suggesting that cryptocurrencies were invented by an advanced artificial intelligence.

It’s always a bit surreal when I learn of entrepreneurs combining AI with blockchain technology. For the past few years, whenever I found my myself bored among Silicon Valley socialites, this was my go-to satirical startup. What do you do? Startup CEO. What does your startup do? Deep learning on the blockchain… in The Cloud. Whoa. Continue reading “The Blockchain Bubble will Pop, What Next?”

From AI to ML to AI: On Swirling Nomenclature & Slurried Thought

Artificial intelligence is transforming the way we work (Venture Beat), turning all of us into hyper-productive business centaurs (The Next Web).  Artificial intelligence will merge with human brains to transform the way we think (The Verge). Artificial intelligence is the new electricity (Andrew Ng).  Within five years, artificial intelligence will be behind your every decision (Ginni Rometty of IBM via Computer World ).

Before committing all future posts to the coming revolution, or abandoning the blog altogether to beseech good favor from our AI overlords at the AI church, perhaps we should ask, why are today’s headlines, startups and even academic institutions suddenly all embracing the term artificial intelligence (AI)?

In this blog post, I hope to prod all stakeholders (researchers, entrepreneurs, venture capitalists, journalists, think-fluencers, and casual observers alike) to ask the following questions:

  1. What substantive transformation does this switch in the nomenclature from machine learning (ML) to  artificial intelligence (AI) signal?
  2. If the research hasn’t categorically changed, then why are we rebranding it?
  3. What are the dangers, to both scholarship and society, of mindlessly shifting the way we talk about research to maximize buzz?

Continue reading “From AI to ML to AI: On Swirling Nomenclature & Slurried Thought”

AI Researcher Joins Johnson & Johnson, to Make More than $19 Squillion

Three weeks ago, New York Times reporter Cade Metz sent shockwaves through society with a startling announcement that A.I. researchers were making more than $1 Million dollars, even at a nonprofit!

AI super-hero and newly minted squillionaire Zachary Chase Lipton feeds a wallaby bitcoins while vacationing on Elon Musk’s interplanetary animal preserve on the Martian plains.

Within hours, I received multiple emails. Parents, friends, old classmates, my girlfriend all sent emails. Did you see the article? Maybe they wanted me to know what riches a life in private industry had in store for me? Perhaps they were curious if I was already bathing in Cristal, shopping for yachts, or planning to purchase an atoll among the Maldives? Perhaps the communist sympathizers in my social circles had renewed admiration for my abstention from such extreme opulence.

Continue reading “AI Researcher Joins Johnson & Johnson, to Make More than $19 Squillion”

Heuristics for Scientific Writing (a Machine Learning Perspective)

It’s January 28th and I should be working on my paper submissions. So should you! But why write when we can meta-write? ICML deadlines loom only twelve days away. And KDD follows shortly after. The schedule hardly lets up there, with ACL, COLT, ECML, UAI, and NIPS all approaching before the summer break. Thousands of papers will be submitted to each.

The tremendous surge of interest in machine learning along with ML’s democratization due to open source software, YouTube coursework, and the availability of preprint articles are all exciting happenings. But every rose has a thorn. Of the thousands of papers that hit the arXiv in the coming month, many will be unreadable. Poor writing will damn some to rejection while others will fail to reach their potential impact. Even among accepted and influential papers, careless writing will sow confusion and damn some papers to later criticism for sloppy scholarship (you better hope Ali Rahimi and Ben Recht don’t win another test of time award!).

But wait, there’s hope! Your technical writing doesn’t have to stink. Over the course of my academic career, I’ve formed strong opinions about how to write a paper (as with all opinions, you may disagree). While one-liners can be trite, I learned early in my PhD from Charles Elkan that many important heuristics for scientific  paper writing can be summed up in snappy maxims.  These days, as I work with younger students, teaching them how to write clear scientific prose, I find myself repeating these one-liners, and occasionally inventing new ones.

The following list consists of easy-to-memorize dictates, each with a short explanation. Some address language, some address positioning, and others address aesthetics. Most are just heuristics so take each with a grain of salt, especially when they come into conflict. But if you’re going to violate one of them, have a good reason. This can be a living document, if you have some gems, please leave a comment.

Continue reading “Heuristics for Scientific Writing (a Machine Learning Perspective)”

Cathy O’Neil Sleepwalks into Punditry

On the BBC’s anthology series Black Mirror, each episode explores a near-future dystopia. In each episode, a small extrapolation from current technological trends leads us into a terrifying future. The series should conjure modern-day Cassandras like Cathy O’Neil, who has made a second career out of exhorting caution against algorithmic decision-making run amok. In particular, she warns that algorithmic decision-making systems, if implemented carelessly, might increase inequality, twist incentives, and perpetrate undesirable feedback loops. For example, a predictive policing system might direct aggressive policing in poor neighborhoods, drive up arrests, depress employment, orphan children, and lead, ultimately, to more crime.

Continue reading “Cathy O’Neil Sleepwalks into Punditry”

ICML 2018 Registrations Sell Out Before Submission Deadline

In a shocking tweet, organizers of the 35th International Conference on Machine Learning (ICML 2018) announced today, through an official Twitter account, that this year’s conference has sold out. The announcement came as a surprise owing to the  timing.  Slated to occur in July, 2018, the conference has historically been attended by professors and graduate student authors, who attend primarily to present their research to audience of peers. With the submission deadline set for February 9th and registrations already closed, it remains unclear if and how authors of accepted papers might attend.

Continue reading “ICML 2018 Registrations Sell Out Before Submission Deadline”

Embracing the Diffusion of AI Research in Yerevan, Armenia

In July of this year, NYU Professor of Psychology Gary Marcus argued in the New York Times that AI is stuck, failing to progress towards a more general, human-like intelligence. To liberate AI from it’s current stuckness, he proposed a big science initiative. Covetously referencing the thousands of bodies (employed at) and billions of dollars (lavished on) CERN, he wondered whether we ought to launch a concerted international AI mission.

Perhaps owing to my New York upbringing, I admire Gary’s contrarian instincts. With the press pouring forth a fine slurry of real and imagined progress in machine learning, celebrating any story about AI as a major breakthrough, it’s hard to state the value of a relentless critical voice reminding the community of our remaining shortcomings.

But despite the seductive flash of big science and Gary’s irresistible chutzpah, I don’t buy this particular recommendation. Billion-dollar price tags and frightening head counts are bugs, not features. Big science requires getting those thousands of heads to agree about what questions are worth asking. A useful heuristic that applies here:

The larger an organization, the simpler its elevator pitch needs to be.

Machine learning research doesn’t yet have an agreed-upon elevator pitch. And trying to coerce one prematurely seems like a waste of resources. Dissent and diversity of viewpoints are valuable. Big science mandates overbearing bureaucracy and some amount of groupthink, and sometimes that’s necessary. If, as in physics, an entire field already agrees about what experiments come next and these happen to be thousand-man jobs costing billions of dollars, then so be it

Continue reading “Embracing the Diffusion of AI Research in Yerevan, Armenia”

A Pedant’s Guide to MLHC 2017

By David Kale and Zachary Lipton

Starting Friday, August 18th and lasting two days, Northeastern University in Boston hosted the eighth annual Machine Learning for Healthcare (MLHC) conference. This year marked MLHC’s second year as a publishing conference with an archival proceedings in the Journal of Machine Learning Research (JMLR). Incidentally, the transition to formal publishing venue in 2016 coincided with the name change to MLHC from Meaningful Use of Complex Medical Data, denoted by the memorable acronym MUCMD (pronounced MUCK-MED).

From its beginnings at Children’s Hospital Los Angeles as a non-archival symposium, the meeting set out to address the following problem:

  • Machine learning, even then, was seen as a powerful tool that can confer insights and improve processes in domains with well-defined problems and large quantities of interesting data.
  • In the course of treating patients, hospitals produce massive streams of data, including vital signs, lab tests, medication orders, radiologic imaging, and clinical notes, and record many health outcomes of interest, e.g., diagnoses. Moreover, numerous tasks in clinical care present as well-posed machine learning problems.
  • However, despite the clear opportunities, there was surprisingly little collaboration between machine learning experts and clinicians. Few papers at elite machine learning conferences addressed problems in clinical health and few machine learning papers were submitted to the elite medical journals.

Continue reading “A Pedant’s Guide to MLHC 2017”