How to make your ML research more impactful
We machine learning researchers all have a very limited amount of time to spend on reading research and there is only so few projects we can take on at a time. Thus, it is paramount to understand what areas of research excite you and hold promise for the future. In May 2018, I looked at precisely this question during a course at University College London:
- What makes ML research impactful?
- And what can you do to increase your impact?
I feel like everyone should at least ponder about this question for a bit, or better, read my technical report on the topic and/or the summary I will provide in this blog post.
What can you do to increase your impact?
In my analysis, I focused on the field of deep learning. I studied the co-author network of three important deep learning researchers and their publications. I looked at the contents of these publications, the context in which these were published, and the changing citation count over time. Here we interpret citation count as a metric for impact which has its limits in particular when looking at the effects on society at scale. In my paper, I also discuss different metrics of impact, but here we focus on citation count as a metric for impact within the scientific community. One might argue that outside of the community a lot of machine learning research is quickly converted to applications by industry. In the following, I present my most important findings and actionable items.
Demonstration of large-scale practical success.
Many particularly successful papers got the majority of their citations only decades later after the large-scale practical success of their methods was evident.
In general, it can be said that success depends on the ability of the approach to be scaled up to large problems.
Thus, always try to show your algorithm works great in large scale!
Focus on novel techniques and large margin improvements.
Small improvements on benchmarks are quickly surpassed.
Working on applications only leads to high impact in the form of citations and adoption in the research community if it uses novel techniques that improve over existing (or non-existing) work by a large margin.
Perseverance.
Because not all ideas will work out and demonstrating large-scale practical success is hard, great ideas in deep learning research often require perseverance for long periods of time.
Do not follow the mainstream.
Backpropagation and LSTMs as examples for impactful ML research demonstrate that not the mainstream research established itself but the novel ideas that were not generally accepted at the time.
General learning algorithms over applications.
Effort should be focused on general learning algorithms over applications.
None of the impactful papers I investigated solely applied existing techniques to a new application.
Trial and error.
Many publications of famous authors are barely cited; research is trial and error.
Don’t be frustrated!
A good intuition.
The most cited publications are distributed among a very small number of authors.
Which means these researchers must have a good intuition about what kind of problems are relevant, and what a good solution should look like.
Learn from them!
What can the community do?
Introduce a ‘Crazy Work Award’ and other incentives.
Many later very successful ideas were rather unpopular during their inception.
We should encourage research groups, conferences, and journals to nurture crazy and currently unpopular ideas.
Conclusions
There you have it. I hope we can learn from my findings and produce exceptional work that pushes our field way beyond what is possible today.
A final word of caution.
Please be aware that many of my observations might not generalize well into the future.
Furthermore, while I tried to extract actionable items from my findings, it may very well be that great research is still mostly determined by chance.