Some wisdoms for ML research
On the importance of asking questions
[By Danial Ally and Jun Liu]
Most people assume that they know answers. Their assumptions actually hold them back from knowing the truth. Truly, you want to ask questions to gain clarity about the direction you are heading. The fact is that many people don’t ask any questions as they habitually guess their way by creating answers for themselves. To become a great researcher, don’t answer your questions, but question your answers. When you need to know the facts, you must inquire, not just make assumptions. Many people don’t want to ask questions because it exposes them to confront the reality of their circumstance, which may scare them. Moreover, asking questions forces them into the laborious task of thinking, which is why they fail to ask questions.
On answering your questions when getting stuck
[By Richard Sutton]
In thinking on important questions, you will often reach an apparent dead end, with no where to go. Here are some techniques for moving forward again:
Define your terms
Go multiple (what are some of the conceivable answers?)
Go meta (what would an answer look like? What properties would it have?)
Retreat (to a clearer question that you can make progress on)
On authority in science
[By Richard Sutton]
Don’t be impressed by what you don’t understand.
Don’t try to impress others by what they don’t understand.
You should be brave, ambitious but also humble and transparant.
Humble before the great task - Understanding the mind. Nature is subtle but not devious. It is waiting to be discovered if we can only see.
Your thoughts are, potentially, of great value.
On thinking carefully and productively
The best way (to think carefully and productively) is to write for yourself and discuss with others. [Richard Sutton]
If others would think as hard as I did, then they would get similar results. [Issac Newton]
If you use the same approach to a problem as others did, do not expect an outcome different from others’. If you want to get a different outcome to the same problem, try different approaches.
The quality and originality of your work is rougly proportional to:
How well you understand the question, understand your data, select good notation (inspired by Terence Tao)
Developing new techniques is cool and important, but remember the goal is to answer questions, and developed techniques should be treated as a mean to an answer, not an initial target. Thus, keep asking bunch of silly questions, keep trying to answer these questions (even though you might oftenly fail to answer, but that’s OK, asking questions and trying to answer them will culmunate in something interestingly and insightfully new).
On writing and solving
Select good notations
Understand your questions, understand your problems
Thinking hard is the only way to make an actual progress
If you dont put your ideas into a gluing story, you cannot convince yourself enough, and reviewers and readers won’t bother to buy it.
On being genuinely practical
[By Mark Twain]
The secret of getting ahead is getting started.
The secret of getting started is breaking your complex,
overwhelming tasks into small manageable tasks, and then starting on the first one.
On being wise in decision making
Knowing that several wise life decisions can change your fate; similarly several unwise decisions might slow you down significantly. Thus, be considerate and wise in making important decisions.
To practice making wise decisions, start with the following practice:
(i) Baby excercise: Start with a goal, a vision or a point of view when doing a research project,
(ii) Bigger excercise: start with a goal, a vision, or a point of view for the field. This will make you stand out of the herd and guide you to valuable resources that you would otherwise not recognize.
Be calm and allow enough time for important decisions also work.
Be humble and charming but don’t be afraid of authority.
Knowing what is important and what is not to allocate resources accordingly are an important skill. That said, don’t be afraid to say no.
Strive for the highest level of scholarship & excellence in whatever you do (Ness B. Shroff)
Some wisdoms learned from Sam Altman
Trust the potential instead of the linear growth, be patient and be pleasantly surprised (like momentum compounds, it gains acceleration along the way you build it).
Self-belief (balanced with self-awareness)
Work hard. Grit. Without burn out. Grit is more important than intelligence.
Grit comes from getting back up after getting knocked down.
Don’t be afraid to work on what you really want to work on. Be bold.
Optimistic and persistent: self-doubt, giving up too soon and not pushing hard enough are enemy of success.
Think from first principles, try to generate new ideas. You fail many times to be right once.
Be able to convince people of what you believe: First, you genuinely believe in it with clear thinking - Second, use plain and concise language to communicate.
Reward is good but sometimes it is also good to take risks.
Once find a right thing to do, get unstoppable focus on accomplishing it quickly.
Build up leverage, something that is hard to be done by someone else but you.
Build network. The best way is to help people as much as you can.
Equity.
Internally driven. Work to impress yourself, not others. Work in areas that are important to you.
Some wisdoms learned from Richard Hamming in “You and Your Research”
First-class research: Do not be afraid to do first-class research.
Independent thoughts in the form of curious questions: Have independent thoughts and have the courage to pursue them. Indepedent thoughts usually come in the form of (curious) questions (Einstein asking the question about the appearance of light wave if travelling in the velocity of light when he’s 12 or 14). Again this further highlights the importance of asking questions.
Great ideas: Great ideas are more important than brains for first-class research (Bill Pfann’s case)
Courage to think hard: Great scientists have courage to think hard about their curious questions, dare to ask the impossible questions (Shannon’s case).
Compound small things into big thing: But do not get the big thing right off, start with small basics and let it compound.
Knowledge and productivity are like compound interest: Given the same ability, working a bit harder AND thinking a bit harder will lead to more productivity. More knowledge → more productive; more productive → more knowledge; and eventually the more opportunity.
Ambiguity: Great scientists tolerate ambiguity (like embrace uncertainty).
Emotional commitment: Most great scientists completely commit to their problem - keeping an eye on the flaws of their theory and improve on it.
Subconscious:
The part of the mind that is not fully aware but influence one’s actions and feelings.
Subconscious → creativity (Research result from creativity study)
Deeply immersed to a problem is a good way to make subconscious automatically work for you on your problem.
“What are the important problems in my field?”:
Important problems = important + you already have some reasonable attack (or an idea) on the problem
Thinking hard on this question and the likes can give clue about what are important problems.
Keep an eye on what the world is doing can help you sense what is important.
Keep a list of 10-20 important problems: Keep a list of 10-20 important problems and look for an attack. When a new idea comes for one problem, drop the other problems for a while and focus on it. This is what a repaired mind does, once they see an opportunity, they go after it.
Change a problem slightly:
Changing a problem slightly can help produce great work instead of good work. By this, it means that your method does not just solve a particular problem, your method might be able to solve a broader class of more general but relevant problems when you change a problem slightly. This is more satisfying and rewarding than just solving one particular problem after another after another.
Do a broad, more general, job that other people can build on top of it and see further. You do not want the next person to duplicate what you has done.
Not just the job itself, but also the way you write it, and the whole attitude you put in your writing.
Change a difficult problem into what is easier done but still important.
Learn to sell your work:
When the readers turn your pages, they must stop and read.
Write clearly and concisely.
Give formal and informal talks.
For general audience they want more survey, background, introduction, motivation (why is this important), and what was done than the details.
“Great Thoughts Time”: Devote some small time of one day in a week (e.g., Friday afternoon) ONLY to think and understand the bigger problems in the field, what’s important, what’s not.
Drive + commitment > talent
Some wisdom from Ness Shroff
How to do research (personalized section)
For me, research should be about dare to ask fundamental questions that are important for many real-world applications. Novelty is a key in research as this is how new knowledge is emerged. But how to improve the ability to ask questions and how to seek for novelty? For me, I believe that these skills can be trained and improved significantly over time (given hard working and passion). For this I treat these skills as the objective in a reinforcment learning problem where we need to balance exploration-exploitation. What is exploration and exploitation in the context of “question asking” and “novelty seeking”?
Exploitation: Spend huge time in keeping myself dive deep into technical subjects and proofs. I am currently interested in RL with function approximation, adversarial robust learning, learning in strategic environments, and non-convex optimization.
Exploration: Stay off from technical details for awhile and asking high-level questions.
These two activities should be practiced frequently to the point that they become habit. Many would emphasize too much in exploration stage whereas I think that exploitation is just as important as exploration, especially in ML theory. The art is how to balance them.
Some more concrete actions:
Spend quality time on thinking about problems and doing the hecking literature review
Prevent youself from jumping too much into technical solutions from in the early stage of a project
Stay bold in a narrow area for long enough time and become an expert in this narrow area
Recall: Strive for the highest level of scholarship & excellence in whatever you do (Ness B. Shroff)