Strategies, Tactics, and Mindset Learned from "The Ph.D. Grind"
Note: This is a Paper Reading for Philip Guo's famous book "The Ph.D. Grind: A Ph.D. Student Memoir."
Main Strategies
- Be careful when choosing advisors and collaborators.
- Consider the background and incentives of the advisors and collaborators, as evidenced by recent papers, grant applications, and future aspirations.
- Think about what you want to do, what that work is like, and how that aligns with mutual interests.
- A match in research philosophy does not imply feeling comfortable working together.
- Be extremely careful working with people with a grind mindset.
I found a master's thesis advisor, and like any ambitious student, I began proposing my own partially developed research project ideas to him. My advisor patiently entertained my ideas but ultimately convinced me to focus on more conventional research topics that aligned with both his academic interests and, more importantly, the conditions of his grant funding. Since my master's program tuition was partially covered by a research grant my advisor had obtained from the U.S. government, I was obligated to work on projects within the scope of that grant. Therefore, I followed his recommendations and dedicated two and a half years to developing prototype tools for analyzing the runtime behavior of computer programs written in the C and C++ languages.
When I arrived on campus, Dawson was a recently-tenured professor who had been at Stanford for the past eight years; professors usually earn tenure (a lifetime employment guarantee) if they have published enough notable papers in their first seven years on the job. Dawson's main research interest was in building innovative tools that could automatically find bugs (errors in software code) in complex pieces of real-world software. Over the past decade, Dawson and his students built several tools that were able to find far more bugs than any of their competitors. Their research techniques were so effective that they created a successful startup company to sell software bug-finding services based on those techniques. Although I somewhat liked Dawson's projects, what appealed more to me was that his research philosophy matched my own: He was an ardent pragmatist who cared more about achieving compelling results than demonstrating theoretical "interestingness" for the sake of appearing scholarly.
During my first meeting with Dawson, he seemed vaguely interested in my broader goals of making computer usage and programming more productive. However, he made it very clear that he wanted to recruit new students to work on an automatic bug-finding tool called Klee that his grant money was currently funding. (The tool has had several names, but I will call it "Klee" for simplicity.) From talking with other professors and senior Ph.D. students in my department, I realized it was the norm for new students to join an existing grant-funded research project rather than to try creating their own original project right away. I convinced myself that automatically finding software bugs was an indirect way to make programmers more productive, so I decided to join the Klee project.
Even though none of my particular ideas managed to persuade Scott, he was still interested in collaborating with me to develop a project aligned with my broader interests. During that period, Scott held the position of an assistant professor, aiming to secure tenure at Stanford, and had been there for just three years. Consequently, he was eager to publish more papers as part of his tenure quest. As I was funded by a fellowship, Scott didn't need to allocate funds from his grants to support me, which made the collaboration appealing to him without any significant downsides.
In hindsight, I can now see why this project was likely to face challenges due to misaligned incentives, but at the time, I lacked the wisdom to anticipate such issues. I had decided to become a Klee assistant for Cristi and Dawson because I wanted to join an experienced older Ph.D. student and a professor who had a track record of publishing papers in their specific subfield. This approach had worked exceptionally well the previous year when I collaborated with Joel, an older Ph.D. student, and Scott, a professor, on their HCI project, which resulted in a top-tier, award-nominated paper.
So, what was different in this case? In short, neither Cristi nor Dawson had a strong urge to publish. They had already authored several Klee papers together, and a cross-checking paper co-authored with me would have been a "nice-to-have" but not an obligatory follow-up publication. Cristi was in the final year of his Ph.D. and didn't require further papers to graduate, while Dawson had already secured tenure and wasn't in a hurry to publish more. In contrast, Joel was a mid-stage Ph.D. student eager to publish the first paper of his dissertation, and Scott was an assistant professor who needed to publish prolifically to earn tenure. These two contrasting experiences taught me the crucial importance of thoroughly understanding the motivations and incentives of potential collaborators before embarking on a project with them.
I believe that Dawson expected Peter and me to achieve publishable results at a faster pace, which may have led him to perceive us as either incompetent or not fully committed to our work. It's a harsh reality that, as a professor at a top-tier university, Dawson's students are likely less accomplished than he was during his own Ph.D. years. The explanation is quite straightforward: Only about 1 out of every 75 Ph.D. students from a top-tier university typically possesses the qualities necessary to become a professor at an institution like Stanford (or perhaps 1 out of every 200 Ph.D. students from an average university). Predictably, neither Peter nor I met those exceptional standards. If Dawson had partnered with a younger version of himself, progress may have been different.
Two years after Peter and I departed from the Klee project, Dawson eventually found a new Ph.D. student who could successfully bring his Klee-UC vision to fruition. In 2011, Dawson and his new student published a significant paper that incorporated both Klee-UC and cross-checking ideas. Ultimately, it took three attempts involving four different Ph.D. students over five years before Dawson's original Klee-UC concept materialized into a published paper. Of those four students, only one persevered— I left the Klee project, and two others decided to exit the Ph.D. program altogether. From an individual student's standpoint, the chances of success appeared rather low.
From a professor's perspective, however, Klee-UC represented a resounding success. Since Dawson held tenure, his job was never at risk. In fact, one of the purposes of tenure is to enable professors to take risks by pursuing more ambitious project ideas. However, the downside of this privilege is that professors often assign students to work on these risky projects, which may have lower success rates. Students often find it challenging to decline such assignments, especially if they are financially supported by their advisors' grants. Fortunately, as I was funded by fellowships, it was considerably easier for me to discontinue my involvement in the Klee project.
Tom's extensive experience in publishing and reviewing numerous empirical software measurement papers made him an invaluable "insider" who understood what types of results and paper writing were well-received by reviewers in that specific subfield. When it came time to submit our paper at the end of that summer, Tom skillfully positioned our contributions within the context of related work, presented compelling arguments for the novelty and significance of our findings, and meticulously polished our paper. Three months later, I was thrilled to receive the news that our paper, which focused on studying the causes of bug fixes, had been accepted at a top-tier conference. This was particularly impressive given that only 14 percent of all papers submitted that year were accepted.
However, Tom's dedication didn't stop there. As a newly-hired researcher at MSR, he was motivated to build his reputation by publishing additional papers. In the following years, we leveraged the results from my summer 2009 internship to write two more top-tier conference papers. One of these papers explored bug report reassignments, while the other delved into bug report reopenings and even earned a Best Paper Award.
During my first month in the new phase of my academic journey, I primarily spent my time reconnecting with old college friends, as my alma mater, MIT, was conveniently located near Harvard. Additionally, I had several meetings with Margo to explore potential research ideas. Margo was open to the idea of me pursuing my own project under her loose supervision, granting me a considerable degree of intellectual freedom. However, I approached my brainstorming process pragmatically because I aimed to generate a project that would genuinely excite her and secure her strong support for its inclusion in my dissertation. To achieve this, I delved into her recent papers and grant applications to gain insight into her research philosophy. I tailored my ideas to align with her preferences, recognizing the importance of harmonizing with the subjective inclinations of senior collaborators, as well as the expectations of paper reviewers, even within fields that are considered technically objective.
At that time, reading a grant proposal was an entirely novel experience for me, and it appeared foreign and unfamiliar. However, with time and practice, I have since become accustomed to writing grants, and it has become a routine part of my academic life.
- Apply to fellowships before starting the Ph.D. for better academic freedom and advisor-advisee relationship.
I was also fortunate to receive two prestigious fellowships, the NSF and NDSEG graduate research fellowships. These fellowships were granted to only about five percent of all applicants. They covered the full expenses for five out of the six years of my Ph.D. studies and relieved me from the obligations of working on specific grant-funded projects. This was a significant advantage over students who had to work on such projects throughout their college years.
Applying to Ph.D. programs and fellowships during my master's year gave me a huge advantage over students who applied during senior year of college, since I had an extra year of research experience.
In other fields, such as the humanities and social sciences, students typically do not receive direct funding from their advisors. This distinction significantly changes the dynamics of the advisor-advisee relationship, turning the Ph.D. experience into more of a solitary journey and less of an employer-employee arrangement.
However, I soon came to the realization that I wasn't obligated to remain tethered to Klee in any way, given that my funding came from the NDSEG fellowship rather than Dawson's grants. In contrast, all of Dawson's other students had no option but to persist with their work on Klee, as they were supported by his Klee-related grants. Therefore, I retained Dawson as my advisor but departed from the Klee project, embarking on the journey to create my own research project entirely from scratch.
From a professor's perspective, however, Klee-UC represented a resounding success. Since Dawson held tenure, his job was never at risk. In fact, one of the purposes of tenure is to enable professors to take risks by pursuing more ambitious project ideas. However, the downside of this privilege is that professors often assign students to work on these risky projects, which may have lower success rates. Students often find it challenging to decline such assignments, especially if they are financially supported by their advisors' grants. Fortunately, as I was funded by fellowships, it was considerably easier for me to discontinue my involvement in the Klee project.
- Question each step in the decision-making process.
- Have an outline or draft of the research paper, especially the evaluation section, before starting a research project.
- This brings the obvious benefit of making research contributions.
- It also forces everyone not to push you around, helping you to jump out of the "pecking order" slowly and surely.
Dawson believed that Klee could uncover new bugs that no automated tool or human being had previously discovered within the code of thousands of Linux device drivers. I recall thinking that while finding new Linux device driver bugs could be interesting to present in a paper, it wasn't entirely clear to me how these results constituted a substantial research contribution. To my understanding, my role was to use Klee to uncover new bugs, essentially applying existing research, rather than significantly enhancing Klee in an innovative manner. Moreover, I couldn't envision how my project would seamlessly integrate with the projects of the other five students for a coherent paper submission in March. Nevertheless, I had faith in Dawson's high-level paper writing strategy. Since I had recently joined the project, I didn't want to immediately question these decisions typically made by professors. I was assigned a specific task, and I was determined to carry it out to the best of my abilities.
My rational understanding acknowledged that experimental research in science and engineering fields often demands an extensive amount of unglamorous and labor-intensive work to produce tangible results. Ph.D. students, particularly those in their first and second years, are typically the ones tasked with undertaking the most tedious tasks—this is essentially what we are compensated for. In a typical research group, the professor and senior Ph.D. students formulate the high-level project plans and then delegate the responsibility of making all the intricate details function in practice to the junior students. First- and second-year students usually have minimal influence on the overall direction of the group's project. Although I fully embraced my position as the lowest-ranking member of the team, my emotional state still suffered significantly during those initial months because the work was exceptionally challenging and lacked immediate rewards.
I met with Dawson to express my frustration regarding the overwhelming task I was currently tackling. It felt ludicrous to spend several days configuring Klee for each new device driver. Not only was it physically exhausting, but it also didn't seem like genuine research. What could we possibly write in our paper? That I had devoted nearly 1,000 hours to manual labor in getting Klee to function with device drivers without gaining any meaningful insights? It didn't feel like a valuable research contribution; it seemed rather futile Additionally, panic set in as there were only five weeks left until the paper submission deadline, and Dawson had yet to discuss our group's paper writing strategy. Typically, writing a respectable paper submission takes a minimum of four weeks, especially when coordinating efforts among six students involved in the project.
However, a significant problem arose. When we finally achieved those favorable results, there were only three days left until the paper submission deadline, and not a single word of the paper had been written yet. In such a short timeframe, it was physically impossible to write, edit, and refine a paper submission that had any chance of being accepted at a top-tier computer science conference. Nevertheless, we decided to give it our best shot.
During the final 72 hours leading up to the deadline, Dawson and five of us students (one had dropped out of the project by this point) practically lived in the office, pulling two consecutive all-nighters to wrap up the experiments and draft the paper. Deep down, all of us students realized that there was virtually no chance that this paper would be accepted, but we followed Dawson's lead and pressed on.
The result was a submission that can only be described as a disorganized mess – it contained numerous typos, nonsensical sentence fragments, graphics lacking explanations, and lacked concluding paragraphs. It was a dismal sight. At that moment, I couldn't fathom how I would ever complete a Ph.D. if it meant working in such a chaotic and haphazard manner. As anticipated, three months later, the reviews for our paper were overwhelmingly negative, filled with harsh comments like, "The program committee believes that this paper is far too sloppily prepared to warrant acceptance; please refrain from submitting papers that are clearly unready for review."
My friend Greg, who was one of Rob's Ph.D. students, emphasized the significance of the third point: thinking about experiments when suggesting research project ideas. Professors are often driven by the desire to have their names associated with published papers, and in the field of computer science, conference papers typically require robust experiments to secure acceptance for publication. Therefore, it's essential to consider experiment design right from the outset when formulating project proposals.
- Target fellow researchers with similar incentives when conducting HCI studies in academia.
- Without exceptionally strong resources, find a novel, meaningful niche for research instead of an overcrowded and highly competitive domain.
- Do not attempt to do in academia what should be done in industry.
In hindsight, I'm not astonished that my efforts to shadow professionals in their workplaces were unsuccessful. I had nothing to contribute to these seasoned programmers; my presence would have likely disrupted their workday. Fortunately, a few years later, I had the opportunity to observe a different group of programmers—fellow graduate students engaged in programming for scientific research. They were open to my occasional inquiries and more than willing to discuss their working environments. These interviews would ultimately serve as a direct source of inspiration for my dissertation work.
Dawson and I encountered significant challenges in getting our research results published. Over the course of a year, we submitted two papers that were both rejected. It would take another full year before our work was finally published as a shorter-length paper in a second-tier conference, which held minimal prestige and did not count as a contribution to my dissertation. However, by that point, I had already moved on to other projects.
The primary reason behind our struggles with publication was that we were not considered "insiders" in the empirical software measurement subfield (sometimes referred to as empirical software engineering), to which our project belonged. When Dawson and I embarked on this work, numerous research teams from various universities and corporate research labs were already engaged in similar endeavors. We were clearly outmatched by the competition, which included professors and research scientists specializing in empirical software measurement, guiding armies of Ph.D. students through the extensive data analysis. These individuals were eager to publish a multitude of papers, especially young professors aspiring to attain tenure. They possessed expertise in statistical methodologies, framing related work, and crafting persuasive narratives needed to secure acceptance for such papers. Most significantly, they frequently served on program committees and acted as external reviewers for relevant conferences, which provided them with in-depth knowledge of the requisites for producing publishable papers in this subfield.
One significant advantage of being an intern at MSR was access to a wealth of internal data sets containing information about Microsoft's software bugs and personnel files. These confidential data sets would have been inaccessible to me as an external researcher. The richness of these Microsoft data sets provided MSR researchers like Tom with a distinct advantage, making it easier to obtain groundbreaking and publishable results compared to competitors who lacked access to such data.
In contrast, when I worked with Dawson, the Linux data sets I had access to were smaller and of lower quality. Open-source software projects typically do not maintain records as meticulously as one of the world's largest software companies. This limitation is something that all university researchers face unless they establish partnerships with companies that can provide them with access to relevant data.
Upon returning to Stanford in the fall of 2009, inspired by my previous HCI work with Scott and Joel during my second year, I embarked on a project to interview colleagues who used Python for data analysis in their research. The objective was to identify the programming-related inefficiencies they faced and explore how IncPy could address and eliminate these inefficiencies. I also leveraged my connections to give presentations about IncPy, even though it was still a half-baked idea at that stage, at various lab group meetings. These early efforts helped generate fresh ideas and refine the project's "marketing pitch." I'm deeply thankful for the friends who supported me in kickstarting my project when I had little more than a few rudimentary PowerPoint slides.
As I continued my interviews and refined my design plans, I grew increasingly optimistic. I discovered that researchers in various computation-based fields, including machine learning, pharmacology, bioengineering, bioinformatics, neuroscience, and ocean engineering, all faced similar challenges in their data analysis workflows, making them potential beneficiaries of IncPy. After a few weeks of interviews and subsequent adjustments to my project's direction, I felt confident that I could convincingly pitch the idea in a future paper submission. The core argument I aimed to convey was that many computational researchers across diverse fields grappled with common inefficiencies in their daily programming tasks, and IncPy presented a novel, fully automated solution to these inefficiencies that had not been previously implemented. This initial pitch would ultimately become the central theme of my entire dissertation.
- Climb the shoulders of giants as much as possible and pick the low-hanging fruit from there before you become a giant.
Following the creation of Klee and related projects between 2005 and 2008, a new subfield emerged. This development led to numerous assistant professors and young research scientists eagerly producing a plethora of papers, each detailing incremental improvements in their quest to secure tenure or job promotions. It was akin to an academic gold rush, spurred by the early insights of Cristi, Dawson, and a select few pioneers. Since Dawson already possessed tenure and had gained fame for his contributions, he was above the fray and lacked the desire to publish solely for the purpose of bolstering his academic resume.
In practice, Ph.D. students collaborating with these young researchers found it comparatively easier to publish their work and complete their graduate programs, while Dawson's students faced considerably more challenges. Over the three years since I departed from the Klee project, research groups worldwide have collectively published hundreds of papers grounded in Klee-like concepts. Remarkably, fifteen of these papers detailed enhancements to Klee itself, as our laboratory released it as open-source software to encourage further research. In the meantime, five of Dawson's Ph.D. students have made serious efforts to work on Klee; however, only one has managed to publish a single paper on Klee-UC.
- Actively expand your network and seek collaboration opportunities.
Just before commencing my second year of the Ph.D. program in September 2007, I took a one-week vacation to Boston to visit friends from college. While in the area, I reached out to a few MIT professors I knew from my undergraduate years, seeking their guidance. During our meetings, they all conveyed a similar message: Take the initiative to engage with professors, explore research topics of mutual interest, and above all, avoid isolation. This straightforward advice, consistently applied over the next five years, ultimately paved the way for a successful completion of my Ph.D. journey.
I wasted no time in taking this advice to heart while still in Boston. I sent a cold email to an MIT computer science professor named Rob, politely requesting a meeting with him. In this initial email, I briefly introduced myself as a recent MIT graduate and a current Stanford Ph.D. student with a keen interest in developing tools to enhance the productivity of computer programmers. Given that I knew Rob shared an interest in this research area, I hoped my email would pique his interest rather than end up in his spam folder. Fortunately, Rob generously agreed to meet with me for an hour in his office, during which I presented a few project proposals and sought his feedback. He appeared to find merit in my ideas, which bolstered my confidence that they held promise in the eyes of a professor working in this research domain. Regrettably, I couldn't collaborate with Rob as I was no longer an MIT student. Nonetheless, at the conclusion of our meeting, Rob suggested that I approach a Stanford computer science professor named Scott to see if I could garner his interest in my ideas.
The lasting impact of an MSR (Microsoft Research) internship often extends beyond research achievements to the friendships forged during the experience. During that particular summer, I had the privilege of forming connections with some of the brightest and most inspiring young computer science researchers of my generation. For example, one of my three office mates was on the verge of beginning her Ph.D. journey at MIT and had already published more top-tier papers during her undergraduate research than most Ph.D. students could ever aspire to. Another office mate was a UC Berkeley Ph.D. student who dedicated his nights and weekends to a separate research project with collaborators from across the country, all while diligently working on his internship project during workdays. These peers are likely to evolve into award-winning professors, research leaders, and high-tech entrepreneurs, and I am genuinely humbled to have had the opportunity to share a summer with them.
- This is how we can evaluate productivity claims.
From the very beginning of my IncPy project, I recognized the challenge of presenting a compelling evaluation. The core premise, that IncPy could enhance the productivity of computational researchers, was inherently subjective. To address this, after studying similar papers, I developed a two-pronged evaluation approach:
Case Studies: I planned to gather a variety of Python programs from computational researchers and simulate the productivity gains they might have achieved using IncPy instead of standard Python.
Deployment: The goal was to encourage researchers to incorporate IncPy into their regular work, allowing them to directly experience and report on its impact on their productivity.
In pursuit of this, I adopted the roles of both salesman and beggar, persistently seeking Python programs from colleagues for my case studies and encouraging them to use IncPy in their research. Despite mostly receiving negative responses, I continued asking for referrals and volunteered to speak at various lab meetings to generate interest in IncPy. After months of effort, I managed to acquire Python programs from researchers across various fields, sufficient for starting my case studies.
As I concluded my fourth Ph.D. year in September 2010, I submitted my IncPy paper to a top-tier conference. The paper included case studies and a few deployment anecdotes. Aware of the low acceptance rate and the unconventional nature of IncPy within academic fields, I was prepared for potential rejection but still aimed high, knowing the value of a top-tier publication for my graduation prospects.
- The runtime is a vital (yet often overlooked) aspect to consider in programming language research.
On July 29, 2010, a year after the initial concept of IncPy was born, I was struck by another idea, this time addressing a common issue in computational research. I noticed that researchers often write their computer programs in an ad-hoc, somewhat careless manner, leading to frequent crashes for trivial reasons. These crashes not only prevent the production of any results but also cause considerable frustration.
My realization was that by modifying the runtime environment of the Python programming language, specifically the interpreter, I could address this issue. The idea was to adapt the Python interpreter in a way that would allow these less rigorously written programs to still run and produce partial results, rather than failing completely and producing none. I decided to name this modified version of the Python interpreter "SlopPy," a playful blend of 'Sloppy' and 'Python', emphasizing its tolerance for less meticulous coding practices. This concept aimed to make the process of data analysis more forgiving and efficient for researchers who may not always adhere to stringent coding standards.
- Focus on acknowledging shortcomings and deriving their insights if experimental evaluation results are suboptimal.
While I was interning at Google during the summer of 2011, I received the joyful news that our ProWrangler paper had been accepted with outstanding reviews. The main factor contributing to our success was Jeff's exceptional work in crafting both the introduction of our paper and the interpretation of our evaluation results. Initially, our user testing had not demonstrated the productivity improvements we had hoped for, which made me concerned that our paper might face rejection. However, Jeff's skill in technical writing and framing our arguments skillfully transformed what seemed like impending failure into a surprising victory. The reviewers appreciated our candid acknowledgment of the shortcomings in our evaluation and the valuable insights we extracted from them. Undoubtedly, our paper would not have gained acceptance without Jeff's rhetorical expertise. He had accumulated substantial experience in this area, having published 19 papers during his Ph.D. studies, mostly in top-tier conferences, which is five to ten times more than what is typically expected from computer science Ph.D. students. This level of dedication and productivity is often necessary to secure a faculty position at a prestigious university like Stanford.
- In fiercely competitive domains, do not self-consciously try to network and seek opportunities, and never schmooze. Instead, rely on bypasses and side roads. Many things grow in the garden that were never sown there（有心栽花花不开，无心插柳柳成荫）.
When I made the decision to leave academia, one of the immediate impacts was that I no longer felt the need to engage in networking activities at the three academic conferences I attended that summer. These conferences included talks I gave on IncPy, SlopPy, and CDE. Academic conferences are typically filled with senior Ph.D. students, postdocs, and pre-tenure professors who are actively networking to impress their more senior colleagues. For these junior researchers, professional networking at conferences is a crucial and time-consuming task as it greatly influences their budding careers and academic reputations. However, since I had decided to step away from the academic world, I found myself enjoying the conferences without the usual nervousness or strategic calculations.
At one of these conferences, I had a casual conversation with John, a professor from the University of Utah who was the keynote speaker. Later on, he wrote a generous blog post that significantly increased the popularity of "The Ph.D. Grind" among professors. He also provided me with practical advice regarding the possibility of returning to academia myself. This unexpected turn of events occurred because I was no longer driven by a desire to stay in academia and simply chatted with people I found interesting at the conference without a specific networking agenda. It was a reminder of how unpredictable life can be.
During a break between sessions at another conference, I noticed Margo sitting alone and working on her laptop. I had previously met Margo during my fourth year at a San Jose workshop where I presented my original IncPy paper. Although I had some reservations about approaching her and reintroducing myself, fearing that she might not remember me or that our conversation might lack substance, my impending departure from academia meant I had no real networking agenda to uphold. I decided to take the chance and greeted her. I reminded her of our previous encounter, and she appeared to recall me. We had a brief five-minute conversation about my new CDE project before I had to rush off to give my talk. After returning home, I sent her a courteous follow-up email with a link to the CDE project webpage, in case her students were interested in using it for their research. This was my standard polite approach when introducing CDE to professional colleagues, and I didn't have high expectations for follow-up.
As it turned out, Margo would later play a pivotal role in helping me secure a professorship. Ironically, if I had initially desired a professorship, I might have been too self-conscious and hesitant to approach her in the first place, which could have diminished my chances of eventually obtaining the position. In the end, it was because I didn't actively seek a professorship at the time that I ultimately ended up getting one. Life has its own unique way of working things out!
The culmination of my graduate school journey wouldn't have been possible if I hadn't actively seized the opportunities that I was fortunate enough to receive. If Robert hadn't informed me about the San Jose workshop two years ago, if I hadn't submitted and presented my IncPy paper there, if Margo hadn't taken an interest in my paper and introduced me to Elaine, if I hadn't maintained contact with Elaine, if I hadn't spontaneously approached Margo again at last summer's conference where I presented CDE, if she hadn't sent me a gracious follow-up email, and if I hadn't taken a risk with my unconventional counterproposal to her, then I would have still been at Stanford struggling to find one last project and thesis committee member.
It's important to acknowledge that achieving this outcome required me to try many different approaches like this, and most of them did not yield the desired results. Success often involves numerous attempts and failures before finding the right path.
Supportive Tactics
- Optimize the process of note-taking for research.
My daily routine primarily revolved around the development of computer programs designed to extract, clean, reformat, and analyze data from the Linux revision control history and the 2,000 bug reports. In my pursuit of gaining insights, I independently acquired a foundational understanding of quantitative data analysis, statistics, and data visualization techniques. Throughout my work, I meticulously recorded my experimental progress in a research lab notebook, carefully documenting which trials were successful and which were not.
Every week or so, I would meet with Dawson to present my findings. Typically, these meetings involved me presenting him with printouts of graphs or data tables generated through my analyses, followed by him offering high-level suggestions, such as, "This part of the graph appears unusual; can you explain why? Try breaking down the data in this manner and delve deeper." It was only years later that I discovered this working style was relatively common among computational researchers in various academic disciplines. For my dissertation, I went on to develop tools aimed at streamlining the typical inefficiencies in this prevalent workflow. However, at that time, I had no such long-term vision; my primary goal was to make intriguing discoveries and have them published.
- Raise encountered problems and frustrations, no matter how minor, at meetings and complain.
- Actively contact people for feedback, motivation, and emotional support.
I found myself navigating unfamiliar territory, making it significantly more challenging to seek assistance compared to my undergraduate years when solutions were more straightforward. As the sole person working with Klee on device driver code, my colleagues couldn't offer much guidance. While Dawson occasionally provided high-level strategic advice, as is customary for tenured professors, his role didn't involve being directly involved in the day-to-day challenges we faced. It fell upon us, the students, to decipher all the intricate details necessary to yield results—my task being to uncover new bugs in Linux device drivers that had not been previously identified. Professors often reiterate the mantra, "If it's already been done before, then it wouldn't be research!" For the first time, I truly grasped the essence of those words.
For the following ten weeks, I found myself daydreaming about my own research concepts in complete isolation, without engaging in any conversations with others. Given my negative initial experience working in a research group over the past few months, I craved solitude to think independently. Dawson was supportive of my absence since he wasn't financially supporting me through his grants.
I lived in complete seclusion, mentally drained but still attempting to make gradual progress. Each day, I dedicated myself to reading numerous computer science research papers and taking notes in the hope of finding inspiration for my own creative ideas. However, lacking proper guidance or context, I often ended up squandering a lot of time without gaining meaningful insights from my readings. I also roamed aimlessly on my bicycle through the neighborhoods around campus, hoping to spark new research ideas to no avail. Most notably, I procrastinated more than I ever had in my life up to that point: I watched countless TV shows, took numerous naps, and wasted countless hours online. Unlike my friends with conventional nine-to-five jobs, there was no supervisor monitoring my daily activities, so I allowed my mind to wander without any structure in my life.
During those ten solitary weeks, I scarcely spoke to anyone, not even my friends or family. Complaining seemed futile because it felt like nobody could truly grasp what I was going through at the time. My friends who were not pursuing Ph.D. programs believed I was simply "in school" and taking classes like a typical student. Meanwhile, the few friends I had made in my department were grappling with their own first-year Ph.D. struggles, primarily the shock of diving headfirst into complex, open-ended research problems without the ability to influence the overarching direction of their assigned projects. We, as young computer scientists, willingly engaged in tasks that were both exceptionally challenging and seemingly devoid of purpose, all while earning a quarter of the salary of our friends in the corporate world. It was so disheartening that it became almost comically tragic. However, I didn't believe that group complaining would be productive, so I chose to remain silent. I avoided the Computer Science Department building, fearing encounters with colleagues who might inevitably inquire about my work, and I had no respectable response to offer. Instead, I preferred secluding myself in libraries and coffee shops.
In hindsight, going solo so early in my graduate school journey was a regrettable decision. Contrary to romanticized notions of a solitary scholar sitting outdoors, sipping a latte, and scribbling on blank notebook pages, real research is never conducted in isolation. It necessitates a solid foundation in intellectual, historical, and sometimes even physical aspects (such as laboratory equipment) to develop innovative ideas. A wiser approach during those weeks would have been to communicate with Dawson more frequently and actively seek collaborations with other professors or senior students. However, at that time, I was so burnt out and frustrated with the traditional hierarchy of group-based research – which often involved new Ph.D. students performing the most unglamorous tasks – that I retreated and embarked on my own path.
By the middle of my third year, many of my fellow students and I found ourselves in a state of "limbo." It became increasingly challenging to muster the motivation to come into the office day in and day out. We also grappled with feelings of isolation and loneliness, as we spent our days and nights immersed in tackling obscure, highly specialized problems that few people in our immediate surroundings comprehended or showed interest in. While our advisors and senior colleagues occasionally offered high-level guidance, they seldom sat down with us to work through the intricate details of our research endeavors.
- If there is work-life balance in a job and stress levels are not too high, i.e., one is not drained or overburned, would it be a viable opportunity to pursue self-improvement and open doors for future pursuits?
Of course, it would be foolish to pursue a Ph.D. solely out of irrational childhood fears. To get a preview of corporate working life, I did internships at engineering companies every summer during college. Since I happened to work in offices where I was the only intern, I was given the full responsibilities of a junior engineer, which was a rare privilege. Although I learned a lot of technical skills, I found the day-to-day work to be mind-numbingly dull. My coworkers were also unenthusiastic about their jobs, and there were few appealing prospects for career advancement. Of course, I'm not claiming that all engineering jobs are mind-numbingly dull; it just happened that the companies I worked for were not first-rate. Many of my college friends who interned at first-rate companies such as Microsoft had great experiences. Ironically, my first full-time job after finishing my Ph.D. was at Google. Google loved their experiences and signed on to work at those companies full-time after graduation.
- Research software tends to be rough prototypes. Keep them simple, stupid.
Similar to other sophisticated software tools, Klee featured numerous configurable options. However, since it was a research prototype assembled by students, most of these options lacked clear documentation regarding their behaviors. Consequently, I spent a significant amount of time grappling with misunderstandings about the subtle interactions between these options as I adjusted them. My research lab notebook became filled with frustrations, including entries like: "OH SH*T, I believe my mistake was failing to realize that there are specific options meant to be passed into Klee (e.g., -emit-all-errors), and others that should be passed into the target program to set up the model environment (e.g., --sym-args). When these get confused, bizarre outcomes occur because Klee ends up executing the target program with argc and argv values that differ from what you'd expect."
From a research perspective, my goal was achieved: I successfully developed an initial prototype of CDE and demonstrated its functionality in a practical use case. In many applied engineering fields, it's widely accepted that research prototypes like CDE primarily serve to prove the feasibility of innovative concepts. The role of a researcher involves creating prototypes, conducting experimental assessments to measure their effectiveness, publishing research papers, and then moving on to the next idea. It would be unrealistic for a researcher to expect people to use their prototypes as if they were fully-fledged products. Instead, if the ideas are promising, professional engineers may incorporate them into their company's future products. At best, some other research groups might utilize your prototypes as a foundation to develop their own prototypes and subsequently reference your work in their papers (for example, more than a dozen university research groups have expanded upon the Klee tool and published papers detailing their enhancements). However, it is exceedingly rare for individuals outside of the research community to employ research prototypes in their daily tasks. In essence, the objective of academic research is to generate validated ideas, not refined products.
Make full use of workshops to disseminate preliminary results and collect feedback rapidly, even if it involves spending money yourself.
Typically, I wouldn't have paid much attention to such an announcement for two reasons. Firstly, Robert's research area, data provenance, had no direct relevance to IncPy, so his paper submissions didn't affect me. Secondly, in our department, workshop papers don't usually count as substantial contributions toward a dissertation. Workshops are primarily meant for sharing early-stage ideas and tend to have much higher acceptance rates (60 to 80 percent) compared to conferences (8 to 30 percent). Many professors prefer their students to focus on publishing in conferences since workshops require them to cover travel, hotel, and registration costs, similar to conferences, but without the same level of prestige. Consequently, top-tier computer science professors often encourage their students to prioritize conference papers over workshop submissions.
While presenting my IncPy workshop paper was beneficial for feedback and networking, particularly with Margo, it didn't qualify as an official publication for my dissertation. I was aware that publishing this work at a recognized conference in my department was necessary. The main difference between a workshop and a conference paper lies in the requirement for a conference paper to have a robust experimental evaluation, demonstrating the effectiveness of the proposed tool or technique. The evaluation part of a paper can vary, from measuring runtime performance to conducting user behavior studies in a controlled environment. Given the commonality of similar research ideas, reviewers pay close attention to the implementation and experimental analysis of these ideas when deciding on the acceptance or rejection of papers.
Mindset
- Find a meaning for your research and build a reputation.
After two months of persistent effort, I began to achieve some modest victories. I managed to get Klee to function well enough to identify my first few bugs in the smallest device drivers. To ascertain whether these bugs were genuine (as opposed to false positives resulting from Klee's limitations), I sent emails outlining each potential bug to the Linux developers responsible for those drivers. Several driver creators verified that I had indeed discovered real bugs in their code. These email confirmations brought me great excitement, as they represented my initial glimpses of external validation.
The skill of crafting concise and impactful professional emails has proven to be highly beneficial for my career.
- Do not bootlick the "establishment" or the "mainstream." Be wise and brave, identify emergent trends, conquer virgin land, and blaze unheard-of new paths.
The second and more significant reason why pursuing a postdoc didn't make sense for me was the nature of the research topics I was truly passionate about. These topics didn't align well with the likelihood of winning grant funding because they weren't widely accepted by the current academic establishment. Without grants, it would be impossible to fund students to work on these topics, and without motivated students to tackle the challenging manual work involved, it would be difficult to produce reputable publications. Furthermore, without a substantial number of publications each year, achieving tenure would be out of reach. Even if I did manage to secure tenure, I would still require new grants to support new students in implementing my ideas, perpetuating an ongoing funding cycle. Given my research interests, I wasn't emotionally prepared to engage in the uphill battles necessary to have my proposals taken seriously by grant funding agencies. Convincing peer reviewers to accept my papers had already been a challenge, and grant reviewers would likely be even less sympathetic, as they control the distribution of significant financial resources and would prefer to allocate funding to colleagues conducting more mainstream computer science research.
When I began my faculty career in 2014, this was my greatest fear. Consequently, I deliberately transitioned to a new research area that had greater potential for securing funding.
Out of the 26 Ph.D. graduates in the Stanford Computer Science Department from my year, I considered myself relatively average from an academic perspective, as most of my papers were second-tier and not well-received by the academic community. My dissertation work straddled multiple computer science subfields, including Programming Languages, Human-Computer Interaction, and Operating Systems, which made it challenging to gain recognition from top experts in any one specific subfield.
Given that my dissertation topic was far from mainstream, any junior professor or scientist attempting to build their academic career around its concepts would face difficulties gaining the approval of grant funding agencies, crucial for launching new projects, and the support of senior colleagues, essential for publication and tenure. While I am more than willing to support anyone willing to take on this commendable challenge, I wasn't courageous enough to risk my own career on it. Instead, I have chosen to pursue an entirely different professional passion, which may become the subject of a future book.
Interestingly, these ideas eventually became more fundable in the year after I graduated, thanks to the emergence of the Big Data and data science movements. However, by that time, I had already shifted my focus to other interests.