Mapping the Maze of the UI Test Suite: lessons learned on contributing to open source without a single line of code

At the end of May, Jieyou Xu (my mentor), outlined four main tasks required for project completion.

A massive README.md to document every single directory in tests/ui/, of which there were roughly 329 at the time I was working on it. It has been completed, you can read it here.
Try to rewrite UI tests that play around with \r\n carriage returns directly in their files. Since these are invisible in most editors, it is a trap for the unwary contributor, as they could delete a single new line, then re-add it, and break the test (as it would change \r\n into \n). A PR was started to get this to work, and it almost passed CI until it hit the wall that is Windows path normalization. After looking at it more, it was determined that getting this conversion to work required such ludicrous Regex patterns to normalize the standard error files that the original test was probably worth keeping around. Nothing was merged, but the experiment was worth it.
Trying to understand what some of the more confusingly similar compiletest directives do. Initially, in this PR, I wanted to update the seemingly "bad" error-pattern into the seemingly "better" check-run-results.. After a bit of running around and trying to swap them out, only to be met with compiletest not being very happy, we realized why they had been put there in the first place - to check the very niche output of specifically runtime-panicking tests, out of reach for normal compiletest annotations such as //~?. The final decision was not to actually change any of the tests, but rather to change the documentation in another PR, as the previous wording made error-pattern sound more like a "hack meant to be replaced" than a "last resort for niche cases".

You'll notice that not a single line of code has been merged into the master branch yet. At this point, the midterm evaluation was creeping in, and I was starting to grow quite anxious. It's called "Google Summer of Code", and no "code" has been merged, therefore, failure is guaranteed, no?

It was decided that the README was quite valuable for contributors, and that the experiments were worth doing, and so, I continued my project successfully, into the realm of Sisyphus.

The fourth task mentioned by my mentor was to "rehome stray files" in tests/ui. There were almost 2000 of these, all unique in their own special way. Here is an example.

use std::ptr;
pub unsafe fn g() {
    return;
    if *ptr::null() {}; //~ ERROR unreachable
    //~| WARNING dereferencing a null pointer
}

pub fn main() {}

What are we even testing here? That the unreachable code is successfully reported? Sure, but why is there a null pointer dereference? Is it testing both at once? Why in the same file?

Then, you do some archaeology, find the issue that spawned it, and realize that, in the highly specific case where you try to read a *bool that happens to be unreachable, this used to cause an internal compiler error. And so, I rename the test unreachable-bool-read-7246. This also requires updating (blessing) the test .stderr files, as well as linking the corresponding issue.

This is a stereotypical example of an easy case, but sometimes, wrenches are thrown into the works. Some tests have their main functions in auxiliary files loading modules, some tests have very uninformative issues and require more archaeology, and sometimes, trying to change the names and files in seemingly trivial ways breaks the tests in a fashion which must then be diagnosed with the error logs.

I have been dwelling in the test mines for

multiple

hundreds

of test files. They are

and

Not to mention I have more where that came from, currently sitting on my branch repository and waiting for a time that won't clog Jieyou Xu's increasingly insane review queue. A huge word of appreciation to you, for combing through these and finding my typos and miscategorizations which occasionally slip themselves into the cracks.

There was another contender on this Sisyphean mountain: Kivooeo. They were producing these with remarkable efficiency. So much, in fact, that one of my first PRs collided with much of their work, and got closed. It was fully my fault for not checking who else was working on this, and got quite frustrated at myself.

This added component of checking for overlaps, with the potential issue of sometimes some still sneaking through, made this already tedious task even more of an ordeal. I almost wanted to quit at this point, but ended up entering communication with Kivooeo and dividing the directories in a more predictable fashion. There were no collisions after that.

There are no zero files directly under tests/ui/ at the time of writing, and there will be under 1100 files in tests/ui/issues after my currently open PRs get merged. Overall, Kivooeo and me cut down over 500 tests. Not bad. But there is much left to do. I wrote quite a few automated tools for this task, and they can still be put to good use.

However, even with them, this is tedium incarnate. The thing is, I made a grave mistake, at the start of all this.

I was initially going to apply for one of the more glamorous GSoC projects, and had even started writing my proposal. It was for the "make rustup concurrent" project, which Francisco Gouveia ended up being assigned to, and seems to be doing quite well in. Maybe I would have failed to get accepted. But, I think I should have tried. I stopped myself out of the idea that I "wasn't good enough" for an "actually difficult and architectural project" and that my place was in the test mines, just like last year.

I can't say I am devoured by regret, because this project apparently really will be useful to contributors. But, I do feel like I missed out on a valuable learning opportunity, and could have made this summer better.