A Retrospective and Review of Unit Testing

Article link: https://dzone.com/articles/why-do-programmers-fail-to-write-good-unit-tests

In this blog post I wanted to examine an article I found about unit testing. I think that since unit tests were one of the first types of testing we learned about and we have used unit tests as a foundation for many other testing methods this semester, reviewing this article serves as a great retrospective (and provides another viewpoint) on the importance of unit testing and defining criteria on writing “good” unit tests.

In the article, the author defines the significance of unit tests in creating software. They then go on to define some various qualities that make for a “good” unit test. I agree with the listed points, but I think this article could benefit by providing a few examples of good unit tests and maybe a few bad ones for comparison, especially since this is an article about programmers not correctly writing unit tests. I also think the article could better define what these “units” that are being tested are, since this term is used frequently throughout the article. Additionally, I think the article should better define what specifically is being tested in these “units” besides just saying “examine and test each unit of code separately”. Again, this is where some examples would greatly help illustrate the author’s point. The article then goes to list many benefits of writing unit tests. It ends with an interesting (and great) point of how just having unit tests isn’t necessarily beneficial (and can even be detrimental) if they are poorly written.

Overall, I think that after learning about and using unit tests a lot this semester, that this article can serve as a good introduction to the topic, and it certainly reaffirms what I have learned about unit testing, but I also think it requires some revisions to be more helpful in demonstrating the points it makes, as I find it to be somewhat vague in the fundamental parts of presenting the main concept of unit testing. I think that having a well written article about this topic is especially important as this type of testing is used as the basis for many other types of testing we have learned about. I also think that this article could include this point of how unit testing is used in the larger contexts of other types of testing such as ensuring code coverage to further prove its importance.

Regression Testing

Article link: https://dzone.com/articles/what-is-regression-testing-and-why-is-it-important

Today I learned about regression testing from an excellent article on DZone and why it is an essential part of software testing. As the article explains, regression testing involves testing the whole software product to ensure that any changes to the code doesn’t break what was already working in the product or other parts of the software. The article then goes through a great example of how fixing a bug in one part of an example piece of software can unintentionally result in another (previously working) part of the system to stop working properly. The article demonstrates really well that this breakage can occur despite unit tests showing that the bug was fixed properly. This to me is an especially important point as it shows the value in using multiple methods of testing to ensure software is performing correctly. One criticism I do have of this article is that I wish they gave an example of how to implement regression (the article does include a link to a tool that performs regression testing) testing in an actual program (or in context of their previous example).

Although after reading this article, the concept of regression testing seems simple, I find it very important as both a software user and software developer. In my personal experience as a user I have seen everything from operating systems to video games release updates that caused features that worked fine previously to become buggy or stop working entirely. From the user perspective I know how frustrating this can be, so I am glad that I learned how to prevent this problem as a software developer. Regression testing is definitely something I now want to implement into my personal software projects. Now I need to look into the tools that can help do this.

Displaying JUnit Test Reports on GitLab

In this blog post I wanted to look at something interesting that I found a while back this semester when we were learning about JUnit testing and combining this with GitLab. In class we learned how to use Gradle to build our Java programs and to run JUnit tests and get GitLab to run them when we pushed our code to GitLab using GitLab’s continuous integration feature. As GitLab’s documentation says, using Gradle and JUnit on Gitlab will show whether the tests fail on GitLab, but I wanted to take this a step further and looked into seeing if it was possible to display more information about JUnit test statuses on GitLab itself. This led me to finding this article in GitLab’s documentation about a feature in GitLab that can display JUnit test reports on merge requests. As this documentation shows this can easily be enabled on any GitLab project that already is a Java project with GitLab’s CI using Gradle to run JUnit tests. All you need to do is add the necessary lines in the GitLab CI/CD config file provided in this document to display the tests reports in a merge request.

I already tried this before a couple of months ago when I first found it, but I wanted a nice demonstration of this feature, so I created a little Java test program that has a basic Student class with a JUnit test class. I then converted this to a Gradle project using some previous programs we have used as examples and instructions provided from this class and then pushed this to a new GitLab project. On the master branch of this project the tests pass as indicated by the previous GitLab CI job. After getting the initial code pushed and passing the tests, I then created a testing branch where I changed the code in the Student class so that one of the tests (testSetLastName) would deliberately fail. Creating a merge request on GitLab for this branch and pushing this “broken” code results in the test failing when it runs on GitLab with Gradle and therefore GitLab displays on the merge request which JUnit test(s) failed:

blog-post-1-screenshot.PNG

I found this little feature to be pretty awesome in combining software testing along with software management tools and can easily see how this would be very useful for checking if new or modified code in a project causes tests to fail. In addition to checking if the tests pass, this feature allows us to easily see which tests fail, and directly on the merge request itself, instead of the alternative that the documentation says of looking through reports possibly containing thousands of lines for the failed test. I will definitely be implementing this on any projects I’m working on that use JUnit tests and are hosted on GitLab.

Link to article: https://docs.gitlab.com/ee/ci/junit_test_reports.html

Link to demo program: https://gitlab.com/cradkowski/gitlab-junit-test-reports-demo

Finalizing Diagrams and Configuring Discord

Last week started on Monday by creating a new issue for adding the LFP upstream master into the CONTRIBUTING document. I then made a new feature branch and started creating diagrams for the commits and their messages document. I also replied to Dr. Jackson’s comment on a merge request and said he can merge the CI/CD documents into master. I then started fixing the workflow diagrams according to Dr. Jackson’s review of them. I did this before continuing on the commits and their messages document since I was using the workflow commit diagrams as the base for these and if the old ones had issues so would the new ones based off these. I fixed the different diagrams and took some new screenshots for the sections that Dr. Jackson said these would work better than diagrams in. I then pushed my changes.

Tuesday started with quickly looking at the coordinating meeting minutes from earlier in the day before the weekly research meeting. In the meeting Dr. Wurst decided my top priority should be working on the SIGCSE paper and writing in the section that pertained to my research project this summer. Then after that would be Discord since we have scheduled a meeting for Friday to configure the LFP Discord server. After the meeting, I read through the SIGCSE paper and then wrote my section about what we have done this summer with researching and testing workflows and features in GitHub and GitLab. After doing that I started looking at how we should configure the LFP Discord server. I read the link Dr. Jackson posted earlier on an issue about Discord and it gave me some ideas we could use. I especially liked the idea of a welcome channel which is where the link to the server should point to to properly induct new users. There were other good ideas too including having different user roles, and limiting sending messages in certain channels. I created a new Discord server to test out a lot of these settings. I also started thinking about how to arrange our current Discord channels and maybe do this by channel type such having text channels be one group and GitLab projects (auto notification bots) be another. I also created a new issue that we discussed during our research meeting about creating a fallback plan for premium features if we weren’t able to keep our GitLab Gold tier membership and were reverted back to Free. Finally, I tested out the README install instructions for the BEAR-Necessities-Market project to see if I could get this to work since someone was having problems getting this to work. I eventually did get this work on both my computers.

Wednesday I checked the paper to see if I needed to make any edits on my section. I then started working on the commits and their messages diagrams. I updated the previously created ones to fix the same errors that the workflow ones had and created a diagram for branch synchronization commits. I added these to the document and opened a new merge request. I then worked on removing the old unneeded GitHub documentation and all GitHub references from all of the ProjectTemplate documents.

Thursday I further fixed the workflow and commits and their messages diagrams according to Dr. Jackson’s new comments. I then worked on writing a Discord configuration and tips guide. I did this by going through all of the user settings available in the Discord desktop application (and web UI) and wrote down what my recommendations were to maximize privacy and minimize annoyances that Discord can cause with its default settings (such as some of the gaming features). I also added the GitLab CI configuration file I created for BEAR-Necessities-Market in the testing group to the actual project on GitLab, so it doesn’t get lost if we delete these testing repositories.

Friday, we had the Discord configuration meeting. This went great and we quickly reached a lot of decisions. The most important topics we covered were the channel structure for the server and the different user roles. We decided that each project would have a channel group with its different channels underneath. We also decided that there would be three roles, Trustees who have all permissions, Mentors who act as moderators with some moderation permissions, and everyone else who have more limited permissions (there is also an admin role for non-trustees for need administrative permissions). We also decided there would be an announcements channel and that we would have a welcome channel to greet new members. After the meeting I started configuring the Discord server and successfully implemented the new configuration plan. Every project got its own group, an announcements channel that bots (such as Yappy) use to automatically post updates for project updates, a text channel, and a voice channel. I also made it so that all of the announcement channels could only be posted to by Trustees and Yappy. I then updated my Discord configuration guide to include the new changes to the channel structure and also added an invite link to our CONTRIBUTING document for the Discord server. Finally, I responded to some questions on Discord about where some documents and work was located in GitLab.

Saturday, I deleted all repositories and groups under the various testing groups we created this summer on GitLab and GitHub so that they weren’t lingering around past this summer. I also added DCO checks to the remaining projects that needed them and double checked that all of the projects had this enabled. Finally, I assigned myself to some issues to work on for next week and created a new issue for the broken screenshots in the various ProjectTemplate documents.

 

Creating Workflow Diagrams & Beginning the GitLab Migration

Last week started on Monday with trying to resolve why the DCO bot would not show up as a status check for a GitHub repository. I tried enabling it for the test organization instead of an individual repository and that still wasn’t working. Eventually by searching the error message that appeared in the box I found out through an article for enabling CI that in GitHub you need to manually trigger the process before you can enable it as a status check. This solved the issue and after creating a branch, committing to it, and merging it with master, the DCO bot ran for the first time and then showed up as a status check in the branch restrictions menu. I updated the pull request with a comment about this since it resolved a discussion that had been ongoing with Dr. Jackson. After that, I commented on the workflow diagrams issue to see which sections should have diagrams created for them. I looked over the contributing document and thought that Getting ready to work, Work, and Getting your work reviewed and merged all seemed like they could use some diagrams. Finally, I figured out what I would be doing for the rest of the week and planned questions for the next day’s meeting. 

Tuesday started with a research meeting that was quickly joined by Dr. Jackson over the phone. We discussed various things including the ongoing project board issue that needs to get resolved so we could update the documents with which board structure to use and how to set it up. We also talked about changing to a different license for some of the project files. I helped to resolve a Discord issue with server joining messages. Most of what I would be working on for the rest of the week was beginning the migration from GitHub to GitLab as it was at this point a pretty sure thing that we were going with this platform. I would start this by deleting the previously imported repositories and re-import them.

Wednesday I started this GitLab migration and deleted all of the previous repositories in the LibreFoodPantry GitLab group. I then imported all of the projects from GitHub to my account and transferred them to the LFP group. One issue that I discovered when importing the projects was that it didn’t link issues that were created by some users on GitHub that didn’t have a GitLab account and it made me the owner of these un-linked issues. After that, I read over the license that Dr. Jackson suggested we start using. After reading it over, I agree with Dr. Wurst that it seems a little too complicated. After getting a response from Dr. Jackson on which sections to create diagrams for, I started working on this by creating a new feature branch. I started with the Getting ready to work section. I looked at the links he suggested for how commit diagrams were designed and somewhat used some of the styling from these later when I was creating my commit diagrams. As I was creating the diagrams, I was running through the workflow myself to see that it worked properly for me. I ended up adding some git commands into the contributing document that I thought would be helpful. I also created a new issue that I discovered when testing this workflow about how to shops update their master branch when changes are made in LFP’s upstream. I suggested using GitLab’s auto repository mirroring function that takes care of this automatically. I also tested out pushing an empty commit to a new branch and creating a work-in-progress merge request back to the LFP project. I thought it was cool that when you push to a branch after doing this the commits also go to the WIP merge request so others can see your progress. By the end of the day I had created diagrams for Getting ready to work and the work sections. 

Thursday was exciting. A decision had finally been reached and we would end up using GitLab and Discord! After seeing this I posted my migration issue and also replied what was the best document to follow for importing repositories. It was decided that shop managers would re-import the projects so that they would be the owners of any issues that didn’t properly link in the import. I then went back to the diagrams and tested out how you update a feature branch from the master branch. I realized that I forgot to add the developer’s computer to the diagrams and went back to the previously created diagrams and added this. I also wanted to figure out where upstream was pointing to and realized that it was the LFP project’s master branch after some searching. I then had to figure out how to set the upstream’s master for a cloned repository of this from the shop fork, this was aided by a past exercise from CS348. I think this command should probably be added to the contributing document since it is not mentioned and not something that happens automatically. I also discovered some other issues that should be touched up on in the contributing document but I wanted to finish the diagrams before migrating to GitLab so my commits would transfer over properly. The hardest part of the diagrams was creating the commit graphs. I figured this out by looking at the commits that were on the different branches through the Web UI and after that I discovered that GitLab has a commit graph for repositories. That helped greatly with creating diagrams for the merge commits part of the workflow. I then finished the Getting your work reviewed and merged diagram and pushed that to the branch. 

On Friday there was a bunch of notifications since a lot of new issues were created on GitLab since we started migration to this platform. I started looking at some of the new ones and also replied to Dr. Wurst’s question about why commits on his projects seem to have transferred correctly. I discovered that the issue with linking accounts is more related to issues and not commits and that GitLab seems to make dummy accounts for commits if it can’t link to an actual account. I then enabled the DCO push rule for Dr. Wurst’s imported projects. I then created a pull request for my diagrams before the ProjectTemplate repository was imported to GitLab. I  also assigned myself some new issues to work on. Later that night Dr. Jackson discovered when importing the ProjectTemplate repository that issues I created on GitHub weren’t being linked to my GitLab account on. We tried to resolve this in various ways by enabling different options. I tried signing in to GitLab with my GitHub account and set the same email address on both platforms to the same one and made it publicly viewable. Sadly, this did not resolve the issue and we would try to fix it the next day.

Saturday, Dr. Jackson decided that I should import the ProjectTemplate repository myself so that it would automatically link all of my work to my GitLab account. This worked fine except for a couple of merge commits I had done with a private GitHub email address through the Web UI. We decided this was fine and I left it as-is. Finally, I enabled DCO checks for the BEAR-Necessities-Market project and updated the issue with this information.

The Workflow in Action

Last week started on Monday with getting back on track with this project. I looked at some of the issue updates on GitHub including the issue about switching to Discord. I then emailed Dr. Wurst asking about how the DCO sign-off works for committing to the project, hoping that I could push my setup documents and diagrams before the next day’s LFP committee meeting. To my surprise, the GitLab Gold issue had been fixed and the WSU account now has access to all of the Gold features again. I was really happy about this since it means I could go back to testing the advanced features offered with this package. Sadly, the GitHub issue still remains but as of Monday they hadn’t deleted my testing accounts yet. After that, I looked at the Probot question one last time and created my reply to Dr. Jackson about this. Finally, I looked at Dr. Wurst’s earlier question about free server time for open source projects. I couldn’t seem to find much information directly about if this was available, at least with Amazon Web Services or Google.

Tuesday started with a research meeting with Dr. Wurst. We covered a lot in this meeting, but the biggest thing was separating out the old issue card and breaking it into multiple new issues, as the old one was starting to get too big and congested. Dr. Wurst then closed the old issue for good. The new issues included:

By doing this it makes it easier to choose one task and finish it and see the whole progress as it moves across the project board on GitHub. 

After the meeting, I read the contributing document to see how I am supposed to be making additions to this project before committing and pushing to the repository. I then updated my local repository for the shop setup documents and re-exported all of the graph files. I then looked at the first pull request for this project and approved it and merged it into the master branch. Dr. Jackson decided that we should start using our own workflow when making additions to the LFP projects now, so I created a new branch for my setup documents, signed the previous commits with the DCO and then push the changes to GitHub. I then created a pull request and requested it to be reviewed by Dr. Jackson. 

Wednesday started with reading issue updates. I then looked to see if there was a way to always sign commits without having to add the -s parameter each time. I couldn’t find what I wanted to for this. I then started working on revising the documentation according to Dr. Jackson’s review comments. I pushed my changes to the branch. Finally, I started looking at what to do for creating workflow documentation. I discovered that Dr. Jackson had already wrote a lot that covered this already and asked on this issue card what more I can add to it. 

Thursday, I found out that the links in the setup documentation for the diagrams had broken when merged into the master branch. I fixed these image links for the setup diagrams by making them relative as Dr. Jackson suggested. I then started figuring out the answer to Dr. Jackson’s question about how labels work on GitLab issues and issue boards. With this I figured out how the different labels work in GitLab. This included group labels that allow a group level issue board to control issues with projects underneath the group. In response to the original question about how the two boards had a “doing” column I finally came to the conclusion that the original hand-off situation Dr. Jackson was asking about must have had two different column names such as “Team1 doing” and “Team2 doing” as if both boards had the same column name, moving an issue on one board automatically moves it on the other. I then updated the issue card with my response. 

Friday, I updated the platform comparison feature sheet with some of the things I’ve discovered since using the platform and closed Dr. Jackson’s comment about WIP merge requests. I then posted the link to this Google Sheet on the issue card on the community board about GitHub vs. GitLab. I then reviewed and approved some pull requests for Dr. Jackson. I then started looking at the documenting continuous integration issue and looked at how to enable CI for a GitHub project specifically using Travis as it seems to be the most popular CI marketplace app on GitHub. I actually found this really easy to do, only needing to set the language in the config file to Java for an example project I had. Finally, I reviewed another pull request for Dr. Jackson and added some suggested edits before approving, especially fixing broken document links. 

Saturday, I approved and merged some more pull requests and continued looking at how to get the DCO bot to work on a GitHub repository in order to double check the instructions Dr. Jackson posted for how to do this in our documentation. 

Sunday, I looked at if GitLab has its own version of a DCO bot. I found through different pages that using GitLab’s Push Rules for commits you can create a rule that enforces all commits to be signed according to a regular expression. I then had to figure out the correct regular expression that checks that all commits have a line matching the form:

Signed-off-by: ‘firstName lastName’ <username@domain>

After reading through a couple of tutorial websites and finding this great one that lets you test in real-time your expression against a string I finally figured out the correct expression to be:

Signed-off-by: \w+ \w+ <.+@.+\..+>

I then tested this on the GitLab web UI to make sure it works, and it did. I then tested it with locally with GitBash and also with branches. I found it works a little differently than the DCO bot on GitHub and blocks all commits from being pushed unless they have the included signature. I actually like this as it prevents the commits from even getting pushed to a project without including the sign-off. The downside is that this sign-off is needed on merge commits too. I then updated the DCO documentation to include instructions for enabling this on GitLab and added some comments on the pull request about this. Finally, I created the high-level continuous integration document, added a definition for it and an overview of how to enable it on GitHub and GitLab. I pushed this and created a pull request for this.

In retrospective after this week, I found that I enjoyed the workflow we are using with selecting an issue to work on, creating a branch, pushing the work, then asking for it to be reviewed by someone else before merging. I especially like the reviewing part as you can always have a second opinion before posting the work you have done to the master branch.