Software Quality Assurance & Testing

This fall semester I am taking a course on Software Quality Assurance & Testing and I will be using this blog to write about new topics I explore in this area as part of this class. I have been looking forward to taking this class all summer and learning more about software development!

Advertisements

Finalizing Diagrams and Configuring Discord

Last week started on Monday by creating a new issue for adding the LFP upstream master into the CONTRIBUTING document. I then made a new feature branch and started creating diagrams for the commits and their messages document. I also replied to Dr. Jackson’s comment on a merge request and said he can merge the CI/CD documents into master. I then started fixing the workflow diagrams according to Dr. Jackson’s review of them. I did this before continuing on the commits and their messages document since I was using the workflow commit diagrams as the base for these and if the old ones had issues so would the new ones based off these. I fixed the different diagrams and took some new screenshots for the sections that Dr. Jackson said these would work better than diagrams in. I then pushed my changes.

Tuesday started with quickly looking at the coordinating meeting minutes from earlier in the day before the weekly research meeting. In the meeting Dr. Wurst decided my top priority should be working on the SIGCSE paper and writing in the section that pertained to my research project this summer. Then after that would be Discord since we have scheduled a meeting for Friday to configure the LFP Discord server. After the meeting, I read through the SIGCSE paper and then wrote my section about what we have done this summer with researching and testing workflows and features in GitHub and GitLab. After doing that I started looking at how we should configure the LFP Discord server. I read the link Dr. Jackson posted earlier on an issue about Discord and it gave me some ideas we could use. I especially liked the idea of a welcome channel which is where the link to the server should point to to properly induct new users. There were other good ideas too including having different user roles, and limiting sending messages in certain channels. I created a new Discord server to test out a lot of these settings. I also started thinking about how to arrange our current Discord channels and maybe do this by channel type such having text channels be one group and GitLab projects (auto notification bots) be another. I also created a new issue that we discussed during our research meeting about creating a fallback plan for premium features if we weren’t able to keep our GitLab Gold tier membership and were reverted back to Free. Finally, I tested out the README install instructions for the BEAR-Necessities-Market project to see if I could get this to work since someone was having problems getting this to work. I eventually did get this work on both my computers.

Wednesday I checked the paper to see if I needed to make any edits on my section. I then started working on the commits and their messages diagrams. I updated the previously created ones to fix the same errors that the workflow ones had and created a diagram for branch synchronization commits. I added these to the document and opened a new merge request. I then worked on removing the old unneeded GitHub documentation and all GitHub references from all of the ProjectTemplate documents.

Thursday I further fixed the workflow and commits and their messages diagrams according to Dr. Jackson’s new comments. I then worked on writing a Discord configuration and tips guide. I did this by going through all of the user settings available in the Discord desktop application (and web UI) and wrote down what my recommendations were to maximize privacy and minimize annoyances that Discord can cause with its default settings (such as some of the gaming features). I also added the GitLab CI configuration file I created for BEAR-Necessities-Market in the testing group to the actual project on GitLab, so it doesn’t get lost if we delete these testing repositories.

Friday, we had the Discord configuration meeting. This went great and we quickly reached a lot of decisions. The most important topics we covered were the channel structure for the server and the different user roles. We decided that each project would have a channel group with its different channels underneath. We also decided that there would be three roles, Trustees who have all permissions, Mentors who act as moderators with some moderation permissions, and everyone else who have more limited permissions (there is also an admin role for non-trustees for need administrative permissions). We also decided there would be an announcements channel and that we would have a welcome channel to greet new members. After the meeting I started configuring the Discord server and successfully implemented the new configuration plan. Every project got its own group, an announcements channel that bots (such as Yappy) use to automatically post updates for project updates, a text channel, and a voice channel. I also made it so that all of the announcement channels could only be posted to by Trustees and Yappy. I then updated my Discord configuration guide to include the new changes to the channel structure and also added an invite link to our CONTRIBUTING document for the Discord server. Finally, I responded to some questions on Discord about where some documents and work was located in GitLab.

Saturday, I deleted all repositories and groups under the various testing groups we created this summer on GitLab and GitHub so that they weren’t lingering around past this summer. I also added DCO checks to the remaining projects that needed them and double checked that all of the projects had this enabled. Finally, I assigned myself to some issues to work on for next week and created a new issue for the broken screenshots in the various ProjectTemplate documents.

 

Creating Workflow Diagrams & Beginning the GitLab Migration

Last week started on Monday with trying to resolve why the DCO bot would not show up as a status check for a GitHub repository. I tried enabling it for the test organization instead of an individual repository and that still wasn’t working. Eventually by searching the error message that appeared in the box I found out through an article for enabling CI that in GitHub you need to manually trigger the process before you can enable it as a status check. This solved the issue and after creating a branch, committing to it, and merging it with master, the DCO bot ran for the first time and then showed up as a status check in the branch restrictions menu. I updated the pull request with a comment about this since it resolved a discussion that had been ongoing with Dr. Jackson. After that, I commented on the workflow diagrams issue to see which sections should have diagrams created for them. I looked over the contributing document and thought that Getting ready to work, Work, and Getting your work reviewed and merged all seemed like they could use some diagrams. Finally, I figured out what I would be doing for the rest of the week and planned questions for the next day’s meeting. 

Tuesday started with a research meeting that was quickly joined by Dr. Jackson over the phone. We discussed various things including the ongoing project board issue that needs to get resolved so we could update the documents with which board structure to use and how to set it up. We also talked about changing to a different license for some of the project files. I helped to resolve a Discord issue with server joining messages. Most of what I would be working on for the rest of the week was beginning the migration from GitHub to GitLab as it was at this point a pretty sure thing that we were going with this platform. I would start this by deleting the previously imported repositories and re-import them.

Wednesday I started this GitLab migration and deleted all of the previous repositories in the LibreFoodPantry GitLab group. I then imported all of the projects from GitHub to my account and transferred them to the LFP group. One issue that I discovered when importing the projects was that it didn’t link issues that were created by some users on GitHub that didn’t have a GitLab account and it made me the owner of these un-linked issues. After that, I read over the license that Dr. Jackson suggested we start using. After reading it over, I agree with Dr. Wurst that it seems a little too complicated. After getting a response from Dr. Jackson on which sections to create diagrams for, I started working on this by creating a new feature branch. I started with the Getting ready to work section. I looked at the links he suggested for how commit diagrams were designed and somewhat used some of the styling from these later when I was creating my commit diagrams. As I was creating the diagrams, I was running through the workflow myself to see that it worked properly for me. I ended up adding some git commands into the contributing document that I thought would be helpful. I also created a new issue that I discovered when testing this workflow about how to shops update their master branch when changes are made in LFP’s upstream. I suggested using GitLab’s auto repository mirroring function that takes care of this automatically. I also tested out pushing an empty commit to a new branch and creating a work-in-progress merge request back to the LFP project. I thought it was cool that when you push to a branch after doing this the commits also go to the WIP merge request so others can see your progress. By the end of the day I had created diagrams for Getting ready to work and the work sections. 

Thursday was exciting. A decision had finally been reached and we would end up using GitLab and Discord! After seeing this I posted my migration issue and also replied what was the best document to follow for importing repositories. It was decided that shop managers would re-import the projects so that they would be the owners of any issues that didn’t properly link in the import. I then went back to the diagrams and tested out how you update a feature branch from the master branch. I realized that I forgot to add the developer’s computer to the diagrams and went back to the previously created diagrams and added this. I also wanted to figure out where upstream was pointing to and realized that it was the LFP project’s master branch after some searching. I then had to figure out how to set the upstream’s master for a cloned repository of this from the shop fork, this was aided by a past exercise from CS348. I think this command should probably be added to the contributing document since it is not mentioned and not something that happens automatically. I also discovered some other issues that should be touched up on in the contributing document but I wanted to finish the diagrams before migrating to GitLab so my commits would transfer over properly. The hardest part of the diagrams was creating the commit graphs. I figured this out by looking at the commits that were on the different branches through the Web UI and after that I discovered that GitLab has a commit graph for repositories. That helped greatly with creating diagrams for the merge commits part of the workflow. I then finished the Getting your work reviewed and merged diagram and pushed that to the branch. 

On Friday there was a bunch of notifications since a lot of new issues were created on GitLab since we started migration to this platform. I started looking at some of the new ones and also replied to Dr. Wurst’s question about why commits on his projects seem to have transferred correctly. I discovered that the issue with linking accounts is more related to issues and not commits and that GitLab seems to make dummy accounts for commits if it can’t link to an actual account. I then enabled the DCO push rule for Dr. Wurst’s imported projects. I then created a pull request for my diagrams before the ProjectTemplate repository was imported to GitLab. I  also assigned myself some new issues to work on. Later that night Dr. Jackson discovered when importing the ProjectTemplate repository that issues I created on GitHub weren’t being linked to my GitLab account on. We tried to resolve this in various ways by enabling different options. I tried signing in to GitLab with my GitHub account and set the same email address on both platforms to the same one and made it publicly viewable. Sadly, this did not resolve the issue and we would try to fix it the next day.

Saturday, Dr. Jackson decided that I should import the ProjectTemplate repository myself so that it would automatically link all of my work to my GitLab account. This worked fine except for a couple of merge commits I had done with a private GitHub email address through the Web UI. We decided this was fine and I left it as-is. Finally, I enabled DCO checks for the BEAR-Necessities-Market project and updated the issue with this information.

The Workflow in Action

Last week started on Monday with getting back on track with this project. I looked at some of the issue updates on GitHub including the issue about switching to Discord. I then emailed Dr. Wurst asking about how the DCO sign-off works for committing to the project, hoping that I could push my setup documents and diagrams before the next day’s LFP committee meeting. To my surprise, the GitLab Gold issue had been fixed and the WSU account now has access to all of the Gold features again. I was really happy about this since it means I could go back to testing the advanced features offered with this package. Sadly, the GitHub issue still remains but as of Monday they hadn’t deleted my testing accounts yet. After that, I looked at the Probot question one last time and created my reply to Dr. Jackson about this. Finally, I looked at Dr. Wurst’s earlier question about free server time for open source projects. I couldn’t seem to find much information directly about if this was available, at least with Amazon Web Services or Google.

Tuesday started with a research meeting with Dr. Wurst. We covered a lot in this meeting, but the biggest thing was separating out the old issue card and breaking it into multiple new issues, as the old one was starting to get too big and congested. Dr. Wurst then closed the old issue for good. The new issues included:

By doing this it makes it easier to choose one task and finish it and see the whole progress as it moves across the project board on GitHub. 

After the meeting, I read the contributing document to see how I am supposed to be making additions to this project before committing and pushing to the repository. I then updated my local repository for the shop setup documents and re-exported all of the graph files. I then looked at the first pull request for this project and approved it and merged it into the master branch. Dr. Jackson decided that we should start using our own workflow when making additions to the LFP projects now, so I created a new branch for my setup documents, signed the previous commits with the DCO and then push the changes to GitHub. I then created a pull request and requested it to be reviewed by Dr. Jackson. 

Wednesday started with reading issue updates. I then looked to see if there was a way to always sign commits without having to add the -s parameter each time. I couldn’t find what I wanted to for this. I then started working on revising the documentation according to Dr. Jackson’s review comments. I pushed my changes to the branch. Finally, I started looking at what to do for creating workflow documentation. I discovered that Dr. Jackson had already wrote a lot that covered this already and asked on this issue card what more I can add to it. 

Thursday, I found out that the links in the setup documentation for the diagrams had broken when merged into the master branch. I fixed these image links for the setup diagrams by making them relative as Dr. Jackson suggested. I then started figuring out the answer to Dr. Jackson’s question about how labels work on GitLab issues and issue boards. With this I figured out how the different labels work in GitLab. This included group labels that allow a group level issue board to control issues with projects underneath the group. In response to the original question about how the two boards had a “doing” column I finally came to the conclusion that the original hand-off situation Dr. Jackson was asking about must have had two different column names such as “Team1 doing” and “Team2 doing” as if both boards had the same column name, moving an issue on one board automatically moves it on the other. I then updated the issue card with my response. 

Friday, I updated the platform comparison feature sheet with some of the things I’ve discovered since using the platform and closed Dr. Jackson’s comment about WIP merge requests. I then posted the link to this Google Sheet on the issue card on the community board about GitHub vs. GitLab. I then reviewed and approved some pull requests for Dr. Jackson. I then started looking at the documenting continuous integration issue and looked at how to enable CI for a GitHub project specifically using Travis as it seems to be the most popular CI marketplace app on GitHub. I actually found this really easy to do, only needing to set the language in the config file to Java for an example project I had. Finally, I reviewed another pull request for Dr. Jackson and added some suggested edits before approving, especially fixing broken document links. 

Saturday, I approved and merged some more pull requests and continued looking at how to get the DCO bot to work on a GitHub repository in order to double check the instructions Dr. Jackson posted for how to do this in our documentation. 

Sunday, I looked at if GitLab has its own version of a DCO bot. I found through different pages that using GitLab’s Push Rules for commits you can create a rule that enforces all commits to be signed according to a regular expression. I then had to figure out the correct regular expression that checks that all commits have a line matching the form:

Signed-off-by: ‘firstName lastName’ <username@domain>

After reading through a couple of tutorial websites and finding this great one that lets you test in real-time your expression against a string I finally figured out the correct expression to be:

Signed-off-by: \w+ \w+ <.+@.+\..+>

I then tested this on the GitLab web UI to make sure it works, and it did. I then tested it with locally with GitBash and also with branches. I found it works a little differently than the DCO bot on GitHub and blocks all commits from being pushed unless they have the included signature. I actually like this as it prevents the commits from even getting pushed to a project without including the sign-off. The downside is that this sign-off is needed on merge commits too. I then updated the DCO documentation to include instructions for enabling this on GitLab and added some comments on the pull request about this. Finally, I created the high-level continuous integration document, added a definition for it and an overview of how to enable it on GitHub and GitLab. I pushed this and created a pull request for this.

In retrospective after this week, I found that I enjoyed the workflow we are using with selecting an issue to work on, creating a branch, pushing the work, then asking for it to be reviewed by someone else before merging. I especially like the reviewing part as you can always have a second opinion before posting the work you have done to the master branch.

GitHub and GitLab Issues

Week 9 started on Monday by dealing with the flagged GitHub accounts. As suggested by Dr. Wurst, I created a support ticket in the account and said that I was a student researcher testing out the GitHub platform for an open source project and if they could unflag the account and to contact Dr. Wurst if they had any questions about the project. I also included the issue of issue cards on project boards getting automatically removed by a spam filter. I copied this response for each one of the testing accounts so I ultimately filed 4 support tickets with GitHub. After that I tested out Dr. Jackson’s question about if shop managers on GitLab had the ability to manage their own subgroups if they are given owner permissions from the parent group owner. I tested this under the testing GitLab group by removing the shop manager account from the top-level group, creating a new shop subgroup, and adding the shop manager as the owner. This worked the way I thought it would and the shop manager was able to create subgroups within the shop subgroup and could add new members that weren’t in the parent GitLab group. I updated the GitHub issue card with confirming this finding. Then, I went on to read Dr. Jackson’s response to my earlier question of if it was still worth it to create documentation for GitLab Free based on his earlier response that GitLab Free doesn’t allow the creation of multiple issue boards and this would impact our use of the platform. Dr. Jackson’s detailed response included different ways of having workflows with the one issue board limit on GitLab Free. I ended up drawing some of these out on a whiteboard to visualize the concepts. I kind of like the concept of Dr. Jackson’s second option of having one board at the LFP level and that would be used by all of the shops. Although he says it is a disadvantage, right now I like the idea of all shops using one unified workflow as it makes documentation simpler and makes sure everyone is following the same steps to contribute to the projects. One question I have from this is how do the shop level boards look with this option or is it only just the one board in the LFP group? I also listened to the podcast he recommend about why a company switched from GitHub to GitLab. I found it pretty interesting and what they were saying mostly goes along with the differences I have found between the two services. I did like hearing from the perspective of someone who has a self-hosted instance of GitLab as this hasn’t been something I have been looking into but was interested in how this worked. Finally, Monday finished with discovering that our GitLab Gold membership wasn’t renewed properly and that we don’t have the Gold features available to us currently. This is a pretty big issue as it limits what I am able to test to just GitLab Free features. I emailed Dr. Wurst about this and he submitted a support ticket. 

For the rest of the week I checked to see if the GitLab Gold or the GitHub support tickets were updated. On Sunday I got a response from GitHub support to all of the support tickets saying that having multiple user accounts was against their terms of service and that they would remain flagged. I emailed Dr. Wurst about this and he would contact GitHub Education to see if there was anything they could do to help us. The GitLab Gold membership still hadn’t been fixed yet at this point. After looking at those, I went back to the diagrams I had been working on the previous week. I updated some of the GitLab diagrams to be consistent and fix an issue that it looked like the shop manager had to add themselves to a group which they don’t have to do. I then looked at some of the other documents in the community section of the LFP organization on GitHub to see how Dr. Jackson had formatted the other documents and what information to include. One thing I have a question on is what is the DCO and do I need to sign off on this for the commits I will be making to the project? Another is what to include for the licensing information at the bottom of the document? I then looked at the question Dr. Jackson had about if GitLab had an equivalent to the GitHub Probot framework for creating and using bots across the service. I found that although it seems possible to create bots on GitLab, they don’t seem to have a framework yet like GitHub does. 

Week 10 was short. On Monday I looked at the email response from GitHub Education regarding the multiple testing accounts. I found their response to be unhelpful to our situation and that their education features weren’t what we are looking for in testing permissions and project configurations for LFP. I took a quick look at the new website that Dr. Wurst created for LFP and how it currently links to the GitHub organization. I also read some of the principles listed on the LFP GitHub Community section like the Agile principles and the “FOSSisms“. I found the FOSSisms to be particularly interesting to read about with the process of contributing and working in open source projects, especially with how to get started with contributing to an open source project since that’s what I have been learning with this project. 

Moving on with Story Mapping Tools and Creating Setup Diagrams

Last week started on Tuesday by meeting with Dr. Wurst. In our research meeting we discussed various topics from the previous week’s work including the story mapping tools and the continued discussion on which platform to use for LibreFoodPantry for communications. We decided that based on my findings and what we currently know, draft.do is still the best story mapping tool and that we will start using this to transcribe the LFP user story map created during the June story map using this and I should update the GitHub issue card regarding this. Frequently when talking about various things it seemed that we will need a server for hosting various tools for the LFP projects so one thing I will look into is what services (if any) have free server time for open source projects. After discussing the workflow documentation I created for the shop setups I learned I needed to create diagrams for these and research or create my own way of modeling different things in GitLab and GitHub such as repository structure and permissions. 

Wednesday I briefly updated the issue card regarding if we are still creating documentation for GitLab Free since it has restrictions on the number of issue boards you can create. I also updated the story mapping card with what Dr. Wurst and I discussed the previous day about reaching a decision on which tool we are using so we can move forward with that. 

Friday I looked at the updates to the story mapping issue and started working on transcribing the user story map created during the June retreat from the pictures taken and put onto the Google Doc during the retreat. I found that this went pretty smoothly using draft.do and following the format for the shop-level workflow we previously did in this tool made it easy. I updated the issue on GitHub with my work in progress story map and asked for it to be reviewed to make sure I got the colors and card positions correctly. I then tried pushing the workflow documentation to the LFP/Community GitHub repository and ran into a permissions error so I had to email Dr. Wurst to get write permissions for the repository. I switched the documentation extension over to .md so it would be recognized as Markdown by GitHub and the operating system. I then started working on creating diagrams for the various setup workflow documents. I started out by drawing the initial diagrams on a whiteboard then I would transcribe them over to a digital format to be stored in the GitHub repository. I tried using the yEd tool that Dr. Wurst suggested for creating diagrams and found that it was a bit tricky with yEd being a lot more restrictive in where you can place things after previously using draw.io for making diagrams. Although after using it for a couple of hours, I found that it is easier to have consistent formatting with this tool versus draw.io and I’m starting to warm up to it more. One thing I want to figure out is where and what format should these diagrams be stored in the repository and should they be included in the setup documents. I ended up finishing the diagrams for both GitLab Gold and GitLab Free, along with refactoring the documentation, and started working on the ones for GitHub Free. When I went to use the shop manager test account on GitHub I found out that all of my GitHub testing accounts had been flagged by some kind of auto spam detection bot so I emailed Dr. Wurst regarding what I should do about this and ended the week with this. Next week I will continue finishing up the setup diagrams and documentation and move on to the workflow ones, along with responding to Dr. Jackson’s additional questions on GitHub.

Creating Workflow Documentation, Testing Boards, and Examining Story Mapping Tools

Last week I had three main areas I wanted to focus on: Creating or at least starting documentation for the workflows, looking at the different story mapping tools, and testing project / issue boards on the different platforms. 

I started on Tuesday by creating the documentation for the different workflows. I began with creating setup instructions for how to create the shops on each of the platforms. Throughout this process I used the new Shop-Level Workflow story map that we created previously and I found that this was very helpful especially with having the user roles above each step so I knew who has to perform which actions. As I was creating the documentation I was going through the steps outline in the story map on each platform with my account and the shop manager account. I did run into some issues doing this such as on GitLab the selected permission level for the shop manager in the LFP group can’t add new members or create shop subgroups, something that the story map says the shop managers should be doing. I commented on this issue on the GitHub issue card and it will probably be something that trustees have to do instead of the shop managers. By the end of Tuesday I had created the initial setup documentation for GitLab Gold and GitHub Free. 

Wednesday I read Dr. Jackson’s reply about the shop managers issue and tested out his question about whether shop managers can create their own subgroups if a trustee creates a shop subgroup for them and makes the shop manager the owner. I found that this is possible and most likely will be the solution to this permission issue. I added documentation for GitLab Free which differs slightly as it would be a standalone group instead of a subgroup of LFP. I also separated the documentation so each platform has its own document instead of having all three together. I then copied the text over and converted the documentation to Markdown so it can be pushed to the LFP GitHub organization. After that I moved on to testing the issue and project boards on GitLab and GitHub. I did this by creating boards for each test group and adding some issue cards and moving them around. I found that basic board functions worked the same on all three platforms. The biggest difference between GitLab and GitHub is that you can create labels in GitLab and that cards can automatically move as labels change or the labels will change as a card is moved across a board. This seems very helpful as it moves the card on multiple boards that use the same labels, making organization more consistent. One problem I found was that GitLab Free has restrictions on the number of boards you can create which may be a problem with the workflow we’ve designed for the shops.Saturday I started testing out the different story mapping tools. I created a new Google Sheet and created a row for each one of the features that both Dr. Wurst & Dr. Jackson were looking for in these tools. I then went through most of the story mapping tools listed on the GitHub issue card. The tools I tested were: cardBoard, miro, LeanBoard, FeatureMap, Draft, Craft, and StoreiesOnBoard. Out of all of these I tested I found that miro and Draft had the best feature sets without having to pay to use the tools. Out of all of them LeanBoard was the only I found to have direct issue tracker integration (it is a GitHub marketplace app). FeatureMap had some good features such as being able to assignee cards to users which I think would be helpful for claiming stories in the shop groups. Currently it looks like Draft is still one of the best options so far for free. Next week I will start on creating documentation for the shop workflows and possibly continue with the story mapping tools.