Crafting a virtual event platform: case study
This story is my personal view on an interesting software development project. I’m a software architect at QAware in Munich, Germany. Magnid is a cloud-based platform for virtual and hybrid events by the company of the same name. They partnered with QAware to create an alliance to realize this extensive project.
The article’s focus is more on the project and business side of developing a virtual event platform, but will touch important technical points as well. It will be an appealing read for someone who is interested in realizing a similar digital product and wants to make ideas visible and touchable.
Have a look at the virtual event platform we’ve developed together in this demo: https://virtual-space.magnid.com/
Let’s get this started
In the first meetings Magnid told us about their current solution and their ideas and visions for the future of virtual events. With their input we created the first concept and came together to some conclusions:
- Use the same powerful concept as the old platform: The event website should be a virtual “Venue”, i.e. a building with rooms through which the attendees can navigate. Each room would have a 3D-rendered background image and clickable or embedded content like documents, chat or little games placed in it.
- To achieve the self-service event configuration, create an authoring tool that allows users to intuitively create their virtual Venue and integrate their content.
- Create a secure and highly automated cloud-based event platform that also supports “self service set up”.
Because of the first decision, the requirements for the “attendee web UI” (Venue UI) were already more or less clear. For the “authoring web UI” (Venue Designer), we created detailed UX concept which proved highly effective in the following months.
Based on the requirements and concept, we created a task breakdown with effort estimation to create a project plan proposal that balanced all factors, like target release date, budget and personnel resources.
We have committed ‘an agile fixed price contract’ to generate the highest possible impact for the customer by guaranteeing a result for a fixed price on a fixed date. In my experience, this is a very healthy reality check instead of making vague promises.
Both only harm themselves: the one who promises too much and the one who expects too much. (Gotthold Ephraim Lessing)
QAware project teams usually include, besides the developers, a project lead and a lead architect. These two are the main interfaces between the customers and the developers and they’re responsible for smooth project progress and technical quality. In this project, besides full-stack development (UI, Backend, Cloud) we worked on UX design, operations support and sparring for product vision and roadmaps.
Laying foundations
Magnid accepted our project proposal with pleasure and a contract was signed. Now the real work began. Many decisions had to be made in little time that would have a huge impact on future productivity, quality and on the success of the product.
On the project management side: Staffing of the team including a healthy ramp-up, initializing processes (like agile sprints) and project management tools (like Jira, Confluence), …
On the technical side: Deciding on the general architecture, development tools, various languages, frameworks and libraries, cloud provider, used products, …
Learning the new stack
Starting a new project with a new team and a new tech stack always requires a certain amount of familiarization until the team can reach full productivity.
We chose the tech stack to include a good mix of well-proven familiar technologies and newer innovative ones. This way the team had a good knowledge foundation and was able to adapt and learn the new parts in a very short time. After the first development sprint we already had a running prototype covering the whole stack.
API surprises
Naturally, you can never predict everything with absolute certainty. Surprises are bound to occur. Therefore, risk buffers are an excellent way to plan development tasks to avoid missing target release dates.
One of these surprises in the Magnid project was that we over-estimated both the feature set and the API performance of Google Cloud’s IAM product (identity access management).
The requirement was to have user accounts with roles. For example, the “attendee” role allows users to participate in the event whereas the “designer” role allows users to modify the Venue configuration and content. User accounts can be created in different ways, most importantly by “designer” users.
Our planning premise was that we could give “designer” users access to Google IAM’s admin UI where they then could manage user accounts. This way we would save the effort implementing a user management UI including the API integration.
However, when trying to integrate, we found out there are multiple Google IAM products and while “IAM” has an admin UI, it is meant for “infrastructure admin users” (like developers and operators), not for “end-users” (actual users of the system). The latter should be managed with the Identity Platform (which itself is based on Firebase Authentication). Identity Platform brings admin and client APIs and libraries and even a re-usable login UI component, but unfortunately no user management UI.
When you think about it, a user management UI can get quite complex because you need to show the list of users, allow changing roles, adding and deleting users etc. All of that must support a potentially huge number of users per event (10.000+) which means you also need search, filter, pagination and bulk operations. For all of that you also need to implement back-end services that use the IAM’s API in the right (read: fast) way.
Together with Magnid we discussed how to handle this issue and found a solution that kept the effort to a reasonable level. We settled with a simple UI integrated in the Designer application and using CSV file upload and download as an easy way to manage the user list. Designer users can open and edit these files in MS Excel or other tools and leverage those tools’ powerful features. We still had to implement the back-end services to parse the CSV files and integrate with the Identity Platform API, but in total this pragmatic solution saved us around half of the additional effort.
Ramping up
In autumn and winter both the pandemic and the demand for virtual events gained traction.
So we had planned to increase the team size: In the first sprint, the team size was around half the size as on the peak shortly before the first release.
For me personally, my work and roles changed a lot together with the increasing team size. With the smaller team in the beginning, as the tech lead, besides the usual architecture and technology decisions, I was very closely involved in the development. I was glad to be able to spend this time because we had to learn a lot and build new structures from scratch in many areas.
Optimizing operation costs
Using Google’s GCP as a modern public cloud for the Magnid platform had many advantages: high automation potentials, auto-scaling and reliability, world-wide server locations to choose from, integration with various ready-to-use secondary products (like container runtime environment, IAM, logging, certificate handling, …), and most importantly not having to maintain your own physical infrastructure.
After we had the first running cloud setup and an automation to fully provision new Venue instances with the click of a button. With this ease of use to create many instances, the infrastructure cost moves more into focus.
The Cloud platform does show which components cause how much cost. And in architecture, cloud product selection and resource usage we have many parameters that we can adjust to reduce the cost. Doing cost analysis and optimization is called “cloud cost engineering” and is something that we regularly do for our customer projects.
In Magnid @ GCP, we fine-tuned the cloud usage to strike an acceptable balance between performance and infrastructure cost.
It’s showtime
With the first virtual events being hold with the new platform, the project entered a new phase. Until here, the path was clear with a detailed concept and plan already in place.
From here on, there were many possible routes to take. Now that the new platform was live, there was a lot of feedback from users. Especially the feedback from (actual and potential) customers from this point on had the biggest impact on the selection of new features and improvements.
Operations
With going live came the need to operate the productive software. Fortunately, with the high grade of automation there was not too much operation overhead.
Still from here on, we planned a few days of developer effort per sprint to help operating the software. This meant primarily supporting the Magnid colleagues to provision, update and delete Venue instances and to do the occasional troubleshooting. To handle operations with this little effort was only possible because of the high grade of automation.
An event usually has a very limited time frame — from a few hours to a few days. If there is a problem in a real world event, the event organizers often have room to improvise. In the virtual world however, there are less options for immediate action. If there is an issue, you usually have to quickly involve the developers.
When called for a live issue, first of all, we needed to find and understand the root cause of an issue as quickly as possible. More often than not the issue was out of our hands, e.g. a cloud infrastructure hiccup or a wrong domain configuration. We tried to quickly find, implement and deploy hot-fixes.
Here is an example of a customer feedback for an event where we had introduced a new feature that didn’t work first as intended, but we were able to deliver a fix within a few hours:
Features, features, features
After the base product was ready, there were endless feature ideas already waiting in the backlog. It became clear quickly that we could not have a roadmap with the scope of a few sprints ahead because priorities changed quickly.
Within a few months, we tuned the development process to strike a suitable balance between being flexible enough to fulfill customer feature requests reasonably quickly, but still allow for dependable planning and dev team commitments. We decided to use a 3-week sprint cycle which would allow the team to develop and test small and medium sized features in one go. Each sprint resulted in a new product release version that was tested by the Magnid team and then quickly used in production. The sprint scope had to be finalized before sprint start and was considered “holy” — changing it was only tolerated in emergencies.
For bigger features, we used “concept user stories” that came before the actual implementation user stories. In the concepts, together with Magnid we designed the new features, both the UX part and the internal architecture. The finished mock-ups then became the template for the actual development. Pictures can demonstrate ideas a lot better than the most detailed text description.
While I still think that long-term roadmaps and goals are desirable because they can be a big motivation and give guidance in the day-to-day micro decisions, we learned and acknowledged in this project that “time-to-market” and delivering the right feature at the right time to get new customers can be more important in a dynamic market.
Quality
Even in a dynamic project context where speed is valued highly, caring for software quality is still ingrained in QAware’s DNA. In all of our projects we value quality highly by
- writing unit and integration tests for back-end and front-end code and reach a good test coverage
- using automatic static code analysis tools (here: Sonar + ESLint) with a zero violation policy
- peer-reviewing all non-trivial code changes before merge — doing this wholeheartedly is not just a quality measure, but also a great tool for learning and knowledge transfer
- performing manual developer tests and acceptance tests
- using a “Definition of Done” checklist for every user story to make sure we do not forget any routine tasks (e.g. updating documentation, ensuring browser and device compatibility)
Good developer documentation is also very important. You can see how useful it really is when a new developer joins the team. When writing and reading documentation directly in the IDE using simple markup like Asciidoc, it can actually be quite fun, because it’s near the code and you can fully use your IDE’s features.
Another important quality aspect is Technical Debt. Some is unavoidable, some results from deliberate shortcuts and some results from inadequate knowledge or time. If uncontrolled, it can lead to becoming unable to develop new features in a reasonable time and/or to an unstable, insecure, badly performing system. In reality, development capacities are always tight. We made sure to track Technical Debt in a transparent way and to find an appropriate balance between feature development and tackling Technical Debt.
Transitioning
In autumn of 2021, around a year after the project start, we entered the third phase. Magnid the product was now an established and stable platform and Magnid the company started to transition to a fully internal developer team.
Transitioning mainly meant that gradually more QAware team members left the team and more Magnid-internal colleagues joined the team. As the development team was already working mostly from remote due to the pandemic, physical distance was no big issue. We integrated the new colleagues by doing a lot of coaching and pair programming. It definitely also helped according to the new colleagues that we had an established and well-documented development process.
Farewell
I want to take this opportunity to express my deep gratitude to the colleagues at Magnid who are full of energy and passion for their product and company which was so cool to see. Our interactions were always positive and constructive which goes a long way to make the work enjoyable for everyone.
Last but not least, big thanks to all QAware colleagues who were on the Magnid project team over the 1,5 years until now. Working and learning with you guys in a “professionally casual way” was — as always — a big pleasure!
Want to know more about this project? Need help crafting a digital product? Leave a comment or contact me at <first name>.<last name>@qaware.de
