Reinventing AI Research & Development: Part I

Early last year, we kicked off an Artificial Intelligence Research and Development (AI R&D) program to stay up to date in the ever-changing field. When reflecting on our beginnings and comparing to where we are today, it is clear we have made significant progress. Growth around the initiative has been organic, stemming from passion derived from teams across the organization that face data challenges.

But with this growth, there came some common organizational challenges:

• No documented processes for project selection: Data scientists worked on what they personally felt was most interesting at any given time.
• Little direction around a technical platform or technologies to utilize: Work was done locally, often without version control.
• Lack of standardization around coding practices, style and quality: A lack of best practices and standard processes led to inconsistencies between code developed by various data scientists.
• Cumbersome and time-consuming knowledge sharing techniques: This either took away from actual R&D time or was simply pushed aside for client work.

As our R&D initiative grew, these challenges became more evident. There became an apparent need to design a platform, develop processes, set standards and employ a dedicated team to help manage the work and ensure smooth operations.

To address these challenges, we took a step back to define what AI R&D meant for our organization, and based decisions regarding our processes and tooling around those key values.

How does WWT define AI R&D?

Research and development departments may operate differently across companies, but functionally, they all aim to provide the same values to the organization. WWT is no different. Our initiative aims to:

• provide time for data scientists and engineers to "skate to where the puck is going to be," and not have to scramble to catch up on the cutting-edge tools, technologies and algorithms;
• hone the skills of our data scientists, engineers and consultants by providing stimulating work on concepts they may not have otherwise encountered, which in turn supports talent retention;
• demonstrate the unique capabilities​ and skillsets we build internally and increase our reputation of being a thought-leader​ in the AI space; and
• improve and innovate our internal processes, operations and lab environment.

In defining our AI R&D program, we found it very important to not only define what we wanted the program to be about, but also what we (explicitly) did not want it to be about to avoid compromising the integrity of our overarching goals.

We asserted that our AI R&D projects will not be:

• Focused on a specific industry or customer need; an R&D topic may leverage an industry-specific use case, but the results shall be presented in a format consumable by other customers and industries.
• Funded externally; this allows the program to remain neutral and unbiased.
• Investigating incremental improvements or seek quick wins; our charge is to better understand what the future of a given technology or concept will be in the 2 to 5-year time frame, not tomorrow.
• Performing research simply for the sake of doing research; they must adhere to strict (though somewhat subjective) guidelines, demonstrating clear value.
• Developing enterprise-grade products; because we are an advisory organization, the AI R&D projects are meant to build our expertise in a variety of areas so we can advise our clients on the latest and greatest tools and concepts available.

Ultimately, our research consists of internally funded, applied projects that advance our knowledge in the AI and ML space, produce reusable components and potentially result in a customer offering. These projects are opportunities to develop broader partnerships, generate intellectual property and potentially someday take on a few "moonshot ideas" of our own. Again, we documented and formalized roles, processes and a technical platform to guide us in the right direction.

People and process for mature AI research and development

In this article, you will read about two major solutions we came up with to combat the AI R&D challenges mentioned above and our learnings along the way of implementation:

• People: Formalize organizational structure and establish dedicated teams to ensure high-quality AI R&D work and smooth project operations.
• Process: Develop processes and standards to improve version control, knowledge sharing and code production.

People: AI R&D organizational structure

As our AI R&D initiative matured, the structure of the roles and people involved became more formalized. We developed an operations team, as well as a new project selection panel since the start of the year:

The wide variety of roles and skillsets, as well as the rotation program, allows interested individuals to participate, drive engagement in their projects and contribute to the growth of the AI R&D initiative.

Process: AI R&D operations

Several processes had to be put in place to mature the AI R&D program. These create a more objective program that can efficiently allow our team to learn and prosper while publishing high-quality white papers in a consistent fashion. Three major processes were incorporated over this past year that are noteworthy.

1. Operations team rotation

The operations team is a newly established structure of the AI R&D initiative that ensures streamlined communications and smooth operations of the program. The team consists of three volunteers from the analyst/consultant level and one representative from both the data science and engineering teams.

Formally the operations team assists with:

• Design and documentation of operational processes (e.g., rotation and knowledge transfer, project selection);
• Strategic communications (e.g., quarterly newsletter, presentations, blog posts);
• Project selection panel (PSP) coordination, weekly meetings and other ad-hoc sessions; and
• Project tracking, white paper editing and platform build-out documentation.

To support the initiative, the operations team introduced several tools to increase efficiency, promote transparency and encourage collaboration. For file sharing, working sessions and quick communications, the team leverages Microsoft Teams. We have also made use of an internal wiki for general knowledge sharing and documentation. Additionally, Trello proves invaluable, as it allows us to track operational tasks and weekly meeting agendas.

2. Project team rotation

The execution of the AI work is performed by our data scientists and engineers. We designed the data science rotation to maximize efficiency and knowledge-sharing, while also balancing client work. The rotation strategy involves two data scientists on R&D in a cycle (during non-overlapped periods) who contribute 50 percent of their time to this work. In the last week of a cycle, the next pair work with the current pair for knowledge sharing​ and hand-off.

We realized this optimal rotation schedule after attempting several different iterations; we found a balance in terms of the number of people on rotation, the percentage of their time and the length of time needed for knowledge share with the next rotation.

With two people simultaneously on rotation, ideas can be exchanged in a fluid manner with parallelization of experimentation. The week of knowledge transfer is long enough to explain the work that had been done, but not too long for four people to spend time on R&D work.

The engineering track is a recent addition to the project team rotation. The rationale behind their establishment is to work on production efforts within the platform, taking trained models developed by our data scientists and testing them in production environment to stress the platform's capabilities. Currently, the engineering team has three focus areas:

• building out a production pipeline;
• building out GPU as a service; and
• building out a data pipeline.

The structure of the engineering team rotation, while longer in length, is similar to the data science's, where a transition period allows the previous rotation to transfer knowledge of their work and efforts to the new rotation.

3. Project selection panel (PSP)

As the AI R&D program matured, the team faced the challenge of prioritizing and selecting projects to best represent our vision. We created the AI R&D PSP with experienced managers to effectively act as the product manager for the initiative as a whole, and vote on new projects. Project ideas flow through a pipeline with the following tollgates:

1. Elevator pitch: Individuals make a one-minute elevator pitch of their rough idea on the weekly AI R&D call to garner interest and exchange ideas with the entire team.
2. Application submission: Once an individual or team of people feel they have an interesting idea, they can submit it to an online application form where they will be required to answer questions across a variety of dimensions described below.
3. Project Pitch: If the PSP approves their application, the project will get an opportunity for a 30-minute pitch at the monthly PSP meeting.
4. Selection to backlog: If the PSP approves the pitch, the project gets added to the certified backlog.
5. Project selection: As the current project is winding down, the PSP meets to select the most appropriate project to go next from the certified backlog.

The PSP is focused on building the certified backlog with projects that will provide significant value for our team and WWT as a whole, as well as our customers. There are six dimensions the PSP grades each project idea across:

1. Innovation – How is this project leading to something uniquely valuable to both WWT and our customers?
2. New learnings – What are we learning that will provide a completely new skillset?
3. Marketability – How will this project create awareness with our customers and potential customers?
4. Client benefits – How will our customers directly benefit from this project in the near future (2-5 years away)?
5. Feasibility – Do we have access to the appropriate tools, technology and data to do this project?
6. Platform benefits – How will our AI R&D platform grow its capabilities based on the unique needs for this project?

Currently, we weigh all aspects equally, and there is both a quantitative and qualitative aspect to the selection process.

Next steps

As our AI R&D program grows, we are working to mature aspects of the people, process and technology so it can scale accordingly. In this post, we described the expansion of the AI R&D organizational structure to include both an operations team and an engineering team, ensuring we are running a tight ship and supporting the R&D work with the latest and greatest tools and technology.

We also discussed the formation of a project selection panel that provides structure around project selection. These additions have been key to the growth of the program, but there are several other areas that we still need to improve upon.

Learn about our most recent work in Part II of this series.