In January 2021 a group of 25 rangatahi from the Growing Up in New Zealand study spent an action-packed four days working in teams with researchers, design and technology experts in a “co-design” process. 

The project team partnered with the Co-design Lab, Rākau Tautoko and Merkle NZ (formerly Davanti) to plan and facilitate the workshop sessions.

As budding researchers, they explored the things that make their lives great and what questions they would ask their peers to find out how they are doing.  They then designed ways that these questions could be asked that would enable participants to take part from anywhere in the world, at any time, even in a pandemic!  

The young people helped to develop five different concepts for an interactive app.  Key features were secure sign-in, sound, friendly avatars, and epic journeys of discovery which the children felt would help users of the app feel safe, curious and engaged.  They told us how the app should look and feel, with preferences for a warm and friendly appearance grounded in nature and Kiwiana. 

Taking all the rich information collated from the workshop sessions, the team at Merkle NZ combined these co-design outputs with game design research and technical considerations to develop a concept. 

The premise of the concept was based on a waka journey between islands, interacting with characters and playing mini games. This concept was prototyped in physical form and play-tested with a small group of participants to further refine and develop the details. The focus of these sessions was on game mechanics, interactions and style preferences.

This concept was prototyped in physical form and play-tested with a small group of participants to further refine and develop the details. The focus of these sessions was on game mechanics, interactions and style preferences.

The concept testing provided useful recommendations and considerations for the user experience. Following further consultation with researchers and Te Ao Māori experts, the concept was further iterated, user journeys mapped and wireframes created.

With the help of our partners at Method Studios Ltd, the concept and wireframes were further fleshed out and a technical specification drawn up.  A narrative was developed to bring the storyline and characters to life, and to tie in with the theme of each island and the mini games. 

The character and avatar style were further refined to be modern and quirky, with opportunities for interaction throughout the journey.

Once the Our Journey app was ready, we invited participants in the Te Rōpū Pīata – Leading Light group of the Growing Up in New Zealand study to try it out over a period of three months. Participants in the pilot navigated to six different islands, and answered questions based on the following themes:

The three key aims of this pilot were:

1. To trial the effectiveness of the app as an engaging method of capturing context-specific rangatahi perspectives.
2. To check the relevance and applicability of the app across key ethnicity groups in Aotearoa New Zealand.
3. To ask rangatahi questions about their identity, experiences of school and family life, peers, relationships and wellbeing that are unique to this project, and outside of the scope of the traditional longitudinal research questions.

Importantly, the pilot also offered the opportunity to trial machine learning techniques to analyse the data and understand the limitations and potential biases.

Many of the young people involved in the initial co-design participated in the pilot of the app, which was an important step in the co-design process.

We gathered feedback from the participants and their whānau through the app, and also held both face-to-face and online workshops to gain a deeper understanding of their experience and opportunities for improvements.

Overall, the rangatahi who engaged with the app had a positive experience and found the style and content to be relatable and relevant to them. They appreciated the local look and feel of the app, particularly the use of te reo Māori and Aotearoa specific imagery, and the ability to respond more freely and openly about topics that are important to them.

Importantly, they considered the app to be fun, friendly and safe. Their comfort in sharing deeper personal information was due to the anonymity and privacy that came with the app being online rather than a face-to-face interview.

The evaluation process did however highlight several features that required further development in order to optimise and sustain engagement with the app. The key areas for improvement from the pilot were:

1. A re-design to streamline the consent and sign-up process to increase engagement.
2. Retaining a range of options for data capture and providing a greater sense of autonomy to participants within the app experience.
3. Adjusting the moderation processes and parameters to ensure the ongoing safety of participants.
4. Collecting a greater volume of multi-modal data in agreed formats to enable further machine learning analysis.

After a further period of development to incorporate additional features, questions and notifications, Tō Mātou Rerenga – Our Journey was ready to be rolled out to the main cohort of the Growing Up in New Zealand study (~6,500). The app was live and available from July 2023 to January 2024.

Over the course of the six-month data collection period, we heard from around 800 young people and gathered over 56,000 pieces of information in a range of formats.

Each response from participants was processed through Amazon Web Services to alert us to any strong negative emotions or indications that the young people may be unsafe. These were then manually reviewed by the team in a newly developed administration portal and acted upon if necessary.

Pre-processing in AWS also helped us to transcribe audio and video files, and identify any personally identifiable information that may need redacting as we prepared the data for analysis.

Given the volume and complexity of data collected, the team have been exploring the utility of advanced machine learning techniques to facilitate timely analysis of the multi-modal data. Using the new data collected and the existing Growing Up in New Zealand longitudinal data from the past 12 years, machine learning has also enabled us to look at wellbeing trajectories over time.

Based on the preliminary analysis and modelling undertaken as part of the pilot, a two-phase machine learning prototype framework is being developed.  In the first phase, the standard machine learning processing pipeline has been expanded to design supervised and unsupervised machine learning methods for the data.

Techniques are being used to handle both structured and unstructured data. This includes multi-modal data, including free text, images, audio and video. We are leveraging existing state-of-the-art techniques by using pretrained models, after fine-tuning them for a New Zealand context.

In the second phase, shifts in indicators in the cohort over time are being analysed using existing longitudinal data. Features/indicators are being extracted using established machine learning techniques, including classical feature selection methods such as principal component analysis, and later modern feature selection methods utilising nonlinear data projections into low dimensions to appropriately visualize. Concept drift detection will then be incorporated across features. Concept drift detection methods are typically used in temporally dependent data. These concept drift detectors (ADWIN, HDDM, SEED, etc.) will be used to evaluate the changes in trajectories of the child outcome/wellbeing variables.

Two main methods of traditional qualitative analysis are currently being used to make sense of the data: reflexive thematic analysis and qualitative content analysis. Reflexive thematic analysis, championed by Virginia Braun and Victoria Clarke, makes sense of qualitative data by developing, analysing and reporting patterns in the data – themes. These themes each capture something important about the data in relation to the research topic, and together provide insights into the lived experiences and perspectives of the participants. Themes are developed from codes and require ongoing reflexivity on the part of the researcher, acknowledging the inherently subjective nature of this process. Reflexive thematic analysis is comprised of six stages: familiarisation; coding; generating initial themes; reviewing and developing themes; refining, defining and naming themes; and writing up.

While the majority of data collected by the Our Voices project was qualitative, the structure and / or content of some of the open-ended questions mean that some responses are more surface level in nature. Accordingly, researchers are also utilising qualitative content analysis to develop categories and themes that describe different aspects of the data.

Projects utilising traditional (non-machine learning) methods of analyses are ongoing and we encourage researchers who are interested in using the data and who have expertise in other types of qualitative analysis to contact us.