27 September, 2023 - WASM is a dead end?

SO there are some limitations with WASM the main one being it can’t access external assets directly. After doing a Rust to WASM tutorial for making Game of Life I realise that to get it to work with all the resources is going to require a lot of custom JavaScript. It might be better to try and compile for iOS and Android separately.

I’m kind of demoralised by this setback. I shelve this project for now. I might try and use Rust to do some MLOps and build a SaaS service, get some experience with Rust and then come back to this.

I thought there would be some easy low hanging fruit in developing educational games but the gatekeeping on iOS and Android is very off putting and I can see why the market is limited at the moment.

Maybe I’ll just start again from scratch and do it in Javascipt?

TODOs:

MVP - Reimplement in Rust

• Unselect when wrong order
  • Track current state of selected
• Sound!
  • Play word sound when word appears.
  • Play phoneme when block selected.
• Exploding blocks
  • Make block disappear
  • Explosion in Rapier?
• Make words from JSON syllabus

Prototype

• Blocks use Pastel colours and selected to be Bright Red
• Splash screen?
• Announce the number of words left to explode.
• Should track what has been played and what took a long time vs what was easy and took less time.
• Make the selection more obvious i.e. rotate colour of selected?
• Bling Effects when:
  • the countdown time is updated.
  • word changes.
  • Reverse gravity for a word?

18 August, 2023 - Pivot back to Rust

So porting to Android and iOS is not going to be trivial. For vanilla pygame its ok but the pymunk is making it a nightmare with long lists of C++ errors. There is a fairly comprehensive guide but I’m not interested it learning all the ins and outs of porting python to Android let alone doing it all again for iOS (I’d need a Mac with enough space to install Xcode on which isn’t readily available.). I could port to Kivy, which might make things better but again fighting to export to different platforms.

Deploy to the Web!

I’m thinking of going back to Rust + Bevy (ECS game engine) + Rapier (physics) with the WASM target output so I can just deploy it to an S3 bucket and any platform can use it. Will mean that I can also share with my son when I’m in Sydney for extended an period. It will also mean no App store(s) to deal with and it will be really easy to share with anyone for testing, it just a link.

I knew this wisdom from my web-dev days but somehow have forgotten it: The web is the best distribution medium for digital anything.

Bye-bye Python

Prototyping in Python was great as its fast which means iteration time is short, which is what you want when building a throw away prototype. Too much effort and you get attached to the code.

12 August, 2023 - more play testing

Got an hour to work on the game today, Saturday. Added all the two word content. The three letter words is going to be a lot of work.

I wrote a script to download the word audio from Google search page results when you proceed the word with “define”. Worked really well and got most of the missing two and three letter words.

Also wrote a quick script to list all the audio and image assets that are missing. Made finding them and updating very quick. Asset testing is very handily when there are lot of details like this which need keeping track of. Some of the letter images I’d downloaded at the beginning of this project need to be redone in Gimp. I also make a script to backup the assets to S3 as I don’t want to keep them in github.

Son played the game again today. I was fixing a bug next to him on the couch while he was watching TV. He stopped wating TV and started to play the game. He really got a kick out of the explosions this time and because most of the content is two letter words now he had an easier time of it.

The selection thing is a bit confusing as the order of phonemes only gets checked once enough of them are selected to make up the word. Needs to happen on every selection and some sort of indication that the wrong order was selected. Perhaps play the phonemes in order as reinforcement?

10 August, 2023 - First play test

So I gave it to my son this morning and he played through it a bit. Some notes:

• Was too fast. One second between words dropping was too short, especially at beginning. Perhaps increment as go.
• Didn’t get the idea of spelling out the word. Need to somehow be explained? Perhaps 3 phoneme words were too hard to start with, try 2. Will make discovery easier.
• Found the explosions fun and laughed when they happened.
• Once all the letters filled the screen it became a challenge. He had to know the order of the letters.
• He found blowing up his own name a kick 😂

New material

I too the kid to school this morning and while I was waiting for the teacher to show up I found a phonetic chart. On the back were lists of words in alphabetical order! Looping through the alphabet can make up a level! Seems obvious now, but I guess since I’m using phonemes not letters maybe its not. Though some letters have limited number of 2 and 3 words… still, will be easy for 3 letter words.

5th August, 2023 - “-er”?

So I ran into an issue with translating “Asher” into phonemes. IPA-dict says its /’æʃɐ/ but I was unsure about the ‘ɐ’ sound/phone for UK English as it’s the same as vowel in fox /fˈɒks/ (according to the same dictionary, but now that I look at it, its a different upside-down ‘a’). According to the video ‘er’ Ending Words and the Schwa Phoneme it should be the Schwa ‘ə’ sound.

According to the IPA interactive chart it seems that ‘ə’ is in fact correct, but all three could be correct depending on which part of the UK your from? The OED says ‘ə’ so I guess I’ll go with that. I’m not sure how one decides what is the official phoneme for a pronunciation for a dialect. Maybe the OED is it?

3rd August, 2023 - Explosions!

Got the explosions working. Had to improvise with pymunk as it doesn’t have a built in explosion effect. I replace the block to blow up with a circle for ~200ms and then make it disappear. I also made the explosion weightless so it wouldn’t roll down.

I got the explosions sounds here and the image from wikimedia. Good bless creative commons!

25th July, 2023 - First demo

So I’ve started to code up the demo of the idea. Still much to do:

• Sound phenome when clicking box
• Play word sound when phonemes appear
• Track order that letters are selected in
• Letters explode when right order is chosen
• Compile rest of the words Content. (I just choose words I had voice recordings for)

Side note

It was pretty cool to get the first(ish) version working, and actually it was very easy with python. When I first saw the thing I’ve had in my head for weeks appear on screen and start to work(ish) there was a buzz of gratification. Its a cool feeling.

I watched a short video of Trevor Noah talking about how he loves stand up comedy and giving the life advice of:

Live in a space where you are doing what you love.

I think that games could be that space for me.

23 July, 2023 - Physics is fun

So I have a basic concept for the game. Its going to involve blocks dropping and using a physics engine to simulate that. I found a simple 2D physics engine called pymunk which is an interface to the C library Chipmunk2D. pymunk is under active development while Chipmunk2D seems to be feature complete. There is heaps of tutorials with pygame and pymunk so getting up-to-speed is quite easy and fun! There was a lot of low hanging fruit! To list what I went through:

• Python Physics Simulation. Galton Board. Pymunk Tutorial I watched this and used the code as a starting point. Great template for getting pygame and pymunk talking to each other.
• Using mouse and keyboard helped me create the touch/click interaction I need for interacting with the blocks.

I did give up on trying to rotate the images to the same angle as the physics block.

14th July, 2023 - Just graphemes to phonemes

Phonetics is quite the rabbit hole. I just need English spellings, or graphemes, and their related phonemes. I found this chart for Australian English (I think)

In order to represent the 44 phonemes of Australian English we use a range of graphemes (letters or letter combinations). Children need to know 150 - 200 of these phoneme/grapheme correspondences in order to read a reasonably complex text and these need to be taught explicitly and systematically. - cite

This two part video series about helping children sounding out words I found quite helpful. The second part mentions some recent research¹ on the topic and apparently the current approach to looking at the pictures and guessing the words is poor or “inefficient”. She recommends a problem solving approach of breaking the word up by underlining the phonemes and helping them with phonemes that the don’t know but letting them sound out the word phoneme by phoneme until they get the word.

An Idea!

This has given me the idea to have words broken up into their respective phonemes, with different colours, but written correctly and then slowly drift apart. The player then has to hit the phonemes in the order they were originally, sounding out the word. This then multiplied into many examples of the same thing with the letters getting mixed up, like soup, and the challenge is to put them all back together, click-on/touch them in the right order to clear the level.

How to start

There is a great set of documents which outline the Australian reading syllabus at Firefly Education.

I’m going to base my syllabus on the Sound Waves NSW Early Stage 1 Syllabus. From that document:

In Sound Waves Foundation, phoneme–grapheme relationships are introduced in a very specific order. This minimises confusion for students and ensures they are up and running quickly with reading and spelling. The order begins with m, a, t, s, i, d, f, n and p so students can read and spell CVC words such as mat, sat, did, nap etc.

They have a system of pairing graphical icons for phonemes with the graphemes which is a good idea but stems from using print as a medium. I hope that I can just relate the sound of the phoneme directly with the graphemes by playing the sound when they touch the grapheme. This is the advantage of using an interactive medium (I hope).

Tracking learning

Also, I want to track the learning of a child. At the very minimum I’m going to need keep track of which part of the syllabus they are up to. The video game way is to just make them start from the beginning every time. Perhaps they unlock levels as they progress which gives them new icons at the start of they game so they can jump start at the latest or revisit previous levels?

Also, some sort of report for for parents and teachers would be good. Not sure what they would like it see? Perhaps what they are good and bad at in terms of grapheme-phoneme mapping?

14th July, 2023 - Deep diving into Phonemes

So I’m trying to build the game content/levels aka the syllabus. The general structure is going to be:

1

Phonemes/Phones -> Words -> Texts

I’m focusing on the first step, but would like to incorporate some of the second. To begin with I need a list of the phonemes in English and grouped into vowels and consonants. I also need a list of short words (2-3 letters/phonemes) starting with the simple consonants, one vowel sound only. I combined the word frequencies with the word->phoneme dictionary and then filtered to a list of words with one-to-one mapping of letters to phonemes (was a crude estimate will update the Gruut IPA module [see bellow])

Gruut IPA module

I found a great python module for working with phonemes called Gruut IPA. It has some data for different languages (I’m keeping this in the back of my mind). I might try and do Russian in parallel so I can use the game for learning Russian and use myself as a test subject, but don’t want to get too side tracked for now. The British English file has phonemes grouped which is handy. There are 42 phonemes. I don’t know where the 44 came from?.

I tried to install the module and use the “print” function but it seems to only spit out 33 phonemes? I’m just going to write my own parser for its data files and move on. Lost a day to digging around it’s code. Its got way more info than I need.

Elsewhere

This is all good research for doing Text-to-Speech which I want to get back into. I might train a model to convert text to IPA phonemes as I think I have an idea of how to do that now.

9th July, 2023 - Chat with a Speech Pathologist

Had a chat with my old friend Khye who is a speech pathologist.

Her advice was to start on the simple one letter to one sound i.e. one-to-one mappings between glyphs and phonemes. Once they are confident with that start them on simple words. Then with words that do not have consonant clusters i.e. /brick/ where the “ck” are acting as one sound, better is /dog/ where each letter is a single sound and those are constant

The words should be two to three letters and also only use one-to-one, one glyph to one phoneme sounds. There are some texts with this goal in mind released by the NSW government called decodable texts

Decodable texts are specifically written for beginning readers as they are developing their blending and segmenting skills and their knowledge of the alphabetic code. Decodable texts support students as they practise by using a continuous meaningful text. Decodable texts contain a very large percentage of words that incorporate the letter-sound relationships that students have been taught. Decodable texts increase in complexity as the student learns more of the phonetic code.

These texts are behind a login so not accessible to the general public (for some reason?), but with the ipa-dict I should be able to make my own: Filter words that have matching number of letters and phonemes, sort by word frequency, filter by number of letters/phonemes i.e. start with 2 and 3 letter words and go up from there. I guess the next step would be to make narratives out of these words.

8th July, 2023 - Building a Syllabus

I finished organising the English phonetic sounds I gathered from the phonic letter sounds video. It has both the English and American pronunciations but seemingly only for 42 out of 44 phonemes for some reason.

But now what?

Data driven syllabus

I was thinking of taking a data driven approach to building a syllabus. Look at word frequencies in large corpus which I found for download on Kaggle’s English Word Frequency: ⅓ Million Most Frequent English Words on the Web. So I can get the most frequent 2, 3 and 4 letter words. But what I really need is frequency of 2-4 letter phonemes. So I need a dictionary that can translate the orthography into IPA phonemes. Fortunately someone has compiled such a dictionary: ipa-dict - Monolingual wordlists with pronunciation information in IPA.

I also found a great collection of African stories for different reading levels which might come in handy with IPA versions.

Name change

Change the project name to “Phonetica”! Maybe that can be the name of a fantasy kingdom and the phonemes that inhabit it can anthropomorphised?

Side note on Rust

I found this really great online synth called Ironfish written in Rust as a demo for a UI lib called makepad (RustNL 2023 talk). Making the game in Rust will mean that a web version will easy(er) to make. Worth considering.

6th July, 2023 - working memory and games

This book review came up on Hacker News today:

He devotes an entire chapter to the issue of computer games and the question of whether they are a negative influence or a positive one (in terms of cognitive ability — he sidesteps the ‘violence debate’) . He points out that there is no general answer, referring back to the earlier discussion: some types of training are more effective than others, and it is critical that the working memory be pushed to its limit repeatedly. Thus it is critical to consider particular computer games, instead of trying to make a general statement about computer games as a whole. He describes a variety of studies that show some types of computer games as having a positive influence on cognitive skills. - Book Review: ‘The Overflowing Brain’ by Torkel Klingberg

I should look into the reference he uses for the game stuff. I haven’t thought about how how games can help train and improve working memory. Could be an interesting direction.

He also goes into Flow which is another topic I need to investigate in regards to education

Klingberg concludes with a short chapter on the “information flood” that exists today and studies of its effects on cognitive ability. The results of some of these studies point to a kind of ‘sweet spot’ of information flow — we perform best when the challenge presented is just at the limits of our skill. He refers to a diagram developed by Mihály Csíkszentmihályi, who defines ‘flow’ as “characterized by a high level of challenge and skill, in which the capacity of the doer exactly matches the demands of the task being done.... [W]hen challenge exceeds skill, we get stress. When skill exceeds challenge, we get a sense of control, which becomes boredom as the level of challenge drops.” (p. 168)

29 June, 2023 - Research time

So, I’ve hit the point in which I’ve run out of things to code and need to do some research and design.

phonemic awareness performance can predict literacy performance more accurately than variables such as intelligence, vocabulary knowledge, and socioeconomic status² The good news is that phonological awareness is one of the few factors that teachers are able to influence significantly through instruction—unlike intelligence, vocabulary, and socioeconomic status ³. cite

Blending phonemes into words and segmenting words into phonemes contribute directly to learning to read and spell well. In fact, these two phonemic awareness skills contribute more to learning to read and spell well than any of the other activities under the phonological awareness umbrella. ⁴

So blending phonemes into words and segmenting words into phonemes are the main objectives of this project.

26 June, 2023 - Pivot (again) to Kivy

So while setting up pygame, which is really going back to basics, I was reminded of Kivy which is designed to make user interfaces. Another issue with pygame is that it is difficult to compile for iOS, and that is a target of this project i.e. touch screens are what I’m designing for, and Kivy makes his easy out of the box.

The only problem with Kivy is that its model is a but opaque and the documentation is in various stages of freshness. The API documentation seems up to date but doesn’t have so many examples of common use cases and the tutorials are over a decade old. There are examples that I have yet to delve into, most of them are pushing a decade old as well.

On the Bevy front

I did a little experiment on Friday. I had a question that I posted in three places, the bevy github discussion, Stack Overflow and r/rust_gamedev. The only one that got a reply and indeed and answer was Reddit (took 3 days). I think that Reddit works so well because developers go there for many reasons, news; gossip; ideas and help, all mixed together. So if you need help you have the highest chance the person with the answer’s eyeballs will see your post. Since the other two are just help support, people are less likely to look there. There is a threaded Help channel in the Discord that I didn’t look at which seems active, I’ll try that next time.

So anyway the answer was you have to figure out scaling stuff yourself but you can get the info in the Transform:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12

fn print_sprite_bounding_boxes(
    mut sprite_query: Query<(&Transform, &Handle<Image>), With<Sprite>>,
    assets: Res<Assets<Image>>,
) {
    for (transform, image_handle) in sprite_query.iter_mut() {
        let image_dimensions = assets.get(image_handle).unwrap().size();
        let scaled_image_dimension = image_dimensions * transform.scale.truncate();
        let bounding_box = Rect::from_center_size(transform.translation.truncate(), scaled_image_dimension);

        println!("bounding_box: {:?}", bounding_box);
    }
}

So since I got an answer I might persevere with it a bit longer.

New Strategy

I might do two versions of the project at the same time. Since it takes soooooo loooong to get Bev answers I’ll work in Bevy until I hit a blocker, ask a question and then swtich to Kivy until I hot a blocker and then switch back again. I really want to learn Rust and I don’t have the bandwidth to learn it separately from the one side project I do have time for so I’ll keep tyring to shoehorn it in.

23 June, 2023 - Pivot to pygame

So as much as I like the setup of Bevy’s ECS I hit a major road block and can’t get any help on it. I basically want to get the pixel size of a scaled image and there doesn’t seem to be any way to do that in Bevy. I think because transforms are in WebGPU (WGPU) and Bevy its a one way street with from Bevy to WGPU.

I found a hard time getting help with this as the Bevy subreddit has been taken down due the the Reddit hullabaloo that is going on. The moderator didn’t bother with a transition period so there is nowhere to ask questions. I tried posting the same question on the bevy’s github discussion board, /r/rust_gamedev and Stack Overflow to see which one would get a response but so far nada.

In the mean time my son needs some help with reading so I’m going to pivot to pygame as I know I can bag out something fast.

I’ll learn Rust on one of my other side projects. I think approaching it from the other side, see what low hanging fruit there is a and make a game around it rather than fighting Bevy to do something specific. There is a fluid simulation lib I found (which is based on a good, pure Rust linear algebra lib) which would be fun for kids as they love water parks especially sandpits with water!

Starting at the end

So the first thing I want to do is figure out how to make a n executable from my python script so I can CI/CD all the time and have it playable on 3 OSes (not sure if i get it working on MacOS if that will work on an iPad as that would be a great market). There are two options I can see pyInstaller and cx_Freeze. In both cases you need to run their build on the target OS.

15th June, 2023 - Sound problems

SO I have the button on the screen and it makes a sound when you press it. If you smash the button, which 5yos will do, it triggers so much that I fear the app will crash. I thought to try and fix this by installing a Bevy Kira extension but it very difficult to install. Will ask on some forums how to get it going. Don’t want to lose momentum so I’ll just use the built in stuff. Sound always plays second fiddle to graphics in games.

Next I need to investigate events in Bevy and get the letters spawning when the button is pressed. I’m thinking SVG for letters so I can go crazy with fonts.

Bevy Events

There are pretty simple. Four steps

1. Create a plain public struct for the event

1

pub struct MyEvent {};

1. Register it on your main app

1
2
3
4
5

fn main() {
    App::new()
        .add_event::<MyEvent>()
        ...
}

1. Trigger events in a system using an EventWriter

1
2
3
4
5

fn some_system(
    mut somthing_happened_event: EventWriter<MyEvent>,
) {
    somthing_happened_event.send(MyEvent{});
}

1. Then receive the event in as many systems as you like

1
2
3
4
5
6
7

fn debug_system(
    mut somthing_happened_events: EventReader<MyEvent>,
) {
    for event in somthing_happened_events.iter() {
        println!("Somthing did indeed happen...");
    }
}

13th June, 2023 - First steps

So after watching a few game dev videos on YouTube about indie success stories, they seem to recommend keeping a game dev log to help with motivation. So here I am.

My son is five and a half and beginning to learn to read. Tonight actually he spelt out the word ‘wash’ in a Sesame Street book I was reading him to my amazement. I wanted to make a game for him to help him with the sounding out of letters and ultimately help him to sound out letters. My only fear is that it will take too long and he wont need it by the time I finish.

I did start to look into phonetics and phenomes, there are 44 in the English language.

Phonemes are taught as part of the Phonics Phases 1-6 and are a key method in helping children learn how to read. Phonics links phonemes, or the sounds, with the graphemes, or letters, they represent. cite

Sound

I found this great video of a woman sounding out each of them.

I downloaded the video, extracted the sound, converted it to WAV, used Audacity to cut it up into individual voicings and then had to convert them to Ogg Vorbis audio format because I couldn’t find any documentation on what audio types Bevy accepts out of the box other than the example of .ogg files. I still have to go through all of them and name them something useful. There are 4 soundings of each phoneme and a word example for each.

Fonts!

I’d forgotten how much fun fonts can be! Its going to be easy to give letters character using different fonts.

Follow the fun

But how to make it fun? I have a very basic idea of what I’m going to start with and hope by getting my son to play test it I can figure out what he likes about it a follow that. I saw a video by Sid Meier (Sim City, The Sims) on Master Class about making games and his advise was to prototype ideas fast and test them for fun. You wont always know until you play what mechanic/dynamic will be the fun part. “Follow the fun!”.