The Mohua iOS App
Perch 2.0, a model of the world's birds, cut down to New Zealand's and run on the phone
Point your phone at a bird and Mohua tells you what it is. Leave it running and it keeps a list of everything that called while you were there.
How a call becomes a name
-
1
Microphone
Sound is read in 5-second windows, a new one every 2.5 s in Identify. The phone's gain control and noise suppression are built to remove the very sounds a bird makes, and both are switched off.
-
2
The first ear
In Monitor mode a light onset detector listens in the model's place and wakes it when a sound starts. On held-out soundscapes it woke the model for 2.9% of frames.
-
3
Perch 2.0
The model takes 27.5 MB on the phone. It scores every window for 215 New Zealand birds and 12 kinds of background noise.
-
4
Its own threshold
Perch's scores run on a different scale for each species. One shared bar would silence some birds and over-claim others. Each species is judged against a threshold fitted on its own recordings, where there are enough of them to fit one.
-
5
A word
Mohua calls the bird Confirmed or Probable and prints the fit behind that word beside the name. When a score falls just short, or the bird is unlikely where you are, Mohua asks you instead.
-
6
The record
The score, the 5 seconds of audio, a spectrogram stamp and the embedding stay on the phone. Every claim can be played back and judged.
Identify
Identify is for a bird in front of you that you want named now. It checks every couple of seconds and draws the sound on screen as it arrives.
Monitor
Monitor wakes the model only when something calls. It uses little battery and can be left running through a walk or a morning.
Play it back
The five seconds behind each name are saved with it. You can listen to them and judge for yourself. Every bird is listed with how many times it called, and when.
Dawn chorus
A ribbon shows when each species sang across the day, the week or the season, lined up against sunrise.
Conservation status
Every bird is shown with DOC's threat assessment. You can see at a glance when the one you have just heard is rare.
What is drowning them out
Wind, rain, traffic, voices, engines, aircraft, thunder, running water and a dog are reported alongside the birds.
Where you are
When location is on, a bird that would be unusual where you are is held back. Mohua puts it to you as a question, with its recording to play.
For 22 introduced birds, among them maina, kookaburra and galah, the New Zealand range is well documented. For these the distribution guide is also weighed into the score. One heard where it is scarce needs a stronger call before Mohua names it. Translocations have put many native birds in sanctuaries that a distribution map does not know about. For that reason native birds are never weighed.
The guide draws on DOC's records of island birds, the Birds New Zealand atlas and a distribution model covering all 215 birds. A rare visitor is the record most worth keeping. Where you are can only turn a detection into a question. A bird Mohua names keeps the score and the word its recording gave it.
Life list
The life list holds every bird this phone has heard, counted against the 215 Mohua can hear. Each bird keeps the recording its place on the list rests on, ready to play.
Te reo first
The te reo name comes first by default, with the English name beneath it. The order is the same in the list, on a bird's page and on the export plate. You can swap it in Settings.
Another microphone
Mohua can record from any input iOS can see, chosen in Settings. The screen always names the microphone actually in use. You will notice straight away if the recorder you meant to use is still in your bag.
Share a record
Confirmed birds, and birds you saw or named yourself, can be exported for eBird or as a Darwin Core archive for GBIF. A threatened bird's location is coarsened, or left out, before it leaves the phone.
On the phone
Mohua needs no account and works without signal. It goes online only for a map's tiles, a bird's Wikipedia picture and a record you choose to share. Pictures are off until you turn them on. At the end of a morning you can make one illustrated plate of what you heard.
The Mohua iOS App in use
From 14,795 classes to 215 birds
Every kind of sound the model knows, from all over the world.
They come from the 491 taxa in DOC's threat classification. Subspecies Perch cannot separate were merged, and extinct birds and vagrants were dropped. They span 52 families.
These are FSD50K classes, reported under 9 plain words: wind, rain, traffic, voices, engine, aircraft, thunder, running water, a dog.
Mohua can only name a bird through one of the model's classes. The recordings behind the calibration tune the thresholds of those classes and cannot add a new one. These 25 birds have no class of their own. Some were split from a close relative the model does know. Most are subantarctic or ocean birds with almost no recordings anywhere. The app lists them as unhearable to keep the count of New Zealand's birds whole.
159 of them come from the OSNZ Checklist of 2022.
Mohua starts from Google DeepMind's Perch 2.0, which knows 14,795 classes of sound from all over the world. The model was opened, inspected and cut down to the birds you can hear in New Zealand. Twelve kinds of background noise were kept as well. They let the app say what is drowning the birds out.
For the species it keeps, the smaller model gives the same answers as the full one. Compressing it to fit on a phone made no difference to which bird comes out on top.
Perch learnt its birds from noisy field recordings. Cleaned-up audio is unfamiliar to it, and the artefacts of denoising can look like faint birds. Mohua leaves the background alone. It listens for the wind, rain and traffic as well, and reports them alongside the birds.
Conservation status
DOC's threat classification of the birds Mohua can hear, shown with every detection
Threat classification · 190 birds
- Of concern – declining or worse
- All other categories
- 59of 190are declining or worse
- 25report no single status
Hearing a kōkako makes a different kind of morning from hearing a blackbird. Every detection shows the bird's threat status, and the screen gives it weight.
DOC's categories are an ordering with no numbers attached. The app can say one is worse than another, and it never averages them. Some subspecies that DOC assessed differently are a single class to Perch. For those 25 classes the app shows the range of statuses.
Thresholds from real recordings
17,781 recordings from 5 archives, 17 times as many as the first calibration used
Perch's own model card recommends tuning thresholds on your own data. Mohua's thresholds are fitted on real recordings of each species. 122 birds have a threshold of their own. Of those, 64 have a fit strong enough for Confirmed. The rest are still heard, as Probable, and each can move up as more of its recordings arrive.
The calibration corpus, run by run
A detection is stored as its raw score. Every record already on the phone is judged again against each new table.
- Fitted
- 122Species with a threshold measured on their own recordings.
- Confirmed
- 64Fits of at least 20 recordings, right at least 0.65 of the time.
- Precision
- 0.76Median across the fitted species, on a median of 36 recordings each.
Where the recordings came from
6,645 are Creative Commons or Crown Copyright. The other 11,136, from the Macaulay Library and NZ Birds Online, are used for non-commercial research and never redistributed.
Continuing work on accuracy
More from the Macaulay Library
The Cornell Lab's archive is the one large collection Perch was never trained on. It has supplied 10,269 recordings over two requests. The second request brought 1,944 recordings of birds the first did not reach, and added thresholds for 3 birds. On recordists held back from the fit, a refit on the new recordings did no better than the thresholds already in the app. Those thresholds were kept.
A classifier on the model's embedding
A small classifier trained on Perch's own embedding is being tested on recordists held out from every fit. On tūī it found more of the real calls than the current threshold. It also made about a fifth as many false detections an hour: 2.2 against 10.0, over 229 recordings. Classifiers for 65 birds now run inside the app beside the current thresholds, and their answers are compared and logged. None of them decides what Mohua names yet.
Your own identifications
An answer to “Did you see it?” is kept beside the model's own embedding of that call. Together they make labelled New Zealand field audio, of a kind no archive holds. The next calibration will draw on them.
Listens for hours
The onset gate that wakes the model in Monitor mode, measured on 242 held-out Macaulay recordings and on soundscapes
Running the full model all the time would be hard on the battery. In Monitor mode a much lighter first ear listens instead, and wakes the model only when a sound starts. A call the first ear misses is never analysed at all. How reliably it wakes matters as much as any figure above.
The recall figures come from 242 Macaulay Library recordings by recordists held back from every threshold fit. Nobody has marked where the calls begin in them, and the model supplies the ground truth instead. By Perch's account, the bird is in the five-second window where the clip's own species scores highest. If the first ear does not wake for that window, the bird cannot be named.
One hundred frames of a soundscape
The model runs on the lit frames. The duty cycle was measured on 200 soundscapes the gate was never tuned against. To keep them a hold-out, its sensitivity was left where it was after the measurement.
- Recall, where selective
- 91.5%119 of the 130 clips on which the first ear fired selectively enough for a miss to mean anything.
- Duty cycle
- 2.9%Of frames, on held-out soundscapes. About 219 inferences a minute.
- Species it woke for
- 88Of 90, each on at least one of its clips. The two it missed had one recording each.
- Set aside
- 95.0%Recall over all 242 recordings. The first ear was awake for 27% of their frames. At that rate it finds most windows by coincidence.
Focal recordings are made by pointing a microphone at a calling bird. Much of their length is bird, and the first ear is right to wake for it. For that reason the first recall figure counts only the clips where it was being selective.
Confirmed, Probable, or a question
Because Perch's scores are uncalibrated, Mohua gives each bird a word in place of a percentage
Bird apps usually turn an uncalibrated score into “87% confident”. Perch's scores are logits with a learned bias for each class, and they are not probabilities. Mohua never shows a score as a percentage.
Inside the app, a word like Confirmed cannot be created without the calibration it was judged against. The word is always shown with the fit behind it. If something in the pipeline breaks, the score falls to zero, and no threshold accepts a zero.
Mohua's own scale
Because Perch's scores differ in scale from one species to the next, each bird's threshold sits somewhere different on this line. The one marked is the mohua's. A bird with no measured threshold of its own is judged against the median fitted threshold, 0.0371.
Confirmed
The score is over a threshold fitted for this species on real recordings. The fit rests on at least 20 recordings and was right at least 0.65 of the time on them. 64 of the 122 fitted species clear that bar.
Mohua – threshold 0.0458, fitted on 110 clips, precision 0.79, recall 0.63. The badge on a Confirmed bird shows how many recordings its threshold was fitted on.
Probable
The score is over a threshold the app cannot vouch for as strongly. It may be a fit too thin for Confirmed. It may be a threshold measured on the species' own recordings that was wrong too often on them. Or it may be the shared default, for a bird with neither.
A bird stays at Probable until its recordings support a fit that clears the bar. 58 fitted species are waiting on that, and 93 are waiting on a first fit. The badge says which.
Did you see it?
When a score reaches half the threshold without clearing it, Mohua plays you the call and asks. It also asks when the bird is unlikely where you are. For a near miss, the bird it is most often confused with is offered beside it, and you choose between the two names. Your answer is recorded and marked as yours. It never becomes Confirmed.
You can answer a Probable claim of Mohua's own with Yes, No, something else, or Not sure. Yes leaves it Probable, with your mark beside the badge. No, something else takes it off every list. The clip is kept, labelled as not this bird.
A detection is stored as its raw score, and the word is worked out each time the record is read. Every record already taken is judged again whenever the calibration changes.