How face recognition works
This page explains what happens behind the People page, and how administrators can tune it.
Two steps: detection, then recognition
Section titled “Two steps: detection, then recognition”Face detection finds faces. Frameleaf sends each photo’s preview image to your machine learning container, which downloads the model the first time it’s needed. A detection model draws a box around each face and gives it a score. Each face is then cropped and passed to a recognition model, which turns it into a set of numbers describing what the face looks like. Those numbers are stored in your database and indexed so similar faces can be found quickly.
Facial recognition groups faces into people. It runs once detection has finished.
For videos, only the thumbnail is checked for faces.
How faces are grouped
Section titled “How faces are grouped”Think of every face as a point on a map, where similar faces sit close together. For each face, Frameleaf looks for other faces within a certain distance, the Maximum recognition distance.
- If any of those nearby faces already belongs to a person, the face joins the person of the most similar face.
- If none does, Frameleaf counts how many similar faces there are, including this one. If there are at least Minimum recognized faces (three by default), the face starts a new person. Faces processed later that are close to it join that person.
- If there aren’t enough, no person is created yet. The face waits until all the other faces in the batch have been processed. If one of its matches has gained a person by then, it joins that person. If not, it’s treated as an outlier, such as a stranger in the background.
A face that has enough similar faces to start a person is called a core point. A face that doesn’t can still join a person through a core point, but can’t extend a person on its own. This stops loosely related faces from chaining two different people together.
Why upload order matters
Section titled “Why upload order matters”Grouping works on the faces waiting in the queue at the time, and keeps the people it already made. So the more faces in one batch, the better the groups. Recognition waits for detection and thumbnails to finish before it starts, for this reason.
If you uploaded in small batches, or the server kept up with your uploads, the groups may not be as good as they could be. Running recognition again for all faces regroups everything at once and gives the best result.
To keep up with new photos, Frameleaf only looks for a handful of the closest matches for each face, rather than every face within the distance. A nightly task, Cluster new faces, then finds unassigned faces that now have matches. The result is very close to grouping everything at once.
Run these from Administration, then Jobs:
| Job | Button | Does |
|---|---|---|
| Face detection | Missing | Finds faces in photos that haven’t been checked yet |
| Face detection | Refresh | Checks every photo again |
| Face detection | Reset | Clears all face data, then checks every photo again |
| Facial Recognition | Missing | Groups faces that don’t have a person yet |
| Facial Recognition | Reset | Groups every face again from scratch |
Settings
Section titled “Settings”Recognition settings are in Administration, then Settings, then Machine Learning Settings, then Facial Recognition.
| Setting | What it does | Default |
|---|---|---|
| Enable facial recognition | Turns face detection and recognition on | On |
| Facial recognition model | The model used to detect and describe faces | buffalo_l |
| Minimum detection score | How confident detection must be before something counts as a face, from 0 to 1 | 0.7 |
| Maximum recognition distance | How alike two faces must be to count as the same person, from 0 to 2 | 0.5 |
| Minimum recognized faces | How many similar faces it takes to create a person | 3 |
Small changes are safer than big ones. If you make one setting stricter, it can help to relax another to compensate, and the other way round.
Facial recognition model
Section titled “Facial recognition model”The models are antelopev2, buffalo_l, buffalo_m and buffalo_s, largest first. Larger models are slower and use more memory but give better results. The default is usually the best choice; pick a smaller one if your server struggles. After changing the model, run Face detection for all photos.
Minimum detection score
Section titled “Minimum detection score”Setting this low finds more faces, but also things that aren’t faces, which are hard to clean up and spoil the groups. Don’t go below 0.5. Very high values like 0.9 aren’t a good idea either: the default already leans towards being sure, so a high value misses real faces.
A change only applies to new detection jobs. To apply it to your whole library, run Face detection for all photos again.
Maximum recognition distance
Section titled “Maximum recognition distance”The default suits most libraries. Lower it if you have twins or people who look very alike. A value that’s too low only means merging duplicate people afterwards, but one that’s too high can produce groups that can’t be untangled. Stay between 0.3 and 0.7.
Minimum recognized faces
Section titled “Minimum recognized faces”This setting does two things:
- It takes effect straight away: people with fewer faces than this are hidden from the list.
- It makes grouping more robust, because a person needs a certain number of similar faces before it exists.
Raise it if you raise the recognition distance or lower the detection score. Setting it to 1 turns off the core point idea entirely, which suits you if you’d rather do more by hand.
Get better groups in a large library
Section titled “Get better groups in a large library”This method suits a large library after you’ve imported most of it. It builds the first groups from only the most photographed faces, then works down.
- Go to Administration, then Settings, then Machine Learning Settings, then Facial Recognition.
- Optionally, lower Maximum recognition distance, for example to 0.4, if your library has people who look alike.
- Set Minimum recognized faces high: about 20 for a library of 100,000 photos or more, or about 10 for around 40,000. Only faces that appear that often can start a person, which gives clean first groups.
- Go to Administration, then Jobs, and run Facial Recognition, then Reset. Wait for it to finish.
- Lower Minimum recognized faces: to 10 for a large library, or to 5 for a medium one. Run Facial Recognition, then Missing.
- Lower Minimum recognized faces to 3 and run Facial Recognition, then Missing, again.
Names in AI descriptions
Section titled “Names in AI descriptions”Named people can appear by name in AI descriptions. Only named people who aren’t hidden are used, and correcting a face marks the descriptions that used the old name as out of date.
Privacy
Section titled “Privacy”Faces are detected and grouped by your own server and are never sent to Frameleaf Cloud. People who appear only in Locked photos are hidden until you unlock.