This post is about Frigate. Frigate is an open source security monitor and recorder that makes heavy use of ai object detection. It can utilize CPU, GPU, or Edge TPU compute power to scan live video feeds for known objects, detect them, track them, and notify the user about their presence. It can also tie into Home Assistant or other automation systems.

I make pretty extensive use of this software in our network. We use it for occupancy automation, security, package deliveries, and more. It is also pretty good at enabling extreme laziness. Why get up off your ass to investigate a sound in the next room when you can just pull up the video feed?

By default, Frigate can reliably detect people and vehicles, some animals, and a few various objects that you will likely not configure. It takes a bit of fiddling and testing (and fiddling and fiddling and testing) to get everything dialed in, but once it is there, it works very reliably. The core dev team also recently started an add-on service (Frigate+) that enables you to train a custom model on your actual camera footage. Currently the Plus model is far more reliable on people, vehicles, and packages (it is even able to identify specific delivery company trucks like fedex vs ups), but it trades that for far fewer options for tracking. (Currently just person, dog, cat, deer, vehicle, and package - with sub labels for license plate, amazon, ups, and fedex (vehicle), and for faces (person)). The Plus subscription is $50/yr, comes with 12 retraining credits (so you can evolve your own model at whatever pace you like over the year), and also supports development of the Frigate project.

I started out with just a few cheap wifi low-resolution cameras (those $20 wyze cams from 10 years ago) covering the entryways of our home, and kept building out. At the time I am writing this, I am up to fifteen cameras (all but one being wired PoE cameras) on my home deployment. My outdoor cameras are all at least 4k resolution, the indoor ones a mix of 4k and 1080p. I am mostly using Amcrest cameras (although I do not use any of their services or hosting), with a few Reolink as well.

Frigate is running as a Docker Container, bundled with Go2Rtc. The later allows you to have one connection to each of your cameras, and then restream that feed to any other clients who request it. This made a huge difference with the cheap wifi cameras, and the wired cameras benefit a bit also. The system has an AMD Ryzen 9 5900HX and a Coral.ai Edge TPU for object detection.

This is running on House, a system with no shortage of things to do, and as you can see, resource utilization is still pretty low. I upgraded to the Ryzen 9 from an older Intel Nuc with an i5 when I noticed the system wasn't keeping up well with everything I was throwing at it.

I specifically selected cameras with lower resolution sub streams available, to help with network traffic and system load. Most of my cameras are configured to offer both a high resolution main stream, which is used for recordings, monitoring, and snap shots. As well as a 640x480 or so sub stream at a lower FPS, which is used for object detection.

With 15 cameras configured at about 5fps, Frigate is often only sitting at about 1-2% of the Coral's CPU and 1% of memory. I see inference speeds between 10-20ms. My CPU has integrated Radeon Graphics, but the driver doesn't behave well in Debian, so I have it disabled. Even without any hardware acceleration, the cameras each occupy about 1% of one cpu core apiece. Plenty of room for growth.

So what is all of this good for?

Well, for one, getting a notification from Home Assistant when packages are dropped off is handy.

Package detection in the Frigate+ Model

Frigate has a mature Home Assistant Integration, with support for using Frigate Events as triggers in automations, as well as providing the count, location, and type of object detection as sensors.

You can do some interesting things with zones in the configuration that let you dial in specific areas of the video feed. For example, if a person or vehicle stops at my mailbox for more than 3 seconds, I get a photo sent to my phone. If you stand in the kitchen with the refrigerator door open for more than a couple minutes, you'll get a snarky bit of dialog from one of the house's voices.

Of course it is also useful for mundane things like turning on lights when people enter rooms,

Also handy for figuring out who is pooping on your deck at night.

Shakes fist in air.

You can pair Frigate up with facial recognition engines and license plate readers as well. Used in conjunction with Who's at my Feeder?, you can even identify individual birds stopping and having a bite on your deck. (Unfortunately not working with Frigate+, since there is no 'bird' label yet).

This project has been a lot of fun to build out, and has also resulted in some unexpected advantages. We had some flooding in our basement last year, and I was able to utilize Frigate to determine where the water was coming from and catch it in the act.