The Critical Path To Performance

Contrary to what people might believe, we’re not just doing reverse engineering and malware analysis here in Labs. In fact, we’re probably home to more people writing code than pulling it apart. We’ve done very little blogging about our engineering to-date, and that’s something we’d like to change. To kick that effort off, I’d like to give you a sneak peek into the lives of our Windows protection component developers.

We just finished engineering some changes that improved the boot time of Windows 10 systems running our software, and had a positive impact on the startup times of selected Windows applications. These improvements are especially noticeable on lower-end systems. In the process of doing this, we’ve managed to get our SAFE product to work comfortably on a 32-bit Windows 10 device with just 1GB of RAM. Not bad, considering the specs.

Low-end tablet

SAFE runs quite comfortably on this little thing.

Every code change, logic change, and feature that we add to our components has the potential to impact our product’s performance, so it goes without saying that we do a lot of performance testing. Every database is checked for performance before it is released to our customers. We have pretty extensive performance test automation running 24/7 against our internal, beta and production builds, on different Windows versions, against all of our products. We’re also closely following performance results published by external AV testing houses.

We frequently set aside time to profile our software under numerous scenarios – some that we define ourselves, and others that are defined by the Windows Assessment Kit. We run tests, collect all the metrics we can, and then study the output data with Windows Performance Analyzer and Windbg. Digging through all this data occasionally leads us to a bottleneck that we can optimize away.

Jose's performance testing setup.

Cutting-edge profiling technology + old school phone == win.

DeepGuard, our behavioral analysis engine, has seen quite a few optimizations of late. One major change we’ve made was to refactor the code we use to handle our hash calculations. Using Intel VTune, we profiled our existing hashing algorithms, looking for stalls in the CPU pipeline. Our findings allowed us to retune the assembly code running these calculations and drastically reduce the number of CPU cycles being used. As you can see in the below screenshot, we managed to squeeze quite a bit of extra performance out of this change.

CPU benchmarking tool

If you like to write asm, you’ll love doing this job.

Performance improvements are something we strive for on a constant basis. Here are some of the things we’re always working on:

Whitelisting Improvements

Every file we don’t need to look up in our cloud, put though a scan-engine, or run through a client-side analysis component translates into a better user experience, whether it be faster application startup times, faster file operations or an improved browsing experience.

Pruning Old Or Overlapping Detections

Most of our detections are intelligent, and are designed to hit thousands of samples. As we develop newer, better detections, we sometimes find that these new versions catch the same samples that an older detection would have. In these cases, removing the old detection gains us a small performance boost. As operating systems and software are patched against older attacks, malware authors need to look for new ways to infect victims. Some malware eventually goes extinct, and in these cases, we remove detections accordingly.

Finding And Fixing Bugs

No code is without bugs. We find and fix them all time. We’re constantly finding new ways to identify bugs we’ve never caught before, and we have systems that provide us with great visibility on crashes throughout our customer base. Occasionally, we’ll find and fix a bug that was causing a performance hit. That’s always a moment to celebrate.

Optimizing Our Critical Path

Scanning logic is complex, and we make changes to that logic all the time to solve customer issues, support new product features, support changes to operating systems or even drop unused functionality. We’re pretty much always working on our scanning logic, and while we’re doing that, we’re looking for ways to streamline it.

I hope this was an interesting look into a tiny slice of what happens in Labs on the engineering side.


Articles with similar Tags