Getting Started with ZAPI

ZAPI — The worker hands behind Brim

Photo by Alina Grubnyak on Unsplash

This article talks about ZAPI, short for Zed API, the worker hands behind BRIM. It will briefly cover zq, zqd, zql, and ZEEK. Why do you need it, you say? Well, if you are someone who deals with a lot of large Packet Captures regularly, ‘large’ being the keyword here, then this ecosystem that is BRIM will make your life a whole lot simpler. You are probably working with Wireshark (or some variant of it) at the moment and are wondering that it is perfect for you and you do not want to move to a different platform for a routine task. Hold up, let me lay out the numbers then.

With BRIM, file sizes are 28x smaller and speeds are 10x faster!!

Now that is a double whammy. If you use Wireshark and are not disturbed by the strenuous time it takes to open a PCAP file, and it has never really crashed or declined to open a PCAP file, chances are your PCAPs are not large enough, and you may want to skip this one. Nevertheless, with BRIM, you will not ever crash it opening a PCAP file and files open almost instantly. Oh, by the way, BRIM also accepts log files if you work with them.

BRIM

Let us talk about BRIM for better context. BRIM is a network analysis tool much similar to Wireshark. It is open-source and community-managed, hosted at GitHub. It brings us efficiency when it comes to query speeds and file sizes. It is primarily developed in typeScript. It has a neatly built frontend as a cross-platform desktop application design with a modular distributed architecture design for its backend.

The way it works is by leveraging the powers of ZEEK, which is a scripting tool for network monitoring. It has this complex network understanding built-in that helps it to contextualize and analyze networks efficiently. BRIM creates zed lakes on the server-side for data handling and processing, which has Zeek streams of the parsed data, and this is where the querying happens using zqd in zql, a query language specific to it.

Wireshark and BRIM speed demo from their GitHub

Installation

The BRIM GitHub Wiki provides step-by-step instructions on how to download and install BRIM. All of BRIM’s dependencies like zqd, ZAPI are installed alongside.

BRIM interface

ZAPI

Short for zed API, ZAPI can be used as a command-line tool to do pretty much all that BRIM does. Below it can be seen used for scripting. Using ZAPI, one can also automate processes. I have set up my BRIM server on a Debian Linux server and used Python 3.6 for scripting.

The python modules used for this script are shipped with python, but still, they can be pip installed if necessary. Following is the script for the basic functionality needed to get started with ZAPI scripting in python.

This script imports os, subprocess, and the JSON modules. The os module is not used in this script but in a sister script to pass the PCAP file as per the folder and location provided.

The execute_cmd function is responsible for creating a new process using subprocess.popen and communicating with the input and output pipes. This is where the ZAPI command goes and it fetches the output for it. Notice the shell=True in the popen parameters; it is used to inform subprocess that we have to use the local shell to execute this query. This is important in my case because ZAPI is not installed natively in my system. However, I have created a global variable that is accessed using $ZAPI, which has the path to my ZAPI folder.

The rest are the different options provided by the ZAPI command to parse and query PCAP files. Each of the four functions take specific parameters and create a command accordingly. They have example queries mentioned in the form of comments. Notice the $ZAPI in the cmd variable, and this is what I was talking about above.

First, we need to call the create_space function by passing in the space name as a parameter to create a new space for our PCAP file. Then we need to pass this space name and PCAP file to the postpcap function for it to parse this file into the space. The query_zq function takes the space name and the zql query as parameters to execute this query and returns the results in string format without any newline characters and backslashes. Finally, we have the delete_space function to remove our space after our work is done and it takes the space name as a parameter.

Alternatively,

This could also be done using the BRIM python library(however, it is in a very early stage of development), as follows:

Install the module directly from GitHub,

pip install 'git+https://github.com/brimsec/zq#subdirectory=python/zqd'

Use the following script to call the zq module and then create a client to connect to the zq server and finally execute the query.

import zqd as zqd

#space = 'tzng'
space='my_test.pcap'

zql = 'count() by _path | sort -r'

# Create ZQD client instance
c = zqd.Client()  

# Send ZQL query to ZQD
s = c.search(space, zql)

This was a very beginner-friendly article about ZAPI and touched the surface barely. There is a whole lot about the origin of this software, the query language, the different queries, visualizations, integrations with Wireshark and Zeek, and a lot more. I hope I was able to provide some insights about a fantastic tool and it proves helpful to you. You can checkout Brimsec’s GitHub for more.

References

https://reposhub.com/go/logging/brimsec-zq.html
https://github.com/brimdata/zed/blob/main/cmd/zed/README.md#zapi
https://github.com/brimdata/brim/wiki/Installation
https://zeek.org/
https://try.zeek.org/#/?example=hello
https://www.brimsecurity.com/
https://github.com/brimdata
https://github.com/brimdata/brim
https://www.youtube.com/watch?v=InT-7WZ5Y2Y

END.