# rare

[![GitHub Workflow Status](https://img.shields.io/github/actions/workflow/status/zix99/rare/main.yml?branch=master)](https://github.com/zix99/rare/actions)
[![GitHub release (latest by date)](https://img.shields.io/github/v/release/zix99/rare)](https://github.com/zix99/rare/releases)
[![codecov](https://codecov.io/gh/zix99/rare/branch/master/graph/badge.svg)](https://codecov.io/gh/zix99/rare)
![GitHub all releases](https://img.shields.io/github/downloads/zix99/rare/total)
![GitHub](https://img.shields.io/github/license/zix99/rare)

A fast text scanner/regex extractor and realtime summarizer. Quickly search, reformat and visualize text files
such as logs, csv, json, etc.

Supports various CLI-based graphing and metric formats (filter (grep-like), histogram, table, bargraph, heatmap, reduce).

`rare` is a play on "more" and "less", but can also stand for "realtime aggregated regular expressions".

![rare gif](images/rare.gif)

## Features

 * Multiple [summary formats](usage/aggregators.md) including: filter (like grep), histogram, bar graphs, tables, heatmaps, sparklines, reduce, and numerical analysis
 * Parse using [regex](usage/regexp.md) (`-m`) or [dissect tokenizer](usage/dissect.md) (`-d`)
 * Manipulate results with handlebars-like [expressions](usage/expressions.md) (`-e`)
 * File [glob expansions](usage/input.md#read-files) (eg `/var/log/*` or `/var/log/*/*.log`) and recursion `-R` with [path filters](usage/input.md#path-filters) (include/exclude patterns)
 * Optional [gzip decompression](usage/input.md#gzip) (with `-z`)
 * [Following](usage/input.md#following-files) `-f` or re-open following `-F` (use `--poll` to poll, and `--tail` to tail)
 * [Ignoring](usage/extractor.md#ignore) lines that match an expression (with `-i`)
 * Aggregating and realtime summary (Don't have to wait for all data to be scanned)
 * Multi-threaded reading, parsing, and aggregation ([It's fast](benchmarks.md))
 * Color-coded outputs (optionally)
 * [Pipe support](usage/input.md#stdinpipe) (stdin for reading, stdout will disable realtime, and `--csv` formatting) eg. `tail -f | rare ... > out`

Take a look at [examples](usage/examples.md) to see more of what *rare* does.

## Installation

!!! note "A Note on PCRE (Perl Compatible Regular Expressions)"

    Besides your standard OS versions, there is an additional `pcre` build which is ~4x faster than go's `re2` implementation in moderately complex cases.  In order to use this, you must make sure that **libpcre2** is installed (eg `apt install libpcre2-8-0` or `dnf install pcre2`).  Right now, it is only bundled with the linux distribution.

    PCRE2 also comes with pitfalls, two of the most important are:
    1. That *rare* is now dynamically linked, meaning that you need to have libc and libpcre installed
    2. That pcre is an exponential-time algorithm (re2 is linear).  While it can be significantly faster than go's `re2`, it can also be catastrophically slower in some situations. There is a good post [here](https://swtch.com/~rsc/regexp/regexp1.html) that talks about regexp timings.

    I will leave it up to the user as to which they find suitable to use for their situation.  Generally, if you know what *rare* is getting as an input, the pcre version is perfectly safe and can be much faster.

### Bash Script

!!! warning "Warning: Verify Script"
    As a best practice, you should always verify any [script](https://github.com/zix99/rare/blob/master/docs/install.sh) before piping to bash.

This script downloads the latest version from github, and installs it to `~/.local/bin` (`/usr/bin` if *root*).

```sh
curl -sfL https://rare.zdyn.net/install.sh | bash
```

### Manual (Prebuilt Binary)

Download appropriate binary or package from [Releases](https://github.com/zix99/rare/releases)

### Homebrew

```sh
brew tap zix99/rare
brew install rare
```

### Community Contributed

!!! note
    The below install methods have been contributed by the community, and aren't maintained directly.

#### MacPorts

```sh
sudo port selfupdate
sudo port install rare
```

### From code

Clone the repo, and:

Requires GO 1.23 or higher

```sh
go mod download

# Build binary
go build .

# OR, with experimental features
go build -tags experimental .
```

Available tags:

* `experimental` Enable experimental features (eg. fuzzy search)
* `pcre2` Enables PCRE 2 (v10) where able. Currently linux only
* `rare_no_pprof` Disables profiling capabilities, which reduces binary size
* `urfave_cli_no_docs` Disables man and markdown documentation generation, which reduces binary size


## Quickstart

The easiest way to start using *rare* is by creating a `histogram`.

Each execution is usually composed of two parts, the [regex extracted match](usage/extractor.md) (`-m`) and the [expression](usage/expressions.md) (`-e`).

```sh
$ rare histo \
    -m '"(\w{3,4}) ([A-Za-z0-9/.]+).*" (\d{3})' \ # The regex that extracts match-groups
    -e '{3} {1}' \ # The expression will be the key, referencing the match-groups
    access.log     # One or more files (or -R for recursion)

200 GET                          160663
404 GET                          857
304 GET                          53
200 HEAD                         18
403 GET                          14
```

## Next

To learn more, check out the [aggregators](usage/aggregators.md), read some [examples](usage/examples.md) or dig into the [overview](usage/overview.md).
