---
title: "Custom Pattern Search with dig()"
output: rmarkdown::html_vignette
vignette: >
  %\VignetteIndexEntry{Custom Pattern Search with dig()}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

```{r, include = FALSE}
knitr::opts_chunk$set(
    collapse = TRUE,
    comment = "#>"
)
```

```{r, include = FALSE}
options(tibble.width = Inf)
```



# Introduction

`dig()` is the general function behind custom pattern search in the `nuggets`
package. It searches for patterns of a custom type by generating conditions as
elementary conjunctions of predicates and by executing a user-defined callback
function on each generated condition.

This makes `dig()` the low-level building block behind more specialized
functions such as `dig_associations()` and `dig_correlations()`. Use it when
you want to keep the search over conditions, but define your own evaluation
logic.

This vignette focuses on how to use the `dig()` function: how to write the
callback function and how to control the search. For preparation of crisp and
fuzzy predicates, see `vignette("data-preparation")`.

Examples in this vignette require loading the following packages:

```{r, message = FALSE}
library(nuggets)
library(dplyr)     # for data manipulation
```



# A Small Working Dataset

`dig()` expects a matrix or data frame whose columns are logical predicates or
numeric fuzzy predicates. We will use `iris` and prepare a small crisp predicate
dataset that is rich enough for the first examples below:

```{r}
crisp_iris <- iris |>
    partition(Species) |>
    partition(Sepal.Length:Petal.Width, .method = "crisp", .breaks = 3)

head(crisp_iris, n = 3)
```

The commands above create crisp predicates for the four numeric columns of
`iris`, plus the three species predicates. The preparation step is intentionally
brief here; the dedicated `vignette("data-preparation")` explains `partition()`,
fuzzy predicates, and breakpoints in detail.



# A Simple `dig()` Call

The `dig()` function generates conditions from the selected predicates in
a recursive manner. It starts with the empty condition and adds one predicate at
a time, up to the specified `max_length`. Meanwhile, it evaluates the generated
condition and tests whether it meets the minimum support requirement. By support
we mean the relative frequency of rows satisfying the condition. If the condition
is frequent enough, `dig()` calls the user-defined callback function with the
generated condition and other information. The callback can then compute any 
output you want. The `dig()` function collects those outputs and returns them as
a list.

The simplest callback function can handle just the generated condition.
In the following example, the callback generates some debug output and
returns the formatted condition:

```{r}
simple_callback <- function(condition) {
    str(condition)
    cat("------\n")
    
    list(condition = format_condition(names(condition)))
}

simple_result <- dig(x = crisp_iris,
                     f = simple_callback,
                     condition = starts_with("Sepal"),
                     min_length = 0,
                     max_length = 2,
                     min_support = 0.2)
```

As you can see from the debug output issued by the `str()` call within the
callback, `dig()` enumerates all conditions that can be formed from the selected
predicates  (in this case, all predicates starting with "Sepal") and that meet
the minimum support requirement. The callback receives each condition in the
form of a named integer vector, where the names  are the predicate names and the
values are the column indices in the original data frame. The callback then
returns a named list with the formatted condition. All callback results are
collected into a list and returned by `dig()`:

```{r}
str(simple_result)
```

As you can see, the result is a list of named lists, one for each condition that
was generated and passed to the callback. You can flatten the result into
a tibble with `dplyr`'s `bind_rows()`:

```{r}
bind_rows(simple_result)
```

Note also the attributes of the result list. They contain information about the
search, such as the search statistics and the arguments that were passed to
`dig()`. You can use this information for debugging or for reproducing the search
later. For example, you can obtain the vector of predicate names that were used
to generate conditions with:

```{r}
attributes(simple_result)$call_args$condition
```

This simple example illustrates the basic workflow:

1. choose which predicates may form conditions;
2. set the search parameters (length, support, etc.);
3. define a callback function that computes the desired output for each condition;
4. let `dig()` enumerate conditions and collect callback results.



# Condition and Focus

The simple example above only used the predicates for generating conditions.
In many cases, you will also want to evaluate other predicates within each
generated condition. For example, you may want to know how often each species
occurs within a condition. That's where the *foci* (plural of *focus*) come into 
play.

Condition and focus predicates are selected separately with the `condition` and `
focus` arguments of `dig()`. These arguments accept
[tidyselect expressions](https://tidyselect.r-lib.org/reference/language.html)
for selecting columns of the input data frame `x`.


## Using Foci

Foci are predicates that are not used to generate conditions, but are evaluated
within each generated condition. You can select foci with the `focus` argument
of `dig()`. The callback function can then receive information about how often
each focus occurs within the generated condition. This is useful, e.g., for
finding conditions that are strongly associated with certain foci.

For instance, let us define a callback that provides the number of occurences
of each species within each generated condition:

```{r}
focus_callback <- function(condition, sum, pp) { 
    str(list(condition = condition,
             sum = sum,
             species = pp))
    cat("------\n")

    NULL
}

focus_result <- dig(x = crisp_iris,
                    f = focus_callback,
                    condition = starts_with("Sepal"),
                    focus = starts_with("Species"),
                    min_length = 2,
                    max_length = 2,
                    max_results = 1)
```

We have defined a callback that, besides `condition`, also receives `sum`  and 
`pp`. The `sum` argument provides the number of rows satisfying the generated
condition, and the `pp` argument contains the number of rows that satisfy both
the generated condition and each focus predicate. In this case, the focus
predicates are the species predicates, so `pp` tells us how many rows of each
species satisfy the generated condition.

Also note that we are using a neat trick that is useful during development of
the callback: we set `max_results = 1` to stop the search after the first
condition that meets the criteria. This allows us to see the output of the
callback without waiting for the entire search to complete, which can be
time-consuming for large datasets or complex conditions.

So far, our callback only prints the information to the console and returns 
`NULL`. Once we understand the structure of the data we receive, we can modify 
the callback to return a list of patterns, where each pattern contains the 
formatted condition, a single species, the count of data rows satisfying the
condition, and the count of the species within that condition:

```{r}
focus_callback <- function(condition, sum, pp) { 
    species_names <- names(pp)
    species_counts <- as.integer(pp)

    lapply(seq_along(species_names), function(i) {
        list(condition = format_condition(names(condition)),
             species = species_names[i],
             condition_count = sum,
             species_count = species_counts[i])
    })
}
    
focus_result <- dig(x = crisp_iris,
                    f = focus_callback,
                    condition = starts_with("Sepal"),
                    focus = starts_with("Species"),
                    min_length = 0,
                    max_length = 2) 
```

The result of `dig()` is a list of lists, where each inner list corresponds to 
a species within a generated condition. We use `unlist(recursive = FALSE)` to 
flatten the list of lists into a single list of patterns, and then `bind_rows()` 
to convert it into a tibble for easier viewing:

```{r}
focus_result |>
    unlist(recursive = FALSE) |>
    bind_rows() |>
    head(n = 6)
```


## Filtering Foci

As discussed in the previous section, `dig()` evaluates the focus predicates 
within each generated condition. You may or may not want to keep all foci for 
each condition. Some patterns may require all foci to be evaluated every time,
while others may only require a subset of foci to be considered that are
sufficiently frequent within the condition. Therefore, `dig()` provides several 
arguments to filter foci based on their support:

- `min_focus_support`: minimum support of a focus within a condition. I.e., the 
  relative frequency of rows satisfying both the condition and the focus must be 
  at least this value for the focus to be kept. Foci with support below this
  threshold are filtered out.
- `min_conditional_focus_support`: minimum conditional support of a focus within
  a condition. I.e., the relative frequency of rows satisfying both the condition
  and the focus, divided by the number of rows satisfying the condition, must be
  at least this value for the focus to be kept. Foci with conditional support
  below this threshold are filtered out.
  
The focus filtering may result in some conditions having no remaining foci.
If you want to skip the callback for such conditions, set `filter_empty_foci = TRUE`. 
Otherwise, the callback will be called with empty focus information. Filtering
empty foci also improves performance, because it also stops early the evaluation
of longer conditions that would not have any remaining foci anyway.

The following example shows how this can be used to emulate association-rule
search with a custom callback. Association rules are implications of the form
"if condition then focus". Condition is named the *antecedent* and focus is named
the *consequent*. The callback computes the confidence of each 
antecedent-consequent pair, filters the pairs based on minimum support and
confidence, and returns a list of rules:

```{r}
min_support <- 0.1
min_confidence <- 0.8

rule_callback <- function(condition, pp, support) {
    conf <- pp / support / nrow(crisp_iris)
    sel <- !is.na(conf) & conf >= min_confidence & !is.na(pp) & pp >= min_support
    conf <- conf[sel]
    supp <- pp[sel] / nrow(crisp_iris)

    lapply(seq_along(conf), function(i) {
        list(antecedent = format_condition(names(condition)),
             consequent = names(conf)[[i]],
             antecedent_support = support,
             rule_support = supp[[i]],
             confidence = conf[[i]]
        )
    })
}

rule_result <- dig(x = crisp_iris,
                   f = rule_callback,
                   condition = !starts_with("Species"),
                   focus = starts_with("Species"),
                   min_length = 1,
                   min_support = min_support,
                   min_focus_support = min_support,
                   min_conditional_focus_support = min_confidence,
                   filter_empty_foci = TRUE) |>
    unlist(recursive = FALSE) |>
    bind_rows() |>
    arrange(desc(confidence))

head(rule_result, n = 6)
```

This is a useful illustration of focus filtering, but association rules already
have a dedicated implementation: `dig_associations()` searches for them more
efficiently. For that purpose, prefer `dig_associations()` and see
`vignette("association-rules")`.




# What the Callback Function Can Receive

As seen in the previous section, the callback function `f` may obtain not only
the generated condition, but also other information. The amount of received
information is controlled by declaring the arguments of the callback function. `dig()`
inspects the callback function argument names and computes only the requested
values. This is important for performance, because some values are expensive to
compute and may not be needed for every search.

The callback function may declare any subset of the following arguments:

- `condition`: named integer vector of column indices representing the generated
  condition.
- `sum`: number of rows satisfying the condition for logical data, or the sum
  of truth degrees for fuzzy data.
- `support`: relative frequency of the condition, i.e., `sum / nrow(x)`.
- `indices`: row indices of the original dataset `x` satisfying the condition
  in crisp searches or the indices of rows with non-zero truth degrees in fuzzy
  searches.
- `weights`: per-row truth degrees of the condition in dataset `x`; logical
  (crisp) data is treated as 0/1 weights.
- `pp`, `pn`, `np`, `nn`: contingency-table entries for foci. The *i*-th* entry 
  of each vector corresponds to the *i*-th focus predicate. The
  entries are defined as follows:
  - `pp`: sum of truth degrees of rows satisfying both the condition and the focus
    (**p**ositive condition, **p**ositive focus),
  - `pn`: sum of truth degrees of rows satisfying the condition but not the focus,
    (**p**ositive condition, **n**egative focus),
  - `np`: sum of truth degrees of rows satisfying the focus but not the condition,
    (**n**egative condition, **p**ositive focus),
  - `nn`: sum of truth degrees of rows satisfying neither (**n**egative condition,
    **n**egative focus).
    
In practice:

- use `condition` when you need condition predicate names,
- use `support` or `sum` for condition-level filtering or ranking,
- use `indices` when you want to compute something on the original rows,
- use `weights` for custom fuzzy summaries,
- use `pp`, `pn`, `np`, and `nn` when your pattern depends on foci and their
  frequency within the condition.
  
Note: only declare the arguments you need. For example, if you don't need foci,
don't declare `pp`, `pn`, `np`, or `nn`. This will save computation time,
especially for large datasets or complex conditions. The most expensive values
to compute are `indices` and `weights`. They require scanning the entire dataset
for each generated condition. So avoid them if not needed.



# Advanced Examples


## Example: Fixed-Variable Correlations

`dig_correlations()` searches over both generated conditions and combinations of
numeric variables. A simpler custom variant can be built directly with `dig()`
when the two variables are fixed in advance and only the condition should vary.

Here we search for conditions under which `Sepal.Length` and `Petal.Length`
correlate strongly. The callback receives `indices`, uses them to select the
corresponding rows from the original `iris` data, and runs `cor.test()` on that
sub-data:

```{r}
correlation_callback <- function(condition, support, indices) {
    if (length(indices) < 10) {
        return(NULL)
    }
    fit <- cor.test(iris$Sepal.Length[indices],
                    iris$Petal.Length[indices],
                    method = "pearson")

    list(condition = format_condition(names(condition)),
         support = support,
         correlation = unname(fit$estimate),
         p_value = fit$p.value,
         n = length(indices))
}

correlation_result <- dig(x = crisp_iris,
                          f = correlation_callback,
                          condition = everything(),
                          min_length = 1,
                          max_length = 2,
                          min_support = 0.1) |>
    bind_rows() |>
    arrange(desc(abs(correlation)))

head(correlation_result, n = 6)
```

This example follows the same idea as `dig_correlations()`:

- generate conditions,
- evaluate a statistic on the sub-data induced by each condition,
- return one row per successful evaluation.

The difference is that `dig()` leaves the statistic entirely in your hands.
That is useful when you want to fix the variables, apply a custom test, return
additional diagnostics, or combine several criteria in one callback.



## Example: Handling Fuzzy Data

For fuzzy searches, conditions are no longer simply satisfied or not satisfied.
Instead, each row has a truth degree in the interval $[0,1]$. In that setting,
`indices` and `weights` play different roles:

- `indices` tell you which rows have a non-zero truth degree for the condition,
- `weights` tell you on scale $[0, 1]$ how strongly each row satisfies the condition.

The following example prepares fuzzy predicates from `iris` and then compares
an unweighted summary based on `indices` with a weighted summary based on
`weights`:

```{r}
fuzzy_iris <- iris |>
    partition(Species) |>
    partition(Sepal.Length:Petal.Width, .method = "triangle", .breaks = 3)

head(fuzzy_iris, n = 3)

fuzzy_callback <- function(condition, indices, weights) {
    if (length(indices) < 20) {
        return(NULL)
    }

    list(condition = format_condition(names(condition)),
         nonzero_rows = sum(indices),
         weighted_support = sum(weights) / nrow(fuzzy_iris),
         mean_petal_length_by_indices = mean(iris$Petal.Length[indices]),
         mean_petal_length_by_weights = weighted.mean(iris$Petal.Length, weights))
}

fuzzy_result <- dig(x = fuzzy_iris,
                    f = fuzzy_callback,
                    condition = starts_with("Sepal"),
                    min_length = 1,
                    max_length = 1,
                    min_support = 0.2) |>
    bind_rows()

fuzzy_result
```

The unweighted mean based on `indices` treats all rows with non-zero membership
equally. The weighted mean based on `weights` respects the fuzzy truth degrees,
so rows that satisfy the condition more strongly contribute more. This is the
main practical difference: `indices` identify the relevant rows, while
`weights` quantify the strength of their membership.



# Practical Notes

- Condition predicates are used to generate conditions, while focus predicates 
  are evaluated within each generated condition. Use `condition` and `focus`
  arguments to select them separately.
- The result of `dig()` is a list. When each callback returns one named list,
  `bind_rows()` is a convenient way to flatten it into a tibble.
- If the callback returns multiple patterns per condition, return a list of
  named lists and flatten the result afterwards.
- Use only the arguments you need in the callback function. This saves 
  computation time, especially for large datasets or complex conditions.
- For fuzzy searches, use `weights` when truth degrees matter, and `indices`
  when you only need the rows with non-zero membership.



# Summary

`dig()` is the most flexible search interface in `nuggets`. It lets you:

1. generate conditions from selected predicates,
2. optionally evaluate focus predicates within each condition,
3. receive only the callback inputs you need,
4. define your own pattern logic, statistics, and output format.

Use `dig()` when the built-in search functions are close to what you need, but
not exact. For related material, see:

- `vignette("data-preparation")` for creating crisp and fuzzy predicates from
  raw data,
- `vignette("association-rules")` for searching for association rules with
  `dig_associations()`,  
- `vignette("nuggets")` for an overview of the package and its main workflows.
