newfol Manual
=============
:doctype: book

== Using newfol

=== Overview

newfol is a combination user interface and database store for small databases
that can fit into memory.  It is designed to represent textual data (stored and
rendered as UTF-8) and provides tools to search this data using Python regular
expressions.

newfol is not inherently transactional, like some other databases; however, it
provides optional support for storing each saved state into Git to preserve a
complete history of changes.  Along with an SHA-2 hash, this provides immediate
detection of and recovery from potential data loss situations.

== Data Layout

Each newfol database is stored in a directory, which can be specified with the
`--homedir` option.  The default is `~/.newfol`.

Each newfol database contains a `schema` file, which describes the layout of the
data, the methods for importing and exporting it, and other metadata about the
intended behavior of the database.  Refer to <<_schema_file>> for a description
of the contents of this file.  It is the only file which may be hand-edited.

Each database also contains a `dtb` file, which contains the actual data.  This
will be in CSV (colon-separated values), pickle, or JSON format, depending on
the contents of the `version` file.  It may also be compressed.

The `dtb.checksum` file provides a cryptographic SHA-2 hash and length to ensure
that the data has not been accidentally corrupted.  newfol will not open a
database where the database contents do not match this file.

The `lock` file is used to prevent access by multiple processes, which may
accidentally corrupt each other.

=== Schema File

The schema file is a CSV (colon-separated values) file.  The first entry in the
file consist of the values `fmt`, the format version number, and `newfol schema
file`.

newfol automatically upgrades schema files to the latest version, which at the
time of this writing, was 3.  If no schema file exists, newfol will
automatically create a version 0 schema file and upgrade it to the latest
version on start.  newfol does not accept schema files that have a newer version
than it currently supports, but it ignores directives it does not understand,
and so can be manually downgraded by editing the `fmt` directive.

Any record beginning with a `#` is ignored as a comment and preserved on
upgrades.  Extra fields in a record are ignored.

The standard rules for formatting colon-separated values apply.  In other words,
the format is defined as identical to the `text/csv` format specified in RFC
4180, except that the delimiter is a colon instead of a comma and that the line
break is a single linefeed instead of a CRLF pair.

==== Format Directives (`fmt`)

Format directives define the file as a schema file or configuration file as
specified in <<_schema_file>> and <<_configuration_file>>.  They must appear
only at the very beginning of a schema or configuration file.  In particular, it
is not allowed for them to be preceded by a byte order mark.

==== Table Definition Directives (`def`)

A table definition directive consists of a record containing the values `def`
and the table name.  A table must be defined before any field definitions can
occur for that table.

==== Field Definition Directives (`fld` and `dsc`)

Field definitions come in two types.  The first, `fld`, provides a mapping for
imported data, and `dsc` does not.  They are otherwise identical.

A field definition directive consists of a record containing the values `fld` or
`dsc`, the name of the table, the field number for imported data (or empty for
`dsc`), the field number for use in newfol, an optional Python 3 expression, and
a description.

The Python 3 expression is evaluated and placed in the field when creating a new
record.  This can be used, for example, to automatically insert the date into a
particular field.  This requires that code execution has been allowed by an
`exe` directive.

==== Execution Directives (`exe` and `execute`)

Execution directives specify whether Python code will be run from the schema
file.  Such a directive consists of a record containing either `exe` or
`execute` and then either `yes` or `no`.

Unlike other directives, where the schema file overrides the configuration file,
if any execution directive is set to `no`, execution is disabled.  This is a
security feature to allow reading an untrusted database.  Execution is also
disabled if more than one execution directive is seen in either the schema or
configuration file.

==== Key Field Directives (`key`)

Key field directives indicate the fields to display in the _Browse All Records_
view.  By default, this value is 0 (display the first field).

==== Column Directives (`col`)

Column directives indicate how many columns to use to display fields.  A column
directive consists of a record with the values `col` and an integer number of
columns.

==== Hook Directives (`txn`)

Hook directives indicate a series of hooks to be run on each load or save
operation.  Several different hooks are available.  The default set of hooks is
`hash`.  This provides an SHA-2 hash of the data in the `dtb.checksum` file, and
cannot be disabled, only replaced with a specific algorithm.

By default, the hash is `sha256` on 32-bit systems and `sha512` on 64-bit
systems; `sha384` is also available.  If a specific algorithm is specified, it
overrides the `hash` default.  Older versions of newfol only supported the
`sha256` algorithm.

The other available hook is `git`.  This hook performs a commit every time the
file is saved, and a git checkout of the `dtb` file before every load.  This
allows a user to keep a history of the data and to easily back it up to another
location.

A hook directive consists of a record starting with the value `txn`, and
followed by a list of hook names.

=== Configuration File

Configuration files are identical to schema files except that instead of the
text `newfol schema file` in the `fmt` entry, the text `newfol config file` is
substituted.  Because newfol parses both files in the same way, any entry that
is valid in a schema file is also valid in a configuration file.  That said,
some directives make more sense in a configuration file than others.

Common directives that are useful in a configuration file are transaction
settings, colors, and keybindings.

The default location is +_$XDG_CONFIG_HOME_/newfol/defaults+.  If
`$XDG_CONFIG_HOME` is not defined, it defaults to +_$HOME_/.config+.
