newfol Manual ============= :doctype: book == Using newfol === Overview newfol is a combination user interface and database store for small databases that can fit into memory. It is designed to represent textual data (stored and rendered as UTF-8) and provides tools to search this data using Python regular expressions. newfol is not inherently transactional, like some other databases; however, it provides optional support for storing each saved state into Git to preserve a complete history of changes. Along with an SHA-2 hash, this provides immediate detection of and recovery from potential data loss situations. == Data Layout Each newfol database is stored in a directory, which can be specified with the `--homedir` option. The default is `~/.newfol`. Each newfol database contains a `schema` file, which describes the layout of the data, the methods for importing and exporting it, and other metadata about the intended behavior of the database. Refer to <<_schema_file>> for a description of the contents of this file. It is the only file which may be hand-edited. Each database also contains a `dtb` file, which contains the actual data. This will be in CSV (colon-separated values), pickle, or JSON format, depending on the contents of the `version` file. It may also be compressed. The `dtb.checksum` file provides a cryptographic SHA-2 hash and length to ensure that the data has not been accidentally corrupted. newfol will not open a database where the database contents do not match this file. The `lock` file is used to prevent access by multiple processes, which may accidentally corrupt each other. === Schema File The schema file is a CSV (colon-separated values) file. The first entry in the file consist of the values `fmt`, the format version number, and `newfol schema file`. newfol automatically upgrades schema files to the latest version, which at the time of this writing, was 3. If no schema file exists, newfol will automatically create a version 0 schema file and upgrade it to the latest version on start. newfol does not accept schema files that have a newer version than it currently supports, but it ignores directives it does not understand, and so can be manually downgraded by editing the `fmt` directive. Any record beginning with a `#` is ignored as a comment and preserved on upgrades. Extra fields in a record are ignored. The standard rules for formatting colon-separated values apply. In other words, the format is defined as identical to the `text/csv` format specified in RFC 4180, except that the delimiter is a colon instead of a comma and that the line break is a single linefeed instead of a CRLF pair. ==== Format Directives (`fmt`) Format directives define the file as a schema file or configuration file as specified in <<_schema_file>> and <<_configuration_file>>. They must appear only at the very beginning of a schema or configuration file. In particular, it is not allowed for them to be preceded by a byte order mark. ==== Table Definition Directives (`def`) A table definition directive consists of a record containing the values `def` and the table name. A table must be defined before any field definitions can occur for that table. ==== Field Definition Directives (`fld` and `dsc`) Field definitions come in two types. The first, `fld`, provides a mapping for imported data, and `dsc` does not. They are otherwise identical. A field definition directive consists of a record containing the values `fld` or `dsc`, the name of the table, the field number for imported data (or empty for `dsc`), the field number for use in newfol, an optional Python 3 expression, and a description. The Python 3 expression is evaluated and placed in the field when creating a new record. This can be used, for example, to automatically insert the date into a particular field. This requires that code execution has been allowed by an `exe` directive. ==== Execution Directives (`exe` and `execute`) Execution directives specify whether Python code will be run from the schema file. Such a directive consists of a record containing either `exe` or `execute` and then either `yes` or `no`. Unlike other directives, where the schema file overrides the configuration file, if any execution directive is set to `no`, execution is disabled. This is a security feature to allow reading an untrusted database. Execution is also disabled if more than one execution directive is seen in either the schema or configuration file. ==== Key Field Directives (`key`) Key field directives indicate the fields to display in the _Browse All Records_ view. By default, this value is 0 (display the first field). ==== Column Directives (`col`) Column directives indicate how many columns to use to display fields. A column directive consists of a record with the values `col` and an integer number of columns. ==== Hook Directives (`txn`) Hook directives indicate a series of hooks to be run on each load or save operation. Several different hooks are available. The default set of hooks is `hash`. This provides an SHA-2 hash of the data in the `dtb.checksum` file, and cannot be disabled, only replaced with a specific algorithm. By default, the hash is `sha256` on 32-bit systems and `sha512` on 64-bit systems; `sha384` is also available. If a specific algorithm is specified, it overrides the `hash` default. Older versions of newfol only supported the `sha256` algorithm. The other available hook is `git`. This hook performs a commit every time the file is saved, and a git checkout of the `dtb` file before every load. This allows a user to keep a history of the data and to easily back it up to another location. A hook directive consists of a record starting with the value `txn`, and followed by a list of hook names. === Configuration File Configuration files are identical to schema files except that instead of the text `newfol schema file` in the `fmt` entry, the text `newfol config file` is substituted. Because newfol parses both files in the same way, any entry that is valid in a schema file is also valid in a configuration file. That said, some directives make more sense in a configuration file than others. Common directives that are useful in a configuration file are transaction settings, colors, and keybindings. The default location is +_$XDG_CONFIG_HOME_/newfol/defaults+. If `$XDG_CONFIG_HOME` is not defined, it defaults to +_$HOME_/.config+.