Skip to main content

Repositories in GoatDB

tip

Repositories in GoatDB are the fundamental unit of data organization, similar to tables in SQL databases or document collections in NoSQL databases. Each repository manages a collection of related data items and provides synchronized, durable storage with efficient read and write operations.

Core Concepts​

Storage Architecture​

notes.goat
  1. c1
    ideas
  2. c2
    quotes
  3. c3
    reading
  4. c4
    ideas
  5. c5
    quotes
  6. c6
    reading
  7. c7
    ideas
  8. c8
    quotes
One .goat file per repository: a single append-only log of signed commits. Every commit — for every item — lands at the end of the same log. A human-readable .jsonl variant exists for debugging.

Each repository is backed by a single file that stores a log of commits. The default format is .goat (length-prefixed binary), with .jsonl (newline-delimited JSON) available as an option for development and debugging. This design takes advantage of modern SSD characteristics:

  • Sequential I/O: Optimized for SSD performance with sequential writes, enabling efficient batching of operations
  • Hardware Parallelization: Leverages SSD internal parallelization through large sequential I/O operations, enabling multiple NAND chips and controllers to work in parallel
  • Write Amplification: Minimized through append-only design

The storage format can be selected via the storageFormat config option:

  • 'goat' (default): Compact binary with 4-byte length prefix per record — best for production; fast I/O
  • 'jsonl': JSON Lines — one JSON object per line; human-readable, useful for debugging

Commit Graphs​

/data/notes
ideas
c1
c4
c7
quotes
c2
c5
c8
reading
c3
c6
The same log, grouped by key: one independent commit graph per item, all inside the repository. Each item's history evolves in parallel — no interference between items.

Repositories are collections of distinct commit graphs - one per item. Each item (identified by its key) has its own independent commit history, allowing for parallel evolution of different items without interference. This design enables efficient concurrent operations and makes it possible to track the complete history of each item separately.

Commit Structure​

Each commit in the repository log contains:

  • ID: Unique identifier for the commit
  • Key: The item key being modified
  • Data: The actual data being written
  • Parents: References to parent commits (for version history)
  • Timestamp: When the commit was created
  • Metadata: Additional information like organization ID and build version
  • Session: The ID of the session that created the commit
  • Signature: In secure mode (default), a cryptographic signature generated using the session's private key

The signature is particularly important as it provides cryptographic proof that:

  1. The commit was created by an authorized session
  2. The commit data hasn't been tampered with
  3. The commit is part of a verifiable chain of changes

In secure mode (default), operations are cryptographically signed, creating a verifiable commit graph where each change can be traced to its signing session. Actor identity remains application-owned.

note

For performance-critical applications or trusted environments (like backend services), GoatDB offers a trusted mode that disables signing, verification, authorization, and provenance. This mode can significantly improve performance, but it should only be used in controlled environments where security is handled at a different layer.

Basic Operations​

Reading Data​

tip

For more details on reading and writing data, see Reading and Writing Data.

GoatDB provides several ways to read data from repositories:

// Get a specific item by its full path
// Returns the current value of the item or undefined if it doesn't exist
const user = db.item('/users/john/foo');

// Query items using a schema filter
// This example gets all notes in john's repository that match the note schema
const allNotes = db.query({
  source: '/users/john', // Repository path to search in
  schema: kSchemaNote, // Schema to filter by
});

// Get all keys in a repository
// This is useful for listing all items or checking what exists
const allKeys = db.keys('/users/john');

Opening a Repository​

In most cases, repositories are opened automatically when you access an item or perform an operation that requires the repository. However, you can also explicitly open a repository using the db.open() method. This is useful if you want to preload a repository or perform operations that require direct access to the repository instance.

// Explicitly open a repository (returns a Promise that resolves to the Repository instance)
const repo = await db.open('/users/john');

While GoatDB will open repositories on demand, this can introduce a performance penalty the first time you access data in a repository, especially in interactive applications. If you know your application will need a repository (for example, when loading a user profile or switching workspaces), it is recommended to manually open (preload) the repository in advance using db.open(). This ensures the repository is ready for immediate use and avoids delays during user interactions.

note

When preloading a repository, you usually don't need to await the result—just call db.open() and continue. It's safe to call open() multiple times for the same repository; only the first call will actually start loading. If the repository isn't fully loaded by the time you access its data, GoatDB will automatically wait for loading to finish. This approach ensures your application remains responsive and avoids unnecessary delays for users.

Closing a Repository​

Repositories remain open in memory for the duration of your application's session unless you explicitly close them. To manually close a single repository and release its resources, use the db.closeRepo() method:

// Close a repository and flush any pending writes to disk
await db.closeRepo('/chats/chatId');

When you close a repository, GoatDB will first commit any in-memory changes for items in that repository, then flush all pending writes to disk. This ensures that all local edits are saved and durable before the repository is fully released from memory.

note

db.close() (no arguments) closes the entire database, releasing all repositories, sync schedulers, queries, and file handles. Use db.closeRepo(path) to close a single repository without affecting others.

Durability​

GoatDB provides strong durability through:

  1. Atomic Commits: Each commit is written atomically - if a crash occurs mid-write, the half-written commit is simply trimmed from the log
  2. Parallel Writes: Changes are written simultaneously to both local storage and replicated to the server
  3. Automatic Recovery: After a crash, the system automatically recovers missing commits through the synchronization protocol. Recovery depends on the availability of authorized replicas and configured retention policies.
note

Traditional database durability often focuses on server-side guarantees - ensuring data survives server crashes. But in GoatDB, we recognize that client durability is fundamentally different. Modern SSDs in laptops and phones rarely fail, and when they do, it's typically due to physical damage rather than data corruption. More importantly, user expectations differ between client and server operations - if your phone dies mid-click, you wouldn't expect that click's effect to be saved, but when a server acknowledges an API call, you rightfully expect that operation to be durable.

Advanced Usage​

The following methods are low-level APIs typically used for advanced scenarios or debugging:

// Get a repository instance directly
const repo = db.repository('/users/john');

// Low-level repository methods
// Get the current value and commit for a specific key
// Returns [value, commit] tuple or undefined if key doesn't exist
const [value, commit] = repo.valueForKey('john');

// Set a new value for a key with an optional parent commit
// Returns a Commit or undefined if no change
await repo.setValueForKey('john', newItem, parentCommit);

// Get all keys in the repository
// Returns an iterable of all keys
const keys = repo.keys();

// Get all full paths in the repository
// Returns an iterable of paths (repository path + key)
const paths = repo.paths();

// Get the commit graph for a specific key
// Returns an array of CommitGraph objects showing the commit history
const graph = repo.graphForKey('foo');

// Get a Cytoscape-compatible JSON representation of the commit network for a key
// This can be used to visualize the commit history in Cytoscape (https://cytoscape.org/)
const network = repo.debugNetworkForKey('foo');

These low-level APIs are primarily useful for:

  1. Debugging and troubleshooting
  2. Building specialized tools that need direct access to the commit graph
  3. Contributing to GoatDB's core functionality

For normal application development, the higher-level APIs (db.item() and db.create()) provide a safer and more convenient interface. However, if you're interested in contributing to GoatDB's development, these low-level APIs give you direct access to the core functionality.