# Universal Collection Framework

Technical architecture for describing **collections** in LifeLoveMe so the platform can support Ramcharitmanas (RCM), Bhagavad Gita, Yoga Sutra, Bible, Quran, Kamasutra, chanting, Ayurveda/wellness, family tree, Bollywood songs, books, poems, essays, documents, audio, and video—**without rewriting application code for each corpus**. Apps read **metadata and data files**; collection-specific behavior is expressed through **types, manifests, and schemas**, not hard-coded branches for every title.

---

## 1. Purpose

**Ramcharitmanas (RCM)** is the first **proof-of-concept** collection: it validates search, indexing, audio, read-along, dictionary hooks, and Smart Search at production-like depth.

The **Universal Collection Framework** (UCF) is the contract by which **future collections** plug into the same platform: same mental model for identity, files, search, media, relationships, and deployment—**different** content and optional capabilities per `CollectionType`.

This document defines **files, fields, identifiers, and boundaries**. It does not mandate a single implementation language or UI framework; it mandates **discoverable, versionable data** that any LifeLoveMe client can load.

---

## 2. Core principle

**Every collection should be described by metadata and data files, not by hard-coded app logic.**

- **Manifests** declare what exists, how it is indexed, and which optional assets are present.
- **Items** carry universal fields; collection-specific nuance uses **optional fields**, **tags**, and **typed relationships**—not forked code paths for “RCM vs Gita.”
- **Apps** implement **generic behaviors** (list, hub, explore, play, search modes, graph) and **collection profiles** (which search modes are enabled, which media types appear, which Part semantics apply).

Hard-coding is allowed only at the **edges** (bootstrap URLs, feature flags, temporary shims) and should shrink over time as the framework absorbs patterns.

---

## 3. Collection types

`CollectionType` is a coarse classifier for routing, UI defaults, and validation rules.

| `CollectionType`      | Typical content                                      |
|-----------------------|------------------------------------------------------|
| `sacred_text`         | Scripture: verse/chapter structure, meanings, refs |
| `poetic_text`         | Couplets, dohas, sakhī-style verse, light stanza metadata (themes, parallel glosses) |
| `chanting`            | Mantras, stotrams, short repeatable audio/text       |
| `song_catalog`        | Film/non-film songs, metadata, performers          |
| `family_tree`         | People, unions, lineage, dates, private graphs      |
| `book`                | Long-form books, chapters, optional page maps       |
| `document_set`        | Essays, letters, mixed documents                    |
| `media_collection`    | Primarily audio/video with light item metadata      |
| `wellness_content`    | Sleep, stress, yoga, Ayurveda-style **supportive** content (not diagnosis) |

A collection may combine types in **profiles** (e.g. `sacred_text` + `media_collection`); the manifest should declare the **primary** type and **secondary** capabilities.

---

## 4. Standard collection files

Each collection SHOULD ship a consistent **directory of standard files**. Names are conventional; paths are implementation-defined (e.g. `/collections/{CollectionID}/`).

| File                     | Role |
|--------------------------|------|
| `collection_manifest.json` | Identity, type, version, list of bundled files, feature flags, default hub/explorer entrypoints |
| `index_definition.json`    | Which Part fields mean what, sort rules, search field map, LineRef/ItemRef rules for this collection |
| `items.json`               | Canonical list of items (or shard pointers if split) |
| `search_index.json`        | Denormalized or auxiliary structures for fast search (optional if generated at build time) |
| `media_manifest.json`      | Catalog of media assets keyed by `MediaID` |
| `timing.json`              | Alignment of media time ranges to `ItemRef` / `LineRef` (optional; collection-specific density) |
| `dictionary.json`          | Lemmas, glosses, cross-references to `ItemRef` |
| `aliases.json`             | Surface forms → canonical keys / refs for search and Ask routing |
| `relationships.json`       | Edges between items (and optionally external entities) |
| `references.json`          | Bibliography, edition notes, URL-stable citations |
| `help.json`                | User-facing help: how to read refs, search modes, limitations |

**Sharding:** Large collections MAY split `items.json` into `items-000.json`, … with manifest entries listing shards. Search shards MAY mirror the same pattern.

**Omission:** Files that do not apply may be absent; the manifest MUST declare which optional files are **included** vs **absent** so clients do not probe blindly.

---

## 5. Universal item schema

### Required fields

| Field | Description |
|-------|-------------|
| `CollectionID` | Stable string ID for the corpus (e.g. `RCM`, `GITA`) |
| `CollectionType` | One of the types in §3 (primary) |
| `ItemType` | Finer grain: `line`, `verse`, `sutra`, `person`, `song`, `chapter`, etc. |
| `LineRef` **or** `ItemRef` | Canonical machine reference (see §6). Use one as primary per `index_definition.json` |
| `DisplayRef` | Human-facing ref string |
| `SortKey` | Single string or composite key for stable ordering within the collection |
| `Part1Raw` / `Part1Sort` / `Part1Display` | Addressing tier 1 (raw source, sort token, display) |
| `Part2Raw` / `Part2Sort` / `Part2Display` | Addressing tier 2 |
| `Part3Raw` / `Part3Sort` / `Part3Display` | Addressing tier 3 |
| `Part4Raw` / `Part4Sort` / `Part4Display` | Addressing tier 4 (metre code, verse tag, sutra label, etc.) |
| `OriginalText` | Primary source text (Unicode) |
| `MeaningHindi` | Optional but required field **key** present; use empty string if N/A |
| `MeaningEnglish` | Same as above |

**Raw vs Sort vs Display**

- **Raw** — as ingested from editorial source (may include punctuation variants).
- **Sort** — normalized token used in **LineRef** construction and binary ordering (see §6).
- **Display** — pretty label for UI (may include Devanagari labels, book names, etc.).

### Optional fields

| Field | Description |
|-------|-------------|
| `Transliteration` | Roman or other transliteration of `OriginalText` |
| `Summary` | Short abstract |
| `Tags` | Free or controlled vocabulary |
| `Themes` | Thematic buckets for exploration |
| `Characters` | Named entities (IDs or strings; relationships.json preferred for graph) |
| `Places` | Location names or IDs |
| `AudioRefs` | List of `MediaID` or segment IDs |
| `VideoRefs` | List of `MediaID` or segment IDs |
| `ImageRefs` | Illustrations, scans |
| `ReferenceRefs` | Pointers into `references.json` |
| `Notes` | Editorial, provenance, QA |

---

## 6. Universal indexing

### LineRef

**LineRef** identifies a **primary textual line** (or atomic item treated like a line):

```text
LineRef = CollectionID.Part1Sort.Part2Sort.Part3Sort.Part4Sort
```

- **Separator:** dot `.` (URL-safe segment rules per deployment; encode when in query strings).
- **Numeric parts:** use **3-digit zero-padding** in **Sort** segments when the segment is purely numeric (e.g. `4` → `004`, `18` → `018`).
- **Alphanumeric parts:** preserve suffix letters after digits where applicable, e.g. `4A` → `004A`, `208A` stays **`208A`** (already three digit stem + letter; do not strip the letter). Rules are fixed per `index_definition.json` so parsers stay consistent.
- **Part4Sort:** may be a short safe code (metre, verse marker, `Verse`, `Sutra`, `Ch`, etc.); length cap is collection-defined but should stay **identifier-safe** (no spaces; prefer ASCII alphanumerics for maximum portability).

### ItemRef

When a collection needs **sub-line** or **multi-line** packaged items (playlists, chanting cycles, UI cards):

```text
ItemRef = LineRef.ItemNumberSort
```

`ItemNumberSort` is typically a padded ordinal under that line’s scope. If the collection has no sub-items, **ItemRef may be omitted** and `LineRef` is the sole canonical key.

### Examples

| Example | Comment |
|---------|---------|
| `RCM.001.004.002.Do` | RCM-style kand / group / line-in-group / metre |
| `RCM.001.208A.001.Do` | Alphanumeric middle segment preserved |
| `GITA.002.018.000.Verse` | Chapter / verse / filler / type |
| `YOGA.001.002.000.Sutra` | Pada / sutra / pad / type |
| `BIBLE.John.003.016.KJV` | Book name as Part1 sort token per collection profile |
| `BOLLY.1975.SHOLAY.001.Song` | Year / work / track / type |
| `FAMILY.AGARWAL.003.002.Person014` | Tree / branch / generation / person ordinal |

Exact Part semantics are **not global**; they are **declared in `index_definition.json`** per collection so the same engine parses different Part layouts.

---

## 7. RCM mapping

Legacy Ramcharitmanas rows often carry RCM-specific columns. During migration and dual-read periods:

| Legacy / RCM field   | Universal role |
|----------------------|----------------|
| `RCMID`              | Maps to **LegacyRef** (display and backward compatibility) |
| **LineRef**          | New **universal** primary identity (built from Parts) |
| **LineID**           | **Legacy / debug only**; not a cross-collection standard |
| `KandNumber`         | **Part1** (Sort/Display per index_definition) |
| `DohaGroupNumber`    | **Part2** |
| `LineWithinGroup`    | **Part3** |
| `MetreType`          | **Part4** (e.g. `Do`, `Ch`) |

**Migration policy**

- Keep **old RCM fields** alongside universal fields until all consumers read `LineRef` / Parts.
- New collections **must not** depend on `RCMID`, `LineID`, `KandNumber`, etc.; they use **CollectionID + Parts + LineRef/ItemRef** only.
- Apps should prefer **LineRef** for linking, highlighting, and analytics; LineID remains for **timeline continuity** and tooling during transition.

---

## 8. Media schema

Each row in `media_manifest.json` (or equivalent) SHOULD support:

| Field | Description |
|-------|-------------|
| `MediaID` | Stable unique ID within the platform or collection |
| `CollectionID` | Owning collection |
| `MediaType` | e.g. `audio`, `video`, `image`, `subtitle` |
| `Title` | Display title |
| `FileName` | **Original** file name (preserve for traceability) |
| `PublicPath` | URL path when hosted on lean deploy |
| `LocalPath` | Path for full/local builds |
| `SourceURL` | Optional canonical external source (archive, rights holder) |
| `StartItemRef` / `EndItemRef` | Optional span in item space (inclusive semantics per manifest) |
| `DurationSeconds` | Optional |
| `Language` | ISO or manifest enum |
| `Performer` | Generic credit |
| `Narrator` | For audiobooks / recitation |
| `Singer` | For songs |
| `IncludeInDeploy` | Boolean: ship with lean Netlify bundle |
| `DeployTier` | See below |
| `Notes` | Rights, quality, alternate encodings |

### DeployTier

| Value | Meaning |
|-------|---------|
| `beta` | Small, curated subset allowed on lean deploy |
| `full` | Part of full corpus package (e.g. `web_public_large`, local archive) |
| `external` | Not bundled; streamed or downloaded from `SourceURL` / CDN with user consent |
| `local_only` | Never shipped to public CDN; developer or personal mirror |

---

## 9. Timing schema

Rows in `timing.json` (or embedded segments) align media to text:

| Field | Description |
|-------|-------------|
| `MediaID` | Links to `media_manifest.json` |
| `CollectionID` | Redundant but useful for sharded queries |
| `ItemRef` **or** `LineRef` | Target passage |
| `StartTimeText` | Optional human cue (`mm:ss` or SMPTE as policy) |
| `StartSeconds` | Float or decimal seconds from media start |
| `EndSeconds` | Exclusive or inclusive per manifest rule (document the rule once) |
| `TimingSource` | e.g. `manual`, `asr`, `imported_air`, `community` |
| `TimingConfidence` | e.g. `high`, `medium`, `low` or numeric |
| `Notes` | QA, alternate takes, drift correction |

---

## 10. Relationship schema

Edges in `relationships.json`:

| Field | Description |
|-------|-------------|
| `RelationshipID` | Stable unique ID |
| `CollectionID` | Scope (cross-collection edges: policy in manifest; may duplicate rows per side) |
| `SourceItemRef` | Starting node |
| `TargetItemRef` | Ending node |
| `RelationshipType` | Machine enum (see examples) |
| `RelationshipLabel` | Optional human label |
| `StartDate` / `EndDate` | Optional ISO dates (family, historical figures) |
| `Notes` | Evidence, confidence, source citation |

**Examples by domain**

- **Family tree:** `ParentOf`, `SpouseOf`, `SiblingOf`, `ChildOf`, `PartnerOf`
- **RCM / narrative text:** `SpeakerOf`, `ListenerOf`, `Mentions`, `BattleWith`, `Blesses`
- **Songs:** `SungBy`, `WrittenBy`, `ComposedBy`, `FromMovie`, `FromAlbum`

Relationship types SHOULD be **registered** in `index_definition.json` or `collection_manifest.json` so UIs can offer filters without code forks per type string.

---

## 11. Search framework

Search is implemented as **modes** over the same item corpus (and optional indices):

| Mode | Role |
|------|------|
| **Smart Search** | Intent parsing: references, dialog patterns, multi-term logic—collection profile defines capabilities |
| **Text Search** | Match on `OriginalText`, transliteration, titles |
| **Starts Search** | Prefix / first-token style matching (performance-sensitive) |
| **Index Search** | Direct lookup by `LineRef`, `ItemRef`, or Part patterns |
| **Meaning Search** | Match on `MeaningHindi`, `MeaningEnglish`, summaries |
| **Boolean search** | AND / OR / NOT with documented precedence |
| **Wildcard search** | Glob or regex subset per security policy |
| **Aliases** | Expand surface forms via `aliases.json` |
| **Future concept search** | Dictionary / ontology → item sets (not required for v1) |

### Default token rule (conventional)

- **No wildcard:** tokens match as **whole words** (word boundaries defined per language in `index_definition.json`).
- **`*` wildcard:** allows **partial word** / substring behavior where enabled (e.g. `ram*`).

Exact tokenization (Devanagari, ZWJ, chandrabindu) is **locale-specific** and must be specified per collection in the index definition—not hard-coded globally in app logic.

---

## 12. Ask LifeLoveMe routing layer

**Ask LifeLoveMe** is a global **intent routing** layer (box on home, plus APIs). It classifies utterances into **intents** and routes to the right surface:

| Intent (example enum) | Typical destination |
|------------------------|----------------------|
| `learn` | Hub, tutorials, primer content |
| `search` | Explorer with query pre-filled |
| `meaning` | Item panel / meaning fields / dictionary |
| `play` | Player + `MediaID` or search-first then play |
| `read` | Reader / Explorer read mode |
| `explain` | Grounded explanation with citations where required |
| `wellness_sleep` | Wellness collection, sleep-safe copy |
| `wellness_stress` | Wellness collection + safety escalation templates |
| `reference` | Reference / Help / `references.json` |
| `relationship` | Graph app or filtered relationship list |
| `navigate` | Library, settings, collection picker |

Routing SHOULD be **data-driven** (intent patterns, aliases) with **collection hooks** (e.g. “Gita 2.18” → `GITA` index profile). Fallback: honest **“not in library yet”** plus suggested collections.

---

## 13. App architecture

Logical apps (may be routes in one SPA or separate deployables):

| Surface | Function |
|---------|----------|
| **LifeLoveMe home** | Ask LifeLoveMe + main entrance, announcements |
| **Library** | Lists all collections from manifests |
| **Hub** | First page for **one** collection: overview, samples, links to Explorer/Player |
| **Explorer** | Search, read, play, study; heavy UX for one collection |
| **Player** | Focused media / read-along; minimal chrome |
| **Reference / Help** | Sources, indexing method, LineRef rules, search help |
| **Graph** | Relationship visualization (family tree, character graph) |

Clients load **`collection_manifest.json`** first, then lazy-load shards (`items`, `search_index`, `media_manifest`, etc.) according to flags.

---

## 14. Netlify deployment rules

- **`deploy/web_public`** remains **lean**: HTML/JS/CSS, small JSON, **beta-tier** media only.
- **`deploy/web_public_large`** (or equivalent) holds **full local / archive** builds: large media, full timings, bulk JSON.
- **Preserve original filenames** in manifests and on disk where possible.
- **Do not** copy full media archives into Netlify-sized deploys by default.
- Use **`IncludeInDeploy`** and **`DeployTier`** on every media row to drive build scripts.
- **Copy only** selected **beta** assets to lean deploy; everything else references `external` or `local_only` until policy changes.

Build pipelines SHOULD fail loudly if a lean deploy exceeds size budgets or if forbidden tiers are included.

---

## 15. Migration strategy

Staged rollout to avoid breaking the working RCM beta:

1. **Keep RCM working** — no regressions on search, play, read-along.
2. **Add universal fields** beside legacy RCM fields (dual-write / dual-read as needed).
3. **Set `CollectionID=RCM`** in manifest and items; treat RCM as first-class UCF collection.
4. **Create generic collection library** UI fed only by manifests (RCM entry first).
5. **Create collection explorer wrapper** that configures Explorer from manifest + index_definition (RCM becomes one profile).
6. **Add small Gita sample** — stress-test Parts and meaning search without full corpus.
7. **Add chanting sample** — short loop, media + timing smoke test.
8. **Add family tree prototype** — small `relationships.json`, Graph surface, privacy model.
9. **Gradually retire RCM-only assumptions** in code (replace with manifest-driven configuration).

---

## 16. Development guardrails

- **Do not break** the working **RCM beta**; ship incremental changes behind flags when risky.
- **Do not edit `deploy/web_public` directly** as a source of truth; promote builds from **`web/apps`** (or documented source tree) through an automated or repeatable copy step.
- **Validate** (lint, JSON schema, sample queries, timing spot-checks) **before** deploy.
- Maintain **source vs deploy folder discipline** (canonical code and data in `web/` / `data/`; deploy is output).
- Add **checkpoints** (tags, branches, or dated bundles) for **major** schema or search-behavior changes.

---

*Document status: architecture spec. Revise when the first non-RCM collection ships or when LineRef rules are extended for new Part layouts.*