Add temporality awareness to openrag responses and chunk creation

.hydra_config/config.yaml

@@ -51,6 +51,9 @@ reranker:
   model_name: ${oc.env:RERANKER_MODEL, Alibaba-NLP/gte-multilingual-reranker-base}
   top_k: ${oc.decode:${oc.env:RERANKER_TOP_K, 5}} # Number of documents to return after reranking. upgrade to 8 for better results if your llm has a wider context window
   base_url: ${oc.env:RERANKER_BASE_URL, http://reranker:${oc.env:RERANKER_PORT, 7997}}
+  # Temporal scoring parameters
+  temporal_weight: ${oc.decode:${oc.env:RERANKER_TEMPORAL_WEIGHT, 0.3}}  # Weight for temporal scoring (0.0-1.0), 0.3 means 30% temporal, 70% relevance
+  temporal_decay_days: ${oc.decode:${oc.env:RERANKER_TEMPORAL_DECAY_DAYS, 365}}  # Days for temporal score to decay to near zero
 
 map_reduce:
   # Number of documents to process in the initial mapping phase

docker-compose.yaml

@@ -4,8 +4,7 @@ include:
   - extern/infinity.yaml
 
 x-openrag: &openrag_template
-  image: ghcr.io/linagora/openrag:dev-latest
-  # image: linagoraai/openrag:latest
+  image: linagoraai/openrag:latest
   build:
     context: .
     dockerfile: Dockerfile
@@ -58,7 +57,7 @@ x-vllm: &vllm_template
 services:
   # OpenRAG Indexer UI
   indexer-ui:
-    image: linagoraai/indexer-ui:v1.1
+    image: linagoraai/indexer-ui:latest
     build:
       context: ./extern/indexer-ui
       dockerfile: Dockerfile

docs/content/docs/documentation/API.mdx

@@ -219,10 +219,45 @@ curl -X POST http://localhost:8080/v1/chat/completions \
       }
     ],
     "temperature": 0.7,
-    "stream": false
+    "stream": false,
+    "metadata": {
+      "use_map_reduce": false,
+      "temporal_filter": {
+        "created_after": "2024-01-01T00:00:00Z",
+        "created_before": "2024-12-31T23:59:59Z"
+      }
+    }
   }'
 ```
 
+**Temporal Filtering:**
+
+OpenRAG supports temporal filtering to retrieve documents from specific time periods. You can include a `temporal_filter` object in the `metadata` field:
+
+- **Automatic extraction**: If no `temporal_filter` is provided, OpenRAG automatically extracts temporal expressions from your query (e.g., "documents from 2024", "last week's updates")
+- **Manual filtering**: Explicitly specify date ranges using the following fields:
+  - `datetime_after` / `datetime_before`: Filter by document content datetime (highest priority)
+  - `modified_after` / `modified_before`: Filter by document modification date
+  - `created_after` / `created_before`: Filter by document creation date
+  - `indexed_after` / `indexed_before`: Filter by document indexing date
+
+All dates should be in ISO 8601 format (e.g., `2024-01-15T00:00:00Z`).
+
+**Example with temporal filter:**
+```python
+# Query documents from 2024 only
+response = client.chat.completions.create(
+    model="openrag-my_partition",
+    messages=[{"role": "user", "content": "What are the latest updates?"}],
+    metadata={
+        "temporal_filter": {
+            "created_after": "2024-01-01T00:00:00Z",
+            "created_before": "2024-12-31T23:59:59Z"
+        }
+    }
+)
+```
+
 * Text Completions
 ```http
 POST /v1/completions
@@ -263,6 +298,51 @@ response = client.chat.completions.create(
 )
 ```
 
+#### Example with Temporal Filtering
+
+```python
+from openai import OpenAI
+from datetime import datetime, timedelta, timezone
+
+client = OpenAI(api_key='your-auth-token', base_url="http://localhost:8080/v1")
+
+# Example 1: Query recent documents (last 30 days)
+thirty_days_ago = (datetime.now(timezone.utc) - timedelta(days=30)).isoformat()
+response = client.chat.completions.create(
+    model="openrag-my_partition",
+    messages=[{"role": "user", "content": "What are the latest developments?"}],
+    extra_body={
+        "metadata": {
+            "temporal_filter": {
+                "indexed_after": thirty_days_ago
+            }
+        }
+    }
+)
+
+# Example 2: Query documents from a specific year
+response = client.chat.completions.create(
+    model="openrag-my_partition",
+    messages=[{"role": "user", "content": "What happened in 2024?"}],
+    extra_body={
+        "metadata": {
+            "temporal_filter": {
+                "created_after": "2024-01-01T00:00:00Z",
+                "created_before": "2024-12-31T23:59:59Z"
+            }
+        }
+    }
+)
+
+# Example 3: Let OpenRAG automatically extract temporal context from query
+# Queries like "last week", "documents from 2024", "recent changes" 
+# will automatically be filtered without explicit temporal_filter
+response = client.chat.completions.create(
+    model="openrag-my_partition",
+    messages=[{"role": "user", "content": "Show me documents from last week"}]
+)
+```
+
 ---
 
 ## ⚠️ Error Handling

docs/content/docs/documentation/data_model.md

@@ -106,3 +106,45 @@ Defines the many-to-many relationship between **users** and **partitions**, incl
 
 ---
 
+## **Vector Database Schema (Milvus)**
+
+In addition to the relational database, OpenRAG uses Milvus to store document chunks with embeddings and metadata.
+
+### Document Chunk Fields
+
+| Field Name | Type | Description |
+|------------|------|-------------|
+| `embedding` | FloatVector | Dense vector embedding of the chunk content |
+| `sparse_vector` | SparseFloatVector | Sparse BM25 vector for hybrid search |
+| `content` | VarChar | The actual text content of the chunk |
+| `partition` | VarChar | Partition name this chunk belongs to |
+| `filename` | VarChar | Source filename |
+| `page` | Int64 | Page number in the source document |
+| `_id` | VarChar (PK) | Unique chunk identifier |
+| `datetime` | VarChar | Primary timestamp from document content (user-provided) |
+| `modified_at` | VarChar | Document modification timestamp (ISO 8601 format) |
+| `created_at` | VarChar | Document creation timestamp (ISO 8601 format) |
+| `indexed_at` | VarChar | When the chunk was indexed into OpenRAG (ISO 8601 format) |
+
+### Temporal Fields Priority
+
+When filtering or scoring documents by time, OpenRAG uses the following priority:
+1. **`datetime`** - Highest priority, user-provided timestamp from document content
+2. **`modified_at`** - Document modification date
+3. **`created_at`** - Document creation date  
+4. **`indexed_at`** - Fallback, when the document was indexed
+
+All temporal fields are stored in ISO 8601 format (e.g., `2024-01-15T10:30:00Z`).
+
+### Temporal Filtering
+
+Vector database queries support temporal filtering with the following parameters:
+- `datetime_after` / `datetime_before`
+- `modified_after` / `modified_before`
+- `created_after` / `created_before`
+- `indexed_after` / `indexed_before`
+
+These filters use **OR logic** between the different date fields to ensure maximum flexibility in retrieving time-relevant documents.
+
+---
+

docs/content/docs/documentation/features_in_details.md

@@ -84,4 +84,49 @@ See the section on [distributed deployment in a ray cluster](#5-distributed-depl
 * **Contextual retrieval** - Anthropic's technique for enhanced chunk relevance
 * **Multilingual reranking** - using `Alibaba-NLP/gte-multilingual-reranker-base`
 
+</details>
+
+### 🕐 Temporal Awareness
+OpenRAG includes intelligent temporal understanding to deliver more relevant, time-aware responses.
+
+<details>
+
+<summary>Temporal Features</summary>
+
+* **Automatic date extraction** - Detects temporal expressions in queries across multiple languages
+  * ISO dates: `2024-01-15`, `2024-01-15T10:30:00`
+  * Numeric formats: `15/01/2024`, `01/15/2024`, `15.01.2024`
+  * Month-year: `01/2024`, `2024/01`
+  * Year only: `2024`, `2023`
+  * Relative time: "last 30 days", "últimos 7 días", "derniers 30 jours"
+  * Keywords: "today", "yesterday", "recent" (and multilingual equivalents)
+
+* **Document timestamp metadata** - Tracks temporal information for each document
+  * `datetime` - Primary timestamp from document content (user-provided)
+  * `modified_at` - Document modification timestamp
+  * `created_at` - Document creation timestamp  
+  * `indexed_at` - When the document was indexed into OpenRAG
+
+* **Temporal filtering** - Automatically filters search results based on detected time ranges
+  * Queries like "documents from 2024" only retrieve relevant documents
+  * "Last week's updates" focuses on recent content
+  * Works across all retrieval methods (base, multi-query, HyDE)
+
+* **Temporal scoring in reranking** - Balances relevance with recency
+  * Combines semantic relevance score with temporal score
+  * Configurable temporal weight (default: 30% temporal, 70% relevance)
+  * Linear decay formula favors more recent documents
+  * Configurable decay period (default: 365 days)
+  * Priority hierarchy: `datetime` > `modified_at` > `created_at` > `indexed_at`
+
+* **Temporal-aware prompts** - LLM receives temporal context
+  * Current date/time injected into system prompt
+  * Document timestamps included in retrieved chunks
+  * LLM instructed to consider recency when answering
+  * Prioritizes newer information for time-sensitive queries
+
+* **Configuration options** via environment variables:
+  * `RERANKER_TEMPORAL_WEIGHT` - Weight for temporal scoring (0.0-1.0, default: 0.3)
+  * `RERANKER_TEMPORAL_DECAY_DAYS` - Days for temporal score decay (default: 365)
+
 </details>

openrag/components/indexer/chunker/chunker.py

@@ -2,6 +2,7 @@
 from abc import ABC, abstractmethod
 from pathlib import Path
 from typing import Optional
+from datetime import datetime, timezone
 
 from components.prompts import CHUNK_CONTEXTUALIZER
 from components.utils import get_llm_semaphore, load_config
@@ -235,12 +236,12 @@ async def split_document(self, doc: Document, task_id: str = None):
             start_page = page_info["start_page"]
             end_page = page_info["end_page"]
             prev_page_num = end_page
-            filtered_chunks.append(
-                Document(
-                    page_content=chunk_w_context,
-                    metadata={**metadata, "page": start_page},
-                )
-            )
+            chunk_meta = {
+                **metadata,
+                "page": start_page,
+                "indexed_at": datetime.now(timezone.utc).isoformat(),
+            }
+            filtered_chunks.append(Document(page_content=chunk_w_context, metadata=chunk_meta))
         log.info("Document chunking completed")
         return filtered_chunks
 
@@ -352,12 +353,12 @@ async def split_document(self, doc: Document, task_id: str = None):
             start_page = page_info["start_page"]
             end_page = page_info["end_page"]
             prev_page_num = end_page
-            filtered_chunks.append(
-                Document(
-                    page_content=chunk_w_context,
-                    metadata={**metadata, "page": start_page},
-                )
-            )
+            chunk_meta = {
+                **metadata,
+                "page": start_page,
+                "indexed_at": datetime.now(timezone.utc).isoformat(),
+            }
+            filtered_chunks.append(Document(page_content=chunk_w_context, metadata=chunk_meta))
         log.info("Document chunking completed")
         return filtered_chunks
 
@@ -457,12 +458,12 @@ async def split_document(self, doc: Document, task_id: str = None):
             start_page = page_info["start_page"]
             end_page = page_info["end_page"]
             prev_page_num = end_page
-            filtered_chunks.append(
-                Document(
-                    page_content=chunk_w_context,
-                    metadata={**metadata, "page": start_page},
-                )
-            )
+            chunk_meta = {
+                **metadata,
+                "page": start_page,
+                "indexed_at": datetime.now(timezone.utc).isoformat(),
+            }
+            filtered_chunks.append(Document(page_content=chunk_w_context, metadata=chunk_meta))
         log.info("Document chunking completed")
         return filtered_chunks