TDS v3 Internals Redesign

CHANGELOG.md

@@ -5,6 +5,78 @@ All notable changes to this project will be documented in this file.
 The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
 and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
 
+## v3.0.0 (2026-03-05)
+
+### Breaking Changes
+
+* **Money columns return `%Decimal{}`** instead of `float`. Use `Decimal`
+  arithmetic for money values. Previously `SELECT CAST(10.50 AS money)`
+  returned `10.5` (float); now returns `Decimal.new("10.5000")`.
+
+* **Date/time columns always return Elixir calendar structs.** No more tuple
+  format. `smalldatetime`, `datetime`, `datetime2` return `%NaiveDateTime{}`;
+  `datetimeoffset` returns `%DateTime{}`; `date` returns `%Date{}`;
+  `time` returns `%Time{}`. The `use_elixir_calendar_types` config option
+  is ignored — struct output is always on.
+
+* **`Tds.Types.UUID` deprecated.** Use `Ecto.UUID` instead. The new
+  `Tds.Type.UUID` wire handler performs MSSQL mixed-endian byte reordering
+  at the protocol level, so `Ecto.UUID` works directly.
+
+* **`Tds.Types` module removed.** The 1815-line monolithic type module has been
+  replaced by 12 focused handler modules under `Tds.Type.*`.
+
+### New Features
+
+* **`Tds.Type` behaviour** — Pluggable type system with 7 callbacks:
+  `type_codes/0`, `type_names/0`, `decode_metadata/1`, `decode/2`,
+  `encode/2`, `param_descriptor/2`, `infer/1`.
+
+* **12 handler modules** — `Tds.Type.{Boolean, Integer, Float, Decimal, Money,
+  String, Binary, DateTime, UUID, Xml, Variant, Udt}`.
+
+* **`Tds.Type.DataReader`** — Shared framing reader with 6 strategies
+  (`:fixed`, `:bytelen`, `:shortlen`, `:longlen`, `:plp`, `:variant`).
+  PLP uses iolist accumulation (8x faster than binary concat).
+  All strategies sever sub-binary references via `:binary.copy/1` to
+  prevent memory leaks.
+
+* **`Tds.Type.Registry`** — Per-connection type registry mapping TDS type
+  codes and atom names to handlers. Supports user-provided `extra_types`
+  that override built-in handlers.
+
+* **`extra_types` connection option** — Register custom type handlers at
+  connect time: `Tds.start_link(extra_types: [MyApp.GeographyType])`.
+
+### Performance
+
+Benchmarked on Apple M4 Pro, 48 GB, Elixir 1.18.1, Erlang/OTP 27.2:
+
+* Integer decode: **54% faster** (7.12M → 10.96M ips)
+* Decimal encode (1000 params): **8.5x faster** (0.63K → 5.39K ips)
+* Decimal encode memory: **8.8x less** (1.99 MB → 226 KB per 1000 params)
+* PLP reassembly: **8x faster** at 1 MB, **7x faster** at 10 MB
+* PLP memory: **~2x less** (iolist vs binary concat)
+
+### Improvements
+
+* Decimal encoding no longer mutates `Decimal.Context` in the process
+  dictionary. Precision and scale are passed via metadata.
+* All decoded values sever sub-binary references to the TCP packet buffer,
+  preventing memory retention when values are stored in ETS or GenServer state.
+
+### Migration Guide
+
+1. **Money values**: Replace float arithmetic with `Decimal` operations.
+   `Decimal.to_float/1` is available if float is needed temporarily.
+2. **Date/time tuples**: Replace `{{y,m,d},{h,min,s}}` with
+   `~N[2013-10-12 00:37:14]` or `NaiveDateTime.new!/3`. For encoding,
+   convert tuples to calendar structs before passing as parameters.
+3. **Remove `use_elixir_calendar_types`**: Delete from your config —
+   calendar structs are now the only output format.
+4. **`Tds.Types.UUID`**: Replace with `Ecto.UUID`. If you called
+   `Tds.generate_uuid/0`, use `Ecto.UUID.bingenerate/0` instead.
+
 ## v2.3.5 (2024-01-23)
 ### Improvements
 * Removed unnecessary append of possibly large binaries for floats
@@ -324,11 +396,7 @@ could not determine if binary is of uuid type, it interpreted such values as raw
 ### Enhancements
   * Added API for ATTN call
 
-<<<<<<< HEAD
-## v0.1.5
-=======
 ## v0.1.5 - 2015-02-19
->>>>>>> 1007dc1 (Misc doc changes)
 ### Bug Fixes
   * Fixed issue where driver would not call Connection.next when setting the state to :ready
   * Fixed UCS2 Encoding

README.md

@@ -8,8 +8,6 @@ MSSQL / TDS Database driver for Elixir.
 ### NOTE:
 Since TDS version 2.0, `tds_ecto` package is deprecated, this version supports `ecto_sql` since version 3.3.4.
 
-Please check out the issues for a more complete overview. This branch should not be considered stable or ready for production yet.
-
 For stable versions always use [hex.pm](https://hex.pm/packages/tds) as source for your mix.exs.
 
 ## Usage
@@ -19,7 +17,7 @@ Add `:tds` as a dependency in your `mix.exs` file.
 ```elixir
 def deps do
   [
-    {:tds, "~> 2.3"}
+    {:tds, "~> 3.0"}
   ]
 end
 ```
@@ -151,52 +149,77 @@ This functionality requires specific environment to be developed.
 
 ## Data representation
 
-| TDS               | Elixir                                                                                     |
-| ----------------- | ------------------------------------------------------------------------------------------ |
-| NULL              | nil                                                                                        |
-| bool              | true / false                                                                               |
-| char              | "é"                                                                                        |
-| int               | 42                                                                                         |
-| float             | 42.0                                                                                       |
-| text              | "text"                                                                                     |
-| binary            | <<42>>                                                                                     |
-| numeric           | #Decimal<42.0>                                                                             |
-| date              | {2013, 10, 12} or %Date{}                                                                  |
-| time              | {0, 37, 14} or {0, 37, 14, 123456} or %Time{}                                              |
-| smalldatetime     | {{2013, 10, 12}, {0, 37, 14}} or {{2013, 10, 12}, {0, 37, 14, 123456}}                     |
-| datetime          | {{2013, 10, 12}, {0, 37, 14}} or {{2013, 10, 12}, {0, 37, 14, 123456}} or %NaiveDateTime{} |
-| datetime2         | {{2013, 10, 12}, {0, 37, 14}} or {{2013, 10, 12}, {0, 37, 14, 123456}} or %NaiveDateTime{} |
-| datetimeoffset(n) | {{2013, 10, 12}, {0, 37, 14}} or {{2013, 10, 12}, {0, 37, 14, 123456}} or %DateTime{}      |
-| uuid              | <<160,238,188,153,156,11,78,248,187,109,107,185,189,56,10,17>>                             |
-
-Currently unsupported: [User-Defined Types](https://docs.microsoft.com/en-us/sql/relational-databases/clr-integration-database-objects-user-defined-types/working-with-user-defined-types-in-sql-server), XML
+| TDS               | Elixir                 |
+| ----------------- | ---------------------- |
+| NULL              | `nil`                  |
+| bool              | `true` / `false`       |
+| char / varchar    | `"text"`               |
+| nchar / nvarchar  | `"text"`               |
+| int / bigint      | `42`                   |
+| float / real      | `42.0`                 |
+| text / ntext      | `"text"`               |
+| binary / varbinary | `<<42>>`              |
+| numeric / decimal | `#Decimal<42.0>`       |
+| money / smallmoney | `#Decimal<10.5000>`   |
+| date              | `%Date{}`              |
+| time              | `%Time{}`              |
+| smalldatetime     | `%NaiveDateTime{}`     |
+| datetime          | `%NaiveDateTime{}`     |
+| datetime2         | `%NaiveDateTime{}`     |
+| datetimeoffset(n) | `%DateTime{}`          |
+| uniqueidentifier  | `<<_::128>>`           |
+| xml               | `"<xml>...</xml>"`     |
+| sql_variant       | varies by inner type   |
+
+User-Defined Types (UDT) are returned as raw binary by default. Register custom
+handlers via `extra_types` to decode specific UDTs (see below).
 
 ### Dates and Times
 
-Tds can work with dates and times in either a tuple format or as Elixir calendar types. Calendar types can be enabled in the config with `config :tds, opts: [use_elixir_calendar_types: true]`.
-
-**Tuple forms:**
+As of v3.0, all date/time columns are decoded as Elixir calendar structs:
 
-- Date: `{yr, mth, day}`
-- Time: `{hr, min, sec}` or `{hr, min, sec, fractional_seconds}`
-- DateTime: `{date, time}`
-- DateTimeOffset: `{utc_date, utc_time, offset_mins}`
+| SQL Type            | Elixir Type          |
+| ------------------- | -------------------- |
+| `date`              | `%Date{}`            |
+| `time(n)`           | `%Time{}`            |
+| `smalldatetime`     | `%NaiveDateTime{}`   |
+| `datetime`          | `%NaiveDateTime{}`   |
+| `datetime2(n)`      | `%NaiveDateTime{}`   |
+| `datetimeoffset(n)` | `%DateTime{}`        |
 
-In SQL Server, the `fractional_seconds` of a `time`, `datetime2` or `datetimeoffset(n)` column can have a precision of 0-7, where the `microsecond` field of a `%Time{}` or `%DateTime{}` struct can have a precision of 0-6.
+SQL Server `time`, `datetime2`, and `datetimeoffset` support precision 0-7.
+Elixir's `microsecond` field supports precision 0-6, so fractional seconds
+are truncated to microsecond precision when the SQL scale exceeds 6.
 
-Note that the DateTimeOffset tuple expects the date and time in UTC and the offset in minutes. For example, `{{2020, 4, 5}, {5, 30, 59}, 600}` is equal to `'2020-04-05 15:30:59+10:00'`.
+The `use_elixir_calendar_types` config option from v2.x is no longer needed
+and is ignored in v3.0.
 
 ### UUIDs
 
 [MSSQL stores UUIDs in mixed-endian
-format](https://dba.stackexchange.com/a/121878), and these mixed-endian UUIDs
-are returned in [Tds.Result](https://hexdocs.pm/tds/Tds.Result.html).
+format](https://dba.stackexchange.com/a/121878) where the first three groups
+are byte-reversed (little-endian) and the last two are big-endian.
+
+As of v3.0, the `Tds.Type.UUID` wire handler performs this byte reordering
+automatically at the protocol level, so `Ecto.UUID` works directly without
+any wrapper module.
+
+`Tds.Types.UUID` is deprecated. Use `Ecto.UUID` for all UUID operations.
+
+### Custom Type Handlers
 
-To convert a mixed-endian UUID binary to a big-endian string, use
-[Tds.Types.UUID.load/1](https://hexdocs.pm/tds/Tds.Types.UUID.html#load/1)
+Register custom type handlers via the `extra_types` connection option:
+
+```elixir
+Tds.start_link(
+  hostname: "localhost",
+  extra_types: [MyApp.GeographyType]
+)
+```
 
-To convert a big-endian UUID string to a mixed-endian binary, use
-[Tds.Types.UUID.dump/1](https://hexdocs.pm/tds/Tds.Types.UUID.html#dump/1)
+Custom handlers implement the `Tds.Type` behaviour and can override built-in
+handlers for the same type codes or names. See `Tds.Type` docs for the
+callback specification.
 
 ## Contributing
 
@@ -216,7 +239,7 @@ use it for the first time.
 
 The tests require an SQL Server database to be available on localhost.
 If you are not using Windows OS you can start sql server instance using Docker.
-Official SQL Server Docker image can be found [here](https://hub.docker.com/r/microsoft/mssql-server-linux).
+Official SQL Server Docker image can be found [here](https://hub.docker.com/r/microsoft/mssql-server).
 
 If you do not have specific requirements on how you would like to start sql server
 in docker, you can use script for this repo.

bench/plp_bench.exs

@@ -0,0 +1,83 @@
+# PLP chunk reassembly: current binary concat vs iolist accumulation.
+# Run: mix run bench/plp_bench.exs
+#
+# Baseline results (2026-03-05)
+# Machine: Apple M4 Pro, 48 GB, macOS
+# Elixir 1.18.1, Erlang/OTP 27.2, JIT enabled
+#
+# Name                            ips        average  deviation         median         99th %
+# 1MB new (iolist)            43.15 K       23.17 us    +-10.68%      23.38 us       29.21 us
+# 1MB current (concat)         5.38 K      186.02 us    +-21.73%     183.63 us      291.98 us
+# 10MB new (iolist)            3.44 K      290.93 us    +-21.21%     280.71 us      730.41 us
+# 10MB current (concat)        0.49 K     2026.15 us     +-6.40%    2018.46 us     2400.42 us
+#
+# Summary: iolist is 8x faster at 1MB and 7x faster at 10MB
+#
+# Memory usage:
+# 1MB new (iolist)             18.13 KB
+# 1MB current (concat)         35.44 KB - 1.96x
+# 10MB new (iolist)           170.70 KB
+# 10MB current (concat)       357.38 KB - 2.10x
+
+defmodule PLPBench do
+  # Current approach: buf <> :binary.copy(chunk)
+  def decode_plp_current(<<0::little-unsigned-32, _rest::binary>>, buf),
+    do: buf
+
+  def decode_plp_current(
+        <<size::little-unsigned-32,
+          chunk::binary-size(size),
+          rest::binary>>,
+        buf
+      ) do
+    decode_plp_current(rest, buf <> :binary.copy(chunk))
+  end
+
+  # New approach: iolist accumulation
+  def decode_plp_iolist(<<0::little-unsigned-32, _rest::binary>>, acc),
+    do: :lists.reverse(acc) |> IO.iodata_to_binary()
+
+  def decode_plp_iolist(
+        <<size::little-unsigned-32,
+          chunk::binary-size(size),
+          rest::binary>>,
+        acc
+      ) do
+    decode_plp_iolist(rest, [chunk | acc])
+  end
+
+  def build_plp_payload(total_size, chunk_size) do
+    chunk = :crypto.strong_rand_bytes(chunk_size)
+    num_chunks = div(total_size, chunk_size)
+
+    chunks =
+      for _ <- 1..num_chunks, into: <<>> do
+        <<chunk_size::little-unsigned-32>> <> chunk
+      end
+
+    chunks <> <<0::little-unsigned-32>>
+  end
+end
+
+payload_1mb = PLPBench.build_plp_payload(1_048_576, 4096)
+payload_10mb = PLPBench.build_plp_payload(10_485_760, 4096)
+
+Benchee.run(
+  %{
+    "1MB current (concat)" => fn ->
+      PLPBench.decode_plp_current(payload_1mb, <<>>)
+    end,
+    "1MB new (iolist)" => fn ->
+      PLPBench.decode_plp_iolist(payload_1mb, [])
+    end,
+    "10MB current (concat)" => fn ->
+      PLPBench.decode_plp_current(payload_10mb, <<>>)
+    end,
+    "10MB new (iolist)" => fn ->
+      PLPBench.decode_plp_iolist(payload_10mb, [])
+    end
+  },
+  warmup: 1,
+  time: 5,
+  memory_time: 2
+)

bench/type_system_bench.exs

@@ -0,0 +1,83 @@
+# Benchmarks for type system decode/encode throughput.
+# Run: mix run bench/type_system_bench.exs
+#
+# Requires benchee: add {:benchee, "~> 1.3", only: :dev, runtime: false}
+# to mix.exs deps if not present.
+
+alias Tds.Parameter
+alias Tds.Type.{DataReader, Registry}
+
+defmodule TypeBench.Fixtures do
+  import Tds.Protocol.Constants
+
+  @registry Tds.Type.Registry.new()
+
+  def registry, do: @registry
+
+  def integer_decode_input do
+    {:ok, handler} =
+      Registry.handler_for_code(@registry, tds_type(:int))
+
+    meta = %{data_reader: {:fixed, 4}, handler: handler}
+    data = <<42, 0, 0, 0, 0xFF>>
+    {meta, data}
+  end
+
+  def string_decode_input do
+    value = String.duplicate("hello", 100)
+    ucs2 = Tds.Encoding.UCS2.from_string(value)
+    size = byte_size(ucs2)
+
+    {:ok, handler} =
+      Registry.handler_for_code(@registry, tds_type(:nvarchar))
+
+    meta = %{
+      data_reader: :shortlen,
+      collation: %Tds.Protocol.Collation{codepage: :RAW},
+      encoding: :ucs2,
+      length: size,
+      handler: handler
+    }
+
+    data = <<size::little-unsigned-16>> <> ucs2 <> <<0xFF>>
+    {meta, data}
+  end
+
+  def decimal_encode_params do
+    for _ <- 1..1000 do
+      %Parameter{
+        name: "@1",
+        value: Decimal.new("12345.6789"),
+        type: :decimal
+      }
+    end
+  end
+end
+
+{int_meta, int_data} = TypeBench.Fixtures.integer_decode_input()
+{str_meta, str_data} = TypeBench.Fixtures.string_decode_input()
+params = TypeBench.Fixtures.decimal_encode_params()
+registry = TypeBench.Fixtures.registry()
+
+Benchee.run(
+  %{
+    "decode integer" => fn ->
+      {raw, _rest} = DataReader.read(int_meta.data_reader, int_data)
+      int_meta.handler.decode(raw, int_meta)
+    end,
+    "decode string" => fn ->
+      {raw, _rest} = DataReader.read(str_meta.data_reader, str_data)
+      str_meta.handler.decode(raw, str_meta)
+    end,
+    "encode 1000 decimal params" => fn ->
+      Enum.each(params, fn p ->
+        {:ok, handler} = Registry.handler_for_name(registry, p.type)
+        meta = %{type: p.type}
+        handler.encode(p.value, meta)
+      end)
+    end
+  },
+  warmup: 2,
+  time: 5,
+  memory_time: 2
+)