# Running in production

Service lifetime, connections, transactions, cancellation and timeouts, retries, logging and security: what an
application needs to decide before it runs ARX for real.

## Startup checks

At startup, `CheckServerAsync()` (from 0.1.0-preview.2) asks the server once what it allows: its version, the
capabilities it removes with the reason (a standalone MongoDB runs no transactions), and problems such as a DynamoDB
table of the model that doesn't exist. `ValidateSchemaAsync()` checks the schema catalog against the tables.

## One service per database, shared

Create the service once at startup and share it (in ASP.NET Core, register it as a singleton; from 0.1.0-preview.2,
as `IArxDataService` where the code shouldn't depend on the provider). It holds the connection
string, the schema catalog and the provider's settings, and no connection: each call takes a connection from the
driver's pool (PostgreSQL, SQL Server, MySQL), or uses the client the service was given (MongoDB, DynamoDB), so
concurrent calls from many requests are fine.

Give it a schema catalog, declared or preloaded at startup. A service created with only the connection string works
(the SQL providers read each table from the database before the call), but every call then pays one more round trip:
much slower, and meant for simple CRUD in small applications. If you do run one, set `RememberResolvedTables = true`
so each table is read once per service.

Set the service's properties (`Strict`, `PagingCookieKey`, `RememberResolvedTables`) when you create it, before the first call, and don't change
them afterwards.

Connections, transactions and batch scopes are the opposite: one per unit of work, never shared between threads or
requests.

## Connections and pooling

The connection string comes from configuration or a secret store, never from code. Pool sizes, timeouts to connect and
TLS settings are the driver's and go in the connection string (Npgsql, Microsoft.Data.SqlClient, MySqlConnector,
the MongoDB driver). For DynamoDB, configure the `AmazonDynamoDBClient` you pass in: region, credentials, retries.

**Pools.** On PostgreSQL, ARX opens its own connections with `Timezone=UTC` added to the connection string (unless it
names a zone), and Npgsql pools by connection string: if the application also opens Npgsql connections with the same
string, they use a second pool beside ARX's, so size `Maximum Pool Size` for both. ARX doesn't take an `NpgsqlDataSource`.

**MongoDB's pool doesn't queue for long.** When every connection is busy, the MongoDB driver puts a call in a wait queue
and fails it once the queue is full or `waitQueueTimeoutMS` passes: ARX7005 ConnectionFailure (transient), with the
driver's `MongoWaitQueueFullException` inside. The SQL drivers wait for a free connection up to their connect timeout
instead. With many concurrent calls on one MongoDB service, size `maxPoolSize` for the peak, and retry ARX7005
with a backoff.
To share one connection and transaction with your own Npgsql or Dapper code, pass them to the interop overloads.

**Driver versions.** Each provider is built on its lowest supported driver version: Npgsql 8.0.3, Microsoft.Data.SqlClient
7.1.1, MySqlConnector 2.6.2, MongoDB.Driver 3.13.0, AWSSDK.DynamoDBv2 4.0.110. The PostgreSQL provider's whole test
suite also runs on Npgsql 10.0.3; the other providers are tested on their lowest version only (Versions and
compatibility, "Supported driver versions"). NuGet takes a higher version when another package needs one (EF Core 10's
provider lifts Npgsql to 10.x). ARX doesn't forbid that: run your own tests after a driver upgrade to a version not
listed as tested.

## Transactions

For several writes that must succeed or fail together, open the provider's connection and begin a transaction, or use a
batch scope:

```csharp
await service.RunBatchAsync(async batch =>
{
    ArxRecordId orderId = await batch.CreateAsync(order, cancellationToken);
    await batch.CreateMultipleAsync(lines, cancellationToken);
}, cancellationToken: cancellationToken);
```

A write by a filter (`UpdateMultipleAsync(values, filter)`, `DeleteMultipleAsync(table, filter)`) is all or nothing
by itself. It refuses an empty filter, so a missing condition doesn't change the whole table; pass
`new ArxFilterWriteOptions { AllRows = true }` only where every row is meant, and check the count it returns against
what you expect. For a large load whose ids you don't need, `new ArxCreateMultipleOptions { ReturnIds = false }` takes
the database's bulk path on PostgreSQL (5,000 rows or more of one shape) and SQL Server (1,000 or more); below those
sizes the ordinary statement was faster on the benchmarks.

`RunBatchAsync` commits when the body returns and rolls back when it throws; a scope from `BeginBatchAsync` commits only
when you call `CommitAsync`, and disposing it without that rolls back. After a database error inside a transaction, the
transaction is aborted: later calls and the commit throw ARX1015 TransactionAborted, and you start again.

A commit can also fail after it was sent: the connection breaks or times out before the database answers. ARX then
can't know whether the work was saved, so the transaction's `State` (or the scope's) is `OutcomeUnknown` and the error
(ARX7005 or ARX7003) has `IsTransient` false, so a retry policy doesn't save the work twice. Check what is in the
database (the order number you wrote, a version) before doing the work again. A commit the database refuses (a
deadlock, a serialization failure) is rolled back and stays transient: retrying the whole unit of work is safe. On
MongoDB the same holds for a single write outside a transaction: one that timed out or lost its connection after it was
sent is ARX7005 or ARX7003 with `IsTransient` false (its outcome is unknown), not a transient error.

## Cancellation and timeouts

Pass the request's `CancellationToken`. A cancelled call throws `OperationCanceledException` (or `TaskCanceledException`),
not an `ArxException`.

Every call takes a token, the interop overloads of the SQL providers too (the ones that take your own `IDbConnection`
and `IDbTransaction`): from 1.0.0-rc.1 their parameters are named `connection` and `transaction`, and they end with a
`CancellationToken`.

Long queries take a timeout in their options (`CommandTimeout`, in seconds); without one the driver's default applies
(30 s for the SQL drivers). A timeout is ARX7003, transient.

**MongoDB writes once sent.** MongoDB has no way to stop a write the server is already running, so ARX honours a
token only until a write outside a transaction (or a commit) is sent, then waits for the outcome: a caller is never
told "cancelled" about a write that was applied. A write blocked behind another transaction therefore waits until that
transaction ends, past the token. Set a server-side limit where that matters (`maxTimeMS`, or
`maxTransactionLockRequestTimeoutMillis` on the server), or write inside an `ArxTransaction`: there a cancelled write
fails the call and nothing of the transaction is kept.

**Version columns protect only writers that use them.** `.Version(table, column)` checks every write that goes through
ARX with that catalog. Another service, a script or a hand-written UPDATE that doesn't change the version can still
overwrite a row unseen: give every writer the same rule (or a trigger that bumps the version).

## Retries

ARX doesn't retry your calls or your transactions. Retry the whole unit of work when an error's `IsTransient` is true,
with a backoff, and keep in mind that the AWS SDK and the MongoDB driver already retry some failures underneath, and that
on MongoDB the transaction ARX opens itself for a call over several documents is run once more on a
`TransientTransactionError` label (and committed once more on `UnknownTransactionCommitResult`), as MongoDB recommends
(Errors in practice, "Retrying"). A write that failed after it was sent may
or may not have happened: supplied ids or a business key let you check before writing again. The Errors in practice
guide has the details.

## Wrapping the service

A decorator around `IArxDataService` (for auditing, tenancy checks, retries) sees the calls made on the service. Calls
made inside a batch scope go to the scope (`IArxBatchScope`), not to the service: wrap `IArxBatchDataService.BeginBatchAsync`
too, and return a wrapped scope, to see them. Every call, in a scope or not, is traced (`ARX.SDK` ActivitySource), which
covers auditing that only needs to observe.

## Tracing, metrics and diagnostics

From 0.1.0-preview.2, every operation is an activity of the `ARX.SDK` ActivitySource and a measurement of the `ARX.SDK`
Meter. With OpenTelemetry:

```csharp
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing.AddSource(ArxDiagnostics.SourceName))
    .WithMetrics(metrics => metrics.AddMeter(ArxDiagnostics.SourceName));
```

Each activity is named after the operation and the table (`Update account`), kind Client, with `db.system.name`,
`db.operation.name`, `db.collection.name` and `arx.provider`; a failure sets the status Error, `error.type` and
`arx.error.code`. The metrics are `arx.operation.duration` (seconds) and `arx.operation.errors`, with the same tags.
Values are never recorded, only names. Without a listener, each call costs a check. An `ActivityListener` or a
`MeterListener` reads them without any package (the "Set the service up for production" samples). To log each call
through ILogger, see [Logging and tracing](logging-and-tracing.md).

Besides those:

- The exception: `Code`, `Category`, `IsTransient`, the table and column where known, the database's own code
  (`SqlState` or `NativeCode`) and the driver's exception as `InnerException`. The message is written not to repeat
  the values sent, so it can go to logs.
- The driver's own logging and diagnostics, configured as its documentation says.
- For DynamoDB, the `AmazonDynamoDBClient` you configure: its logging, metrics and retry settings are yours.

## Security

- Values are always sent as parameters, never written into SQL or query text.
- Names (tables, attributes, match columns) are checked: a name carrying SQL or query syntax is refused before the
  database as ARX1017 SuspectedInjection. Build names from your own code or an allow list, not from requests; log
  ARX1017, since it is an attempt rather than a mistake (the GuardAgainstAttacks samples show one way).
- MongoDB and DynamoDB paging cookies carry the last item's key values (an email, a tenant id). From 0.1.0-preview.2
  they are sealed by default: encrypted and authenticated, so a client can neither read nor change one. Without a key,
  ARX seals with a random key of the process: right for one process. With several instances, or cookies that must
  survive a restart, give every instance the same `PagingCookieKey` (32 bytes or more, from your secret store); a
  cookie that reaches an instance with another key is ARX1001. `PagingCookieProtection = Signed` keeps cookies
  readable but unchangeable; `Plain` leaves both open.
- **Run under a least-privilege database role.** The schema catalog says which tables ARX writes and reads by id; it
  doesn't stop a query (`RetrieveMultipleAsync`) reading any table or view the database user can see, system views
  included, or `ExecuteAsync` calling any function it may run. `service.Strict = true` (with a preloaded catalog) limits
  queries to the preloaded tables and columns, but not functions or procedures. What the database user may read and
  call is the boundary: grant it only the application's tables and routines, never a superuser or `db_owner`. ARX
  needs no administrative rights for its operations.