Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
36 changes: 1 addition & 35 deletions architecture/consistency.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -27,43 +27,9 @@ Only once all the client-side mutations have been acknowledged by the server, an
There is one nuanced case here, which is buckets with [Priority 0](/sync/streams/prioritized-sync#special-case-priority-0) if you are using [Prioritized Syncing](/sync/streams/prioritized-sync).
</Note>

## Types of Client-Side Mutations/Operations

The client automatically records mutations to the client-side database as `PUT`, `PATCH` or `DELETE` operations — corresponding to `INSERT`, `UPDATE` or `DELETE` statements in SQLite. These are grouped together in a batch per client-side transaction.

Since the [developer has full control](/architecture/client-architecture#writing-data-via-sqlite-database-and-upload-queue) over how mutations are applied to the source database, more advanced operations can be modeled on top of these three. See [Custom Conflict Resolution](/handling-writes/custom-conflict-resolution) for examples.

## Validation and Conflict Handling

With PowerSync offering [full flexibility](/architecture/client-architecture#writing-data-via-sqlite-database-and-upload-queue) in how mutations are <Tooltip tip="I.e., processed via your backend API and applied to the source database">applied on the server</Tooltip>, it is also the developer's responsibility to implement this correctly to avoid consistency issues.

Some scenarios to consider:

While the client was offline, a row was modified on the client-side. By the time the client is online again, that row has been deleted on the source database. Some options for handling the mutation in your backend:

- Discard the mutation.
- Discard the entire transaction.
- Re-create the <Tooltip tip="PowerSync supports NoSQL source databases like MongoDB that have 'documents' rather than 'rows', but we often refer just to 'row' for brevity.">row</Tooltip>.
- Record the failed mutation elsewhere, potentially notifying the user and allowing the user to resolve the issue.

Some other examples include foreign-key or not-null constraints, maximum size of numeric fields, unique constraints, and access restrictions (such as row-level security policies).

In an online-only application, the user typically sees the error as soon as it occurs, and can correct the issue as required. In an offline-capable application that syncs asynchronously with the server, these errors may occur much later than when the mutation was made, so more care is required to handle these cases.

Make sure that errors like these do not block the upload queue. The upload queue in the PowerSync Client SDK is a blocking [FIFO](https://en.wikipedia.org/wiki/FIFO_%28computing_and_electronics%29) queue, and the queue cannot advance if the backend does not <Tooltip tip="The upload queue advances when your 'uploadData()' function calls 'complete()' on a transaction. Your function typically does this after your backend API acknowledges the write with a 2xx response. If 'uploadData()' throws or returns without calling 'complete()', the transaction stays at the front of the queue.">acknowledge a mutation</Tooltip>. As described above, while the queue has writes, the client does not apply the next checkpoint of synced data.

There is no single correct choice for handling write failures like these. The best action depends on the specific application and scenario. However, we do have some suggestions for general approaches:

1. In general, consider relaxing constraints somewhat on the backend where they are not absolutely required. It may be better to accept data that is somewhat inconsistent (e.g. a client not applying all expected validations), rather than discarding the data completely.
2. If it is critical to preserve all client mutations and preserve the order of mutations:
1. Block the client's upload queue on unexpected errors (don't <Tooltip tip="The upload queue advances when your 'uploadData()' function calls 'complete()' on a transaction. Your function typically does this after your backend API acknowledges the write with a 2xx response. If 'uploadData()' throws or returns without calling 'complete()', the transaction stays at the front of the queue.">acknowledge the mutation</Tooltip> in your backend API).
2. Implement error monitoring to be notified of issues, and resolve the issues as soon as possible.
3. If it is critical to preserve all client mutations, but the exact order may not be critical:
1. On a constraint error, persist the transaction in a separate queue on your backend, and acknowledge the change.
2. The backend queue can then be inspected and retried asynchronously, without blocking the client-side upload queue.
4. If it is acceptable to lose some mutations due to constraint errors:
1. Discard the mutation, or the entire transaction if the changes must all be applied together.
2. Implement error notifications to detect these issues.
Your backend decides how each uploaded mutation is applied, so it also decides what happens when one fails, for example because it violates a constraint or an access rule. Because the client doesn't advance to a new checkpoint while mutations are queued, a mutation that stays in the upload queue also holds back newer synced data. [What Happens When a Write Fails](/handling-writes/write-path/concepts#what-happens-when-a-write-fails) explains how to handle each kind of failure.

See also:

Expand Down
9 changes: 6 additions & 3 deletions client-sdks/frameworks/next-js.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -263,8 +263,11 @@ export class BackendConnector implements PowerSyncBackendConnector {
}
await transaction.complete();
} catch (error: any) {
console.error(`Data upload error - discarding`, error);
await transaction.complete();
// Throw so that the SDK keeps the transaction and retries it.
// If your backend permanently rejects a change, complete the transaction instead:
// https://docs.powersync.com/handling-writes/write-path/concepts#what-happens-when-a-write-fails
console.error(`Data upload error`, error);
throw error;
}
}
}
Expand All @@ -273,7 +276,7 @@ export class BackendConnector implements PowerSyncBackendConnector {
There are two core functions to this file:

- `fetchCredentials()` - Used to return a JWT token to the PowerSync Service for authentication.
- `uploadData()` - Used to upload changes captured in the local SQLite database that need to be sent to the backend source database, in this case Supabase. We'll get back to this further down.
- `uploadData()` - Used to upload changes captured in the local SQLite database that need to be sent to the backend source database, in this case Supabase. To pick how to implement it, see [Choosing Your Implementation](/handling-writes/write-path/choosing-your-implementation).

You'll notice that we need to add a `.env` file to our project which will contain two variables:

Expand Down
9 changes: 6 additions & 3 deletions client-sdks/frameworks/nuxt.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,7 @@ export class PowerSyncConnector implements PowerSyncBackendConnector {

async uploadData(db: any) {
// Send local changes to your backend.
// See https://docs.powersync.com/client-sdk-references/javascript-web#3-integrate-with-your-backend
// To pick how to upload writes, see https://docs.powersync.com/handling-writes/write-path/choosing-your-implementation
const transaction = await db.getNextCrudTransaction();
if (!transaction) return;

Expand All @@ -148,8 +148,11 @@ export class PowerSyncConnector implements PowerSyncBackendConnector {
}
await transaction.complete();
} catch (error: any) {
console.error('Data upload error - discarding', error);
await transaction.complete();
// Throw so that the SDK keeps the transaction and retries it.
// If your backend permanently rejects a change, complete the transaction instead:
// https://docs.powersync.com/handling-writes/write-path/concepts#what-happens-when-a-write-fails
console.error('Data upload error', error);
throw error;
}
}
}
Expand Down
15 changes: 3 additions & 12 deletions client-sdks/reference/capacitor.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -191,22 +191,13 @@ export const setupPowerSync = async () => {

### 3. Integrate with Your Backend

The backend connector connects the PowerSync client SDK to your application backend. The SDK uses it to:

1. Get an auth token to connect to the PowerSync instance.
2. Upload client-side writes to your backend API. The SDK places every write to the SQLite database in an upload queue and uploads the queue to your backend when the user is connected. Your backend then applies the changes to the source database.

The connector must implement two methods:
The backend connector gives the SDK the credentials it needs to connect to your PowerSync instance and download data. It also uploads your app's local writes to your backend. The connector must implement two methods:

1. [PowerSyncBackendConnector.fetchCredentials](https://github.com/powersync-ja/powersync-js/blob/ed5bb49b5a1dc579050304fab847feb8d09b45c7/packages/common/src/client/connection/PowerSyncBackendConnector.ts#L16) - The SDK calls this method to get authentication credentials. It caches the credentials and calls the method again only when needed, for example on the first connection or when the token is near expiry. See [When `fetchCredentials()` is Called](/configuration/app-backend/client-side-integration#when-fetchcredentials-is-called) for details and [Authentication Setup](/configuration/auth/overview) for how to generate credentials.
2. [PowerSyncBackendConnector.uploadData](https://github.com/powersync-ja/powersync-js/blob/ed5bb49b5a1dc579050304fab847feb8d09b45c7/packages/common/src/client/connection/PowerSyncBackendConnector.ts#L24) - The SDK calls this method whenever it has client-side writes to upload to your backend API. Implement how those writes are processed and uploaded. See [When `uploadData()` is Called](/configuration/app-backend/client-side-integration#when-uploaddata-is-called) for triggers, throttling, and retry behavior, and [Write Path Overview](/handling-writes/write-path/overview) for the app backend implementation.
2. [PowerSyncBackendConnector.uploadData](https://github.com/powersync-ja/powersync-js/blob/ed5bb49b5a1dc579050304fab847feb8d09b45c7/packages/common/src/client/connection/PowerSyncBackendConnector.ts#L24) - The SDK queues local writes and calls this method to send them to your write API, which applies them to the source database. If your app only reads synced data for now, you can implement this method later. To pick how to upload writes, see [Choosing Your Implementation](/handling-writes/write-path/choosing-your-implementation).

**Example:**

<Tip>
See the [JavaScript Web SDK backend integration section](/client-sdks/reference/javascript-web#3-integrate-with-your-backend) for connector examples with Supabase and Firebase authentication, and handling `uploadData` with batch operations.
</Tip>

```js
import { UpdateType } from '@powersync/web';

Expand All @@ -225,7 +216,7 @@ export class Connector {
// Implement uploadData to send local changes to your backend service.
// You can omit this method if you only want to sync data from the database to the client

// See example implementation here: https://docs.powersync.com/client-sdks/usage-examples#send-changes-in-local-data-to-your-backend-service
// To pick how to upload writes, see https://docs.powersync.com/handling-writes/write-path/choosing-your-implementation
}
}
```
Expand Down
79 changes: 7 additions & 72 deletions client-sdks/reference/dotnet.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -207,50 +207,33 @@ The initialization syntax differs slightly between the Common and MAUI SDKs:

### 3. Integrate with Your Backend

The backend connector connects the PowerSync client SDK to your application backend. The SDK uses it to:

1. Get an auth token to connect to the PowerSync instance.
2. Upload client-side writes to your backend API. The SDK places every write to the SQLite database in an upload queue and uploads the queue to your backend when the user is connected. Your backend then applies the changes to the source database.

The connector must implement two methods:
The backend connector gives the SDK the credentials it needs to connect to your PowerSync instance and download data. It also uploads your app's local writes to your backend. The connector must implement two methods:

1. [IPowerSyncBackendConnector.FetchCredentials](https://powersync-ja.github.io/powersync-dotnet/api/PowerSync.Common.Client.Connection.IPowerSyncBackendConnector.FetchCredentials.html) - The SDK calls this method to get authentication credentials. It caches the credentials and calls the method again only when needed, for example on the first connection or when the token is near expiry. See [When `fetchCredentials()` is Called](/configuration/app-backend/client-side-integration#when-fetchcredentials-is-called) for details and [Authentication Setup](/configuration/auth/overview) for how to generate credentials.
2. [IPowerSyncBackendConnector.UploadData](https://powersync-ja.github.io/powersync-dotnet/api/PowerSync.Common.Client.Connection.IPowerSyncBackendConnector.UploadData.html) - The SDK calls this method whenever it has client-side writes to upload to your backend API. Implement how those writes are processed and uploaded. See [When `uploadData()` is Called](/configuration/app-backend/client-side-integration#when-uploaddata-is-called) for triggers, throttling, and retry behavior, and [Write Path Overview](/handling-writes/write-path/overview) for the app backend implementation.
2. [IPowerSyncBackendConnector.UploadData](https://powersync-ja.github.io/powersync-dotnet/api/PowerSync.Common.Client.Connection.IPowerSyncBackendConnector.UploadData.html) - The SDK queues local writes and calls this method to send them to your write API, which applies them to the source database. If your app only reads synced data for now, you can implement this method later. To pick how to upload writes, see [Choosing Your Implementation](/handling-writes/write-path/choosing-your-implementation).

**Example:**

```cs
using System;
using System.Collections.Generic;
using System.Net.Http;
using System.Text;
using System.Text.Json;
using System.Threading.Tasks;
using PowerSync.Common.Client;
using PowerSync.Common.Client.Connection;
using PowerSync.Common.DB.Crud;

public class MyConnector : IPowerSyncBackendConnector
{
private readonly HttpClient _httpClient;

// User credentials for the current session
public string UserId { get; private set; }

// Service endpoints
private readonly string _backendUrl;
// PowerSync instance URL
private readonly string _powerSyncUrl;
private string? _clientId;

public MyConnector()
{
_httpClient = new HttpClient();

// In a real app, this would come from your authentication system
UserId = "user-123";

// Configure your service endpoints
_backendUrl = "[Your backend API URL]";
// Configure your PowerSync instance URL
_powerSyncUrl = "https://your-powersync-instance.powersync.journeyapps.com";
}

Expand All @@ -275,57 +258,9 @@ public class MyConnector : IPowerSyncBackendConnector

public async Task UploadData(IPowerSyncDatabase database)
{
// Get the next transaction to upload
CrudTransaction? transaction;
try
{
transaction = await database.GetNextCrudTransaction();
}
catch (Exception ex)
{
Console.WriteLine($"UploadData Error: {ex.Message}");
return;
}

// If there's no transaction, there's nothing to upload
if (transaction == null)
{
return;
}

// Get client ID if not already retrieved
_clientId ??= await database.GetClientId();

try
{
// Convert PowerSync operations to your backend format
var batch = new List<object>();
foreach (var operation in transaction.Crud)
{
batch.Add(new
{
op = operation.Op.ToString(), // INSERT, UPDATE, DELETE
table = operation.Table,
id = operation.Id,
data = operation.OpData
});
}

// Send the operations to your backend
var payload = JsonSerializer.Serialize(new { batch });
var content = new StringContent(payload, Encoding.UTF8, "application/json");

HttpResponseMessage response = await _httpClient.PostAsync($"{_backendUrl}/api/data", content);
response.EnsureSuccessStatusCode();

// Mark the transaction as completed
await transaction.Complete();
}
catch (Exception ex)
{
Console.WriteLine($"UploadData Error: {ex.Message}");
throw;
}
// Implement UploadData to send local changes to your backend service
// Leave it empty if you only want to sync data from the server to the client
// To pick how to upload writes, see https://docs.powersync.com/handling-writes/write-path/choosing-your-implementation
}
}
```
Expand Down
11 changes: 3 additions & 8 deletions client-sdks/reference/flutter.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -195,15 +195,10 @@ class _DemoAppState extends State<DemoApp> {

### 3. Integrate with Your Backend

The backend connector connects the PowerSync client SDK to your application backend. The SDK uses it to:

1. Get an auth token to connect to the PowerSync instance.
2. Upload client-side writes to your backend API. The SDK places every write to the SQLite database in an upload queue and uploads the queue to your backend when the user is connected. Your backend then applies the changes to the source database.

The connector must implement two methods:
The backend connector gives the SDK the credentials it needs to connect to your PowerSync instance and download data. It also uploads your app's local writes to your backend. The connector must implement two methods:

1. [PowerSyncBackendConnector.fetchCredentials](https://pub.dev/documentation/powersync/latest/powersync/PowerSyncBackendConnector/fetchCredentials.html) - The SDK calls this method to get authentication credentials. It caches the credentials and calls the method again only when needed, for example on the first connection or when the token is near expiry. See [When `fetchCredentials()` is Called](/configuration/app-backend/client-side-integration#when-fetchcredentials-is-called) for details and [Authentication Setup](/configuration/auth/overview) for how to generate credentials.
2. [PowerSyncBackendConnector.uploadData](https://pub.dev/documentation/powersync/latest/powersync/PowerSyncBackendConnector/uploadData.html) - The SDK calls this method whenever it has client-side writes to upload to your backend API. Implement how those writes are processed and uploaded. See [When `uploadData()` is Called](/configuration/app-backend/client-side-integration#when-uploaddata-is-called) for triggers, throttling, and retry behavior, and [Write Path Overview](/handling-writes/write-path/overview) for the app backend implementation.
2. [PowerSyncBackendConnector.uploadData](https://pub.dev/documentation/powersync/latest/powersync/PowerSyncBackendConnector/uploadData.html) - The SDK queues local writes and calls this method to send them to your write API, which applies them to the source database. If your app only reads synced data for now, you can implement this method later. To pick how to upload writes, see [Choosing Your Implementation](/handling-writes/write-path/choosing-your-implementation).

**Example:**

Expand All @@ -229,7 +224,7 @@ class MyBackendConnector extends PowerSyncBackendConnector {

// Implement uploadData to send local changes to your backend service
// You can omit this method if you only want to sync data from the server to the client
// See example implementation here: https://docs.powersync.com/client-sdks/usage-examples#send-changes-in-local-data-to-your-backend-service
// To pick how to upload writes, see https://docs.powersync.com/handling-writes/write-path/choosing-your-implementation
@override
Future<void> uploadData(PowerSyncDatabase database) async {
// This function is called whenever there is data to upload, whether the
Expand Down
Loading
Loading