Temporal.IO in .NET

Back in the day, I used to build monolithic workflows that just relied on a database. It always felt tricky and hard to maintain. But when I moved to a microservices setup, I realized how complicated distributed workflows really are. Handling things like distribution, fault tolerance, and long-running tasks can quickly become a nightmare. That’s where Temporal.io comes to the rescue.

Photo by Kelly Sikkema on Unsplash

What is Temporal?

Defining Temporal is not a simple task. Let’s begin by clarifying what Temporal is not. Temporal isn’t a no-code workflow engine. It’s not an orchestration tool for maintaining microservices. It’s not a general tool like durable functions in Azure.

Temporal is a highly opinionated, open-sourced platform that helps you write durable and distributed workflows in code via either SKDs or low-level gRPC API. If you’re hearing about Temporal for the first time, this information might still seem a bit confusing. Let’s look at the architecture of Temporal first.

Temporal’s architecture

Temporal consists of three components:

Temporal Server
This is the core of Temporal. It is responsible for storing the state of your workflows and activities, scheduling tasks, and executing them. The Temporal Server is open-source software written in Go that can be self-hosted or used as a platform-as-a-service in the Temporal Cloud.

Temporal Worker
A Temporal Worker is a process that executes your workflows and activities. It communicates with the Temporal Server to retrieve tasks and report the results. The Worker utilizes the workflows you have created, functioning much like a runtime for them.

Temporal Client
The Temporal Client is where you initiate and interact with the workflows.

Clients and workers are usually applications developed in your preferred language or platform, running in your chosen host environment, whether in the cloud or on-premises.

Functionality

Temporal primary functionality is centered around running workflows, but it also supports some less common use cases.

In addition to executing workflows, Temporal allows you to send signals to a running workflow to notify it of important events. You also have the ability to cancel or terminate an entire workflow if needed.

Temporal includes a concept called queries, which lets you retrieve the state of a running or completed workflow. You can inspect the data at all steps of the workflow. Additionally, it supports scheduled workflows and a few more edge-case-oriented features.

Workflows vs activities

Temporal.IO defines two key concepts: workflows and activities. Workflows serve as the master blueprint that orchestrates the entire process, outlining the sequence and logic of tasks to ensure reliable and deterministic execution. Activities are essentially those tasks.

A workflow must be:

• deterministic
• idempotent
• free of side effects

Deterministic, idempotent, and free of side effects means that a workflow cannot do things such as:

• Perform IO (network, disk)
• Access/alter external mutable state
• Do any threading
• Do anything using the system clock (e.g. DateTime.Now)
• Use .NET timers (e.g. Task.Delay or Thread.Sleep)
• Invoke any methods that return non-deterministic values, such as Guid.NewGuid().

All of these limitations mean that activities must be a place where the important stuff — like your business logic — happens. Activities can have side effects, be non-deterministic, and maintain state.

This intentional distinction between workflows and activities is not a new concept at all. Just for fun, here are three similar concepts:

• Actor model: workflows are like actors that can communicate with each other, and activities are like the methods of an actor.
• Saga pattern: workflows are like sagas that can orchestrate multiple activities.
• Functional core and imperative shell: workflows are like the functional core of your application, and activities are like the imperative shell.

Temporal and .NET?

Temporal.io provides six official SDKs for various languages, along with three unofficial ones. One of these is the SDK for .NET.

The documentation for .NET is generally good. However, I encountered some difficulties when trying to run the examples locally, and it’s worth noting that examples in Java and Go are more prevalent. Additionally, there are no project-based tutorials available for the .NET SDK, unlike those for Java or Go.

What’s more interesting is the limitation in your code that stems from the requirement for workflows to be deterministic.

Why is deterministic important?

Temporal workflows must be deterministic, meaning they should produce the same results when given the same input, regardless of when or how many times they are replayed.

When a workflow is started, Temporal records every decision and event in an event history, which includes activities like execution, timer starts, and signals received. This event history serves as a comprehensive log of all actions that have occurred in the workflow.

Deterministic means:

• NO side effects
• NO non-deterministic behavior.

Spotting the side effects in programming is generally straightforward, like IO operations. However, non-deterministic behavior presents a more complex challenge. For instance, in workflows, you cannot use methods like Task.Delay or other task-related methods. An interesting example is that you can use Task.Factory.StartNew, but you cannot use Task.Run. This limitation is due to the way Microsoft implemented these two methods. It’s a bit more technical, but if you’re interested in learning more, check out this video discussing the challenges of the Temporal team while building the .NET SDK. I recommend familiarizing yourself with the constraints that ensure workflow determinism described here.

Note:
Since the common Task.Delay method is also one of the forbidden methods, the Temporal team decided to create wrappers for these standard methods. I.e., to ensure that delays are recorded in the workflow’s event history and can be replayed deterministically, you should use the special method called Workflow.DelayAsync instead of the standard Task.Delay method. The same applies to Task.Factory.StartNew or Task.Run, instead of these methods, it’s recommended to use Workflow.RunTaskAsync.

Testing

Since Temporal Server is an external dependency, I wanted to know how easy it is to test.

Activities are easy to test. They are usually a standard class — so nothing fancy here; you can just cover them with unit or integration tests.

On the other hand, integration tests for workflows can be challenging. Temporal simplifies this by allowing you to run tests against a local Temporal Server.

// Start local dev server
await using var env = await WorkflowEnvironment.StartLocalAsync();

// Create a worker
using var worker = new TemporalWorker(
    env.Client,
    new TemporalWorkerOptions($"task-queue-{Guid.NewGuid()}").
        AddWorkflow<SayHelloWorkflow>());

// Run the worker only for the life of the code within
await worker.ExecuteAsync(async () =>
{
    // Execute the workflow and confirm the result
    var result = await env.Client.ExecuteWorkflowAsync(
        (SayHelloWorkflow wf) => wf.RunAsync("Temporal"),
        new(id: $"wf-{Guid.NewGuid()}", taskQueue: worker.Options.TaskQueue!));
    Assert.Equal("Hello, Temporal!", result);
});

End-to-end (E2E) tests for workflows that take a significant amount of time to complete — such as those lasting an hour or more — are usually not easy to test. Temporal.IO offers time skipping, which allows you to skip time in your workflows, making it easier to test long-running workflows. It’s handy for integration tests, even for some E2E2 tests, but it’s still not feasible if you want to test the entire system (usually via smoke tests).

In these cases, I recommend focusing on observability and setting up alerts to notify you when issues arise. Fortunately, Temporal.io offers built-in observability through its user interface and provides various metrics that tools like Prometheus can scrape and use.

Data Confidentiality

A somewhat overlooked aspect of replayability is that the data passed to the workflow gets stored in the Event History of your Workflow Executions. Sometimes, it’s better to encrypt the data as it enters the Temporal Service and then decrypt it upon exit. By default, Temporal doesn’t provide any encryption mechanism. You need to take care of it yourself. Read more here.

The reason why I’m talking about this is that you may fail to notice the duality of data in an activity.

• Variables that aren’t part of the input or output of the activity are absolutely safe. Temporal doesn’t log your local variables anywhere.
• On the other hand, returning or sending confidential data to activities may cause security/privacy issues, as input and output data are stored in plain text format in Temporal logs. So be careful.

Versioning

When updating your existing workflow, there are multiple things you need to take into consideration:

• There might be multiple workers running at the same time. This means that the old workflow in the queue can be picked up by the new version of the worker.
• Should the workflow that has already started be finished using the old version of the workflow code?
• When a workflow is upgraded to a new version, the workflow history from previous executions (based on the old version) still exists. If the new version of the workflow tries to replay this history and encounters discrepancies, errors will occur.

I can think of two main strategies to address these issues:

Backward compatible workflows

Ensure that your workflows and activities are backward compatible. You can utilize a handy Patched method.

if (Temporalio.Workflows.Workflow.Patched("my-patch-01"))
{
  // Logic for new version
}
else
{
  // Logic for old version
}

Workflow.Patched("MyPatchId") will return true for any workflow execution after deploying the patch.

Task queues

When making all activities and workflow backward compatible is difficult, you can utilize task queues to route the tasks to the correct workflow version.

var client = new WorkflowClient(…);
var workflow = client.NewWorkflowStub<IMyWorkflow>(
new WorkflowOptions { TaskQueue = "task-queue-v2" }); // task-queue-v1
await workflow.MyWorkflowMethod();

This allows you to run two versions of the workflow simultaneously and completely separately.

Alternative way

There is also another way to deal with changes in workflow. For short-lived workflows, you can stop old workers before starting new ones with updated workflow definitions. Temporal simplifies this with its persistence layer: you can stop all workers and start new ones, and any new workflow requests will be stored in the Temporal database, ready for processing once the new workers are operational.

What about old, long-running workflows?

Some workflows can run for months, making backward compatibility essential. The alternative is to stop the old workflow, potentially leaving the result incomplete, and then initiate a new workflow that can build upon that partial data. There are countless options to manage these situations.

Just keep in mind that when you begin working on workflows, ask questions such as: “What happens if there’s a change in requirements”?

Activities

In Temporal.io, activities are the units of work in a Temporal workflow that represent external tasks or operations. They are designed to handle the actual business logic, computation, or I/O tasks that cannot (or should not) be performed directly inside a workflow due to workflow constraints, such as long-running or blocking operations.

However, activities can be expensive. When Temporal starts an activity, multiple things happen:

• the activity is scheduled,
• executed
• the result is returned.

All of this is done in a distributed manner involving network calls and data transfer. To reduce this overhead, Temporal introduced Local Activity.

Local Activity

Local Activity is a way to run activities in the same process as the workflow. It’s faster and cheaper. For example, you can use Local Activity to generate a random number or a new unique identifier.

public class GenerateGuidActivity
{
    [Activity]
    public Task<Guid> GenerateId()
    {
        return Task.FromResult(Guid.NewGuid());
    }
}


...
// optional if you have a reasonable workflow execution timeout
var localActivityOptions = new LocalActivityOptions
{
    ScheduleToCloseTimeout = TimeSpan.FromSeconds(5)
};

var id = await Workflow.ExecuteLocalActivityAsync(
(GenerateGuidActivity activity) => activity.GenerateId(), 
localActivityOptions);

As you can see, a local activity is the same as a normal one, but you need to call it via ExecuteLocalActivityAsync method.

No such thing as a free lunch

Temporal is not a silver bullet. It has its own limitations and constraints. One of the biggest challenges is its complexity. As a cloud-native solution, Temporal aims to follow cloud-native architectural principles. However, a distributed system can fail in various ways, making debugging and understanding the internal workings quite difficult.

If you’re looking for a simple workflow engine, Temporal may not be the right choice for you. Temporal truly excels in scenarios involving distributed workflows, where multiple microservices need to communicate with one another.

On a positive note, Temporal.io can be used for free. It is an open-source platform, allowing users to download, use, and host the Temporal server and SDKs under the MIT License without any cost.

Update: Thanks to feedback from Chad Retz, a primary author of several Temporal.io SDKs, I’ve clarified some points in this post. Appreciate your insights, Chad!