NiFi hands-on examples

The NiFi hands-on examples page demonstrates how to use NiFi.

​

Managing data flows

This section outlines the basic workflow for creating, configuring, and managing data flows.

​

Create a new flow

To create a new flow, you have to add components to the canvas by doing the following:

1. Open the NiFi UI and navigate to the canvas.
2. Use the toolbar to select the component you want to add. This can be any of the following:
   • Processor
   • Process group
   • Input/Output port
   • Funnel
3. Drag the component onto the canvas and position it as needed.

​

Configure processors

Each processor defines a specific action within a flow. To configure a processor do the following:

1. Right-click or control-click a previously added processor, and then select Configure.
2. Set required properties, such as the connection URLs, file paths, or format settings.
3. Adjust Scheduling options such as the run frequency or concurrent tasks.
4. Review and apply any relationships that determine FlowFile routing after processing.

​

Connect components

Connections define the path that data follows. To connect two or more components, do the following:

1. Hover over a processor’s arrow and drag it to another component to create a connection.
2. Configure the connection to specify the following:
   • Accepted relationships
   • Back-pressure thresholds
   • Prioritization or load-balancing options

Connections serve as queues, buffering FlowFiles between components.

​

Using variables

Variables help simplify configuration and improve maintainability. Do the following when using a variable:

1. Define variables at the process group level.
2. Reference variables in processor properties using ${variable_name} syntax.

These allow the management of common values such as the directory paths, hostnames, or credentials references in one place.

​

Enable and use controller services

Controller services provide shared configurations used by multiple processors. It also helps avoid redundant configuration and ensure consistency. To enable and use controller services, do the following:

1. Open the controller services tab by right-clicking or control-clicking an empty canvas or a process group and selecting Controller Services.
2. Add and configure a service, such as a database connection pool, record reader, or SSL context.
3. Enable the service.

After completing the steps, processors, depending on the service, automatically detect and use it.

​

Start and stop components

NiFi allows granular control over component execution. This includes the following:

• Start: Begin execution of a processor or process group.
• Stop: Halt execution without removing configuration.
• Enable or Disable: Control whether a component can run.
• Start or stop process group: Apply actions to all components within the group.

Before starting a flow, ensure you enable all required services and verify if all components are valid.

​

Additional options for managing a data flow

Effective flow management helps ensure reliable operation and simplifies debugging and maintenance. Several options help manage and maintain dataflows. This includes:

• Status bar: Observe the status bar for component states, errors, and back-pressure.
• Data provenance: Use data provenance to trace the lifecycle of FlowFiles.
• Bulletin board: Check the bulletin board for warnings and error messages.
• Templates or versioned flows: Apply templates or versioned flows for reusability and consistent deployment.

​

Managing processors

Processors are the core building blocks of NiFi dataflows. Each processor performs a specific operation on FlowFiles, such as ingesting data, transforming content, routing, or delivering data to a destination. While NiFi provides hundreds of processors, some are frequently used across a variety of flows. This section highlights commonly used processors, grouped by their typical functions.

​

Data ingestion processors

These processors ingest data into NiFi from various sources. One example is the GetFile processor. To configure the GetFile data ingestion processor, do the following:

1. Drag a processor component onto the canvas.
2. Search for the GetFile data ingestion processor, and click Add.
3. Right-click or control-click the processor, and then select Configure.
4. Set the Input Directory property to the folder containing files to ingest, then click Apply.
5. Connect it to the next processor in your flow.
6. Start each processor to begin pulling files into NiFi.

Other examples of data ingestion processors include:

• GetS3Object/ListS3: Retrieves objects from Amazon S3
• ConsumeKafka/ConsumeKafkaRecord_2_0: Consumes messages from Kafka topics
• ListenHTTP/HandleHTTP: Accepts data via HTTP requests
• QueryDatabaseTable/ExecuteSQL: Reads data from relational databases

​

Data transformation processors

These processors modify or enrich the content or attributes of FlowFiles. One example is the UpdateAttribute processor. To configure the UpdateAttribute data transformation processor, do the following:

1. Drag a processor component onto the canvas.
2. Search for the UpdateAttribute data transformation processor, and click Add.
3. Right-click or control-click the processor, and select Configure.
4. Add a new attribute, then click Apply.
5. Connect it to another processor.
6. Start each processor to modify the FlowFile attributes.

Other examples of data transformation processors include:

• ReplaceText/ReplaceTextWithMapping: Performs string replacements or regular expression transformations
• JoltTransformJSON: Transforms JSON data using Jolt specifications
• AttributesToJSON/AttributesToXML: Converts FlowFile attributes to structured content
• ConvertRecord/ConvertAvroToJSON/ConvertJSONToCSV: Converts data between formats

​

Data routing processors

These processors determine the direction of FlowFiles through a flow. One example is the RouteOnAttribute processor. To configure the RouteOnAttribute data routing processor, do the following:

1. Drag a processor component onto the canvas.
2. Select the RouteOnAttribute data ingestion processor, and click Add.
3. Right-click or control-click the processor, and then select Configure.
4. Configure a routing strategy.
5. Connect it to another processor producing FlowFiles.
6. Start each processor to route FlowFiles based on attributes.

Other examples of data routing processors include:

• RouteOnContent: Routes FlowFiles based on content patterns
• RouteOnRelationship: Routes FlowFiles based on processing outcomes
• MergeContent/MergeRecord: Combines multiple FlowFiles into one, based on content or schema

​

Data delivery processors

These processors send data to external systems. One example is the PutFile processor. To configure the PutFile data delivery processor, do the following:

1. Drag a processor component onto the canvas.
2. Select the PutFile data ingestion processor, and then click Add.
3. Right-click or control-click the processor, and then select Configure.
4. Configure the Directory property where FlowFiles write to.
5. Connect it to another processor with a custom configuration.
6. Start each processor to deliver FlowFiles to the target directory.

Other examples of data delivery processors include:

• PutS3Object: Uploads FlowFiles to Amazon S3
• PutDatabaseRecord/PutSQL: Writes data into relational databases
• PublishKafka/PublishKafkaRecord_2_0: Sends messages to Kafka topics
• PutHDFS/PutHBase: Writes data to Hadoop or HBase

​

Utility and monitoring processors

These processors provide operational or auxiliary functions. One example is the LogAttribute processor. To configure the LogAttribute utility and monitoring processor, do the following:

1. Drag a processor component onto the canvas.
2. Select the LogAttribute utility and monitoring processor, and then click Add.
3. Right-click or control-click the processor, and then select Configure.
4. Add a custom configuration.
5. Connect it to another processor with a custom configuration.
6. Start each processor to log FlowFile attributes for monitoring and debugging.

Other examples of utility and monitoring processors include:

• GenerateFlowFile: Creates test data
• UpdateRecord: Updates record fields using schema-aware operations
• ValidateRecord: Validates data against a schema or rules
• ControlRate: Throttles the FlowFile flow rate

​

Controller services and shared resources

A controller service is a NiFi component that exposes a common capability or configuration to other processors. Examples include:

• Database connection pools
• SSL/TLS context services for secure connections
• Record readers and writers for structured data such as CSV, JSON, or Avro
• Schema registries for data validation and transformation

Processors reference enabled controller services instead of duplicating configuration settings individually.

​

Add and configure a controller service

You can add and configure a controller service by doing the following:

1. On an empty canvas or process group, right-click, or control-click and then select Controller Services.
2. Click the plus sign at the left to add a controller service.
3. Search for a service type, select it, and click Add.
4. Click the three dots at the right, and then select Edit to configure the service type properties.
5. Click Apply to save your edit.

​

Manage a controller service state

Controller services have an operational state that controls whether processors can use them. You change this state by doing the following:

1. Add a controller service
2. On an empty canvas or process group, right-click, or control-click, and then select Enable or Disable.

Any processor that references the service automatically detects and uses it.

​

Monitoring, logging and troubleshooting

Effective monitoring and troubleshooting are essential to ensure NiFi dataflows operate reliably. NiFi provides a range of tools and features to observe flow activity, diagnose issues, and maintain operational visibility.

​

Monitor the flow

You can view a processor and its connection status by taking the following steps:

1. Observe the status bar on processors and connections.
2. Identify running, stopped, or invalid components.
3. Identify back-pressure indicators on connections.

Connections act as queues between processors. You can inspect a queue by doing the following:

1. Locate a connection between two components.
2. View the queued FlowFile count and data size displayed on the connection.
3. Right-click or control-click the connection to view additional queue details.

NiFi generates bulletins for processors reporting warnings or errors. You can view bulletins by doing the following:

1. At the top right corner of the NiFi page, click the three lines and select Bulletin Board.
2. Review the generated bulletins.

​

View data provenance

NiFi tracks the full lifecycle of each FlowFile so it can provide visibility into how data moves through a flow. You can view data provenance by doing the following:

1. At the top right corner of the NiFi page, click the three lines and select Data Provenance.
2. Select a specific event to view its details or see its lineage.

​

View logs

NiFi logs events and system activity to files for operational analysis. You can view these logs by doing the following:

1. Access the NiFi node.
2. Navigate to the NiFi logs directory.
3. Open log files to inspect runtime activity.
4. Review entries categorized by log level, such as DEBUG, INFO, WARN, or ERROR.

​

Troubleshooting tips

Use the following actions to inspect and resolve issues in a NiFi flow:

• Check processor status and bulletins: Quickly identify processors that are failing or misconfigured.
• Inspect logs: Look for errors, exceptions, or connectivity issues in the NiFi app logs.
• Monitor queue sizes: Large queues may indicate bottlenecks or downstream performance issues.
• Review data provenance: Trace the path of individual FlowFiles to understand where problems occur.
• Test with small data samples: Use test FlowFiles to isolate and debug flow logic without processing large volumes of data.
• Validate configuration: Ensure processors, controller services, and connections are properly configured and enabled.

​

Additional resources

• To get an overview of NiFi, refer to the NiFi in NexusOne page.
• To learn about best practices when using NiFi, refer to the NiFi best practices page.
• For more details about NiFi, refer to the NiFi official documentation.