NetFlow Optimizer Troubleshooting

Introduction

NetFlow Optimizer (NFO) software keeps track of its activity by logging to various files in $NFO_HOME/logs (NFO_HOME default is /opt/flowintegrator).

The NFO internal log files are rolled based on size. Ten last files are kept in $NFO_HOME/logs/bak directory. You can change the default log rotation size and the number of “bak” files to keep by editing $NFO_HOME/server/etc/server.cfg.

Internal logs

Here is a list of the internal logs in $NFO_HOME/logs. The internal logs from NFO software are useful for troubleshooting or monitoring the health of the system.

Log file name

Description

nfc_server. .<timestamp>.log

Contains information about NFO core process and enabled NFO Modules and errors detected

server.<timestamp>.log

Contains information when NFO core process was started as well as redirected process standard output

process.log<.#>

Contains information about CPU and memory usage

nf2sl.log<.#>

Contains information about NFO controller component (including keep alive) and errors detected

localhost_access_log.<date>.txt

Contains information about requests to the GUI (including IP address and access time)

catalina.#.log

Contains information about Tomcat process and errors detected

catalina.out

Contains Tomcat process redirected standard output

Log levels

NFO logging levels are (from least to most verbose) Error, Debug, Verbose, and Flood. Default level is Error.

You can change the log level by navigating to NFO > Advanced > Server tab and selecting a Tracing verbosity level you need. Press <Save>. Do not restart NFO.

Troubleshoot drops with NFO logs

If you see drop counts at NFO > Status, you can find more information in NFO logs. The following table contains description and general explanation for packet drops.

Drop

Description

dropped by input threads

Total number of packets received by NFO and dropped because they did not pass basic validation tests, e.g. packets are not one of known flow format – NetFlow v5/v9, sFlow, IPFIX, etc. Packets can also be dropped by input threads if NFO is unable to queue them for subsequent processing, e.g. out of memory, queue overflow, etc.

dropped by work threads

The number of packets dropped when flow records is processed by Modules, e.g. there is no NFv9 or IPFIX Template for the flow record, or when there are other problems when processing flows, or when processed flows could not be placed in the Output queue. A small number of these drops is expected when NFO is restarted, while Templates are not yet received.

dropped by kron thread

The number of packets dropped by “Data collection interval” triggers caused by queue overflow.

dropped at output

The number of packets dropped by NFO due to Output queue overflow.

dropped by QoS

The number of packets dropped by NFO internal Quality of Service mechanism to avoid congestions. These drops are also included in one of the drops statistics above.

 

This section contains additional information for each type of drops.

Dropped by input threads

Problem

What to look for / Recommendation

Unknown flow format

Debug level:

"Input thread %d: NFv5 packet %u failed sanity check: %d\n"

"Input thread %d: NFv9 packet %u failed sanity check: %d\n"

"Input thread %d: IPFIX packet %u failed sanity check: %d\n"

"Input thread %d: sFlow packet failed sanity check: %d\n"

"Input thread %d: src addr: %s P2 FDR id: %u size: %u\n"

 

Flood level:

"Input thread %d: UNRECOGNIZED: src addr: %s size: %u\n"

 

Recommended action: Check if UDP packets sent to NFO are one of the supported flow protocols: NFv5/v9, IPFIX, sFlow, FDR P2

NFO is unable to read packet from NFO input port

Debug level:

"Input thread %d: receiver failure %d\n"

Recommended action: Check health of the sockets opened by NFO for input ports

Insufficient memory when NFO reads packet from NFO input port

Error level:

"Input thread %d: failed to allocate buffer for derived objects\n"

Recommended action: Check free RAM availability on NFO host

Problem related to templates for NFv9 or IPFIX data records

Debug level:

"TFS CHECK: pkt longer than size in hdr: %u expected %u\n"

"TFS CHECK: pkt shorter than size in hdr: %u expected %u\n"

Recommended action: Check that all incoming NFv9/IPFIX packets meet specification

Problems when placing packets from input thread to work thread after validation. You may also get +1 in dropped by QoS if RED queue is full

Error level:

"Input thread %d: worker thread %d QoS queue failure %d\n"

Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available

Verbose level:

"Queue is full.\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

 

Dropped by work threads

Problem

What to look for

Problems when placing packets from work thread to output thread

Error level:

"Work thread %d: output thread %d QoS queue failure %d\n"

Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available

Debug level:

"WT %d: failed to enqueue to output thread: %d\n"

"WT %d: failed to enqueue to output thread: %d\n"

Recommended action: Increase output threads count or configure output using faster network interface if available

Received discard status for a packet while processing it

Verbose level:

"Work thread %d: discarded message by status\n"

Recommended action: Check if UDP packets sent to NFO are one of the supported flow protocols: NFv5/v9, IPFIX, sFlow, FDR P2. Check if all incoming data has corresponding templates (NFv9/IPFIX). Check if incoming templates meet minimal input fields requirements for enabled Modules.

Problems when placing packets from one work thread to another work thread. You may also get +1 in dropped by QoS if RED queue is full

Error level:

"Worker thread %d: QoS queue failure %d\n"

Recommended action: Check free RAM availability on NFO host or report a bug if sufficient memory is available

Debug level:

"WT %d: failed to enqueue derived message to work thread qos queue: %d\n" и

"WT %d: failed to enqueue derived message to work thread qos queue: %d\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Work thread overflow

Debug level:

"Work thread %d: failed to enqueue derived message: queue full\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

 

Dropped by kron thread

Problem

What to look for

Problems when placing packets from kron thread to work thread. You may also get +1 in dropped by QoS if RED queue is full

Error level:

"KRON EVAL: %d: failed to enqueue message to work thread QoS queue: %d\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

Work thread overflow

Verbose level:

"Queue is full.\n"

Recommended action: Increase worker threads count (and increase the number of cores/processors if needed) or disable unused Modules.

 

Dropped at output

Problem

What to look for

Output thread overflow

Verbose level:

"Queue is full.\n"

Recommended action: Increase output threads count or configure output using faster network interface if available

 

Dropped by QoS

Problem

What to look for

Drops caused by NFO RED (https://en.wikipedia.org/wiki/Random_early_detection) implementation

These drops are included in one of the drops described before. There are no special logs for these drops.

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.