This article describes the conditions under which Orchid Fusion VMS may report an I/O Error via its Email Notifications system, and how an administrator can troubleshoot and resolve these errors.

Background

Orchid Fusion VMS may be configured to send Email Notifications based on I/O Error triggers.  See Notifications Overview for information on Notifications in Orchid Fusion VMS and other types of triggers.

Administrators who are experiencing I/O Errors and have a support agreement are encouraged to contact IPConfigure Technical Support for assistance in diagnosing and resolving these errors.

I/O Errors

If so configured, Orchid Fusion VMS will send an I/O Error email notification if video files cannot be written to disk quickly enough.  This condition may occur due a network volume with insufficient throughput, virtualized storage configured on a VM, or a degraded RAID volume.  

How Orchid Core VMS Saves Video

In order to understand why I/O Errors occur, it is helpful to understand how Orchid Core VMS records video.  Orchid Core VMS saves video from every camera that it's configured to record in one minute clips.  In order to guarantee that a slow storage device can't interfere with system availability, Orchid Core VMS implements timeouts when writing out these video clips.  If a video clip cannot be written within these timeouts, Orchid Core VMS is forced to discard the clip in order to continue receiving the flow of new video coming from the camera.  Orchid implements three separate limits:

• Operation timeout: every individual file operation -- such as opening a new video file, writing a chunk of data, or closing a file -- is by default given 30 seconds to complete.
• Finalize timeout: after Orchid Core VMS has finished with a one minute video clip, it needs to confirm with the operating system that the file has been committed to disk before Orchid saves a record of this file in its own database.  By default, Orchid Core VMS wait 500ms for this finalization to happen.
• Concurrent file limit: if file operations become backed up, Orchid Core VMS will allow the previous minute's or minutes' files to stack-up while the current minute's file is being written.  By default, Orchid Core VMS will attempt to write out at most two files for a single camera simultaneously.

Diagnosing Slow Disk Performance

Both Windows and Linux provide tools for diagnosing storage device throughput.  A typical troubleshooting workflow for storage errors is:

1. Use the Windows Resource Monitor or sudo iotop -P  on Linux to establish a server's current write throughput.
2. Decrease camera resolution and framerate in order to establish a baseline disk throughput that does not generate I/O Errors alerts.
3. Increase camera resolution and framerate until write throughput no long increases (indicating that the maximum available throughput has been reached), or until I/O Errors resume.

To gather disk throughput metrics on Windows, the winsat command may be used:

winsat disk -seq -write -drive [DRIVE LETTER]

See: Use WinSAT and determine how fast my hard disk is!

To gather disk throughput metrics on Linux, the fio command may be used:

fio --randrepeat=1 --ioengine=libaio --direct=1 --gtod_reduce=1 --name=test \
--filename=random_read_write.fio --bs=4k --iodepth=64 --size=250M \
--readwrite=randrw --rwmixread=75

More information: How to measure disk performance with fio and IOPing

Increasing I/O Timeout Thresholds

Orchid Core VMS's I/O timeouts use default values that attempt to preserve maximum system availability even on low-powered, low-memory systems.  On systems that are recording a large number of cameras (e.g., 100 or more) or whose current memory footprint with Orchid Core VMS running is relatively low (e.g., under 50%), it is typically safe to increase these default values in order to reduce the occurrence of I/O Errors at the expense of increased memory usage.  For example, the following Orchid Core VMS properties file settings are appropriate for enterprise environments where Orchid Core VMS's memory footprint is under 50%:

filewriter.max.finalize.duration = 90000
filewriter.max.op.duration = 90000
filewriter.max.threads = 3

For information on modifying the Orchid Core VMS properties file and restarting the service to apply those changes, please see the Orchid Core VMS Administrator Guide.

Administrators with a valid support agreement are encouraged to contact IPConfigure Technical Support for assistance in adjusting Orchid Core VMS I/O timeout thresholds in order to eliminate I/O Errors.