This article describes the conditions under which Orchid Fusion VMS may report a Server Error via its Email Notifications system, and how an administrator can troubleshoot and resolve these errors.

Background

Orchid Fusion VMS may be configured to send Email Notifications based on Server Error triggers.  See Notifications Overview for information on Notifications in Orchid Fusion VMS and other types of triggers.

Server Errors

If so configured, Orchid Fusion VMS will send a Server Error email notification for any of the following reasons:

Error: Server Degraded

Server Degraded errors occur when an Orchid Core VMS recording server enters a degraded state.  In this mode, Orchid Core VMS is no longer capable of recording video and will show a bright orange warning  and error message when logging in:

The reason for Orchid Core VMS being in a degraded state is provided in both the Orchid Fusion VMS Notification email, and when logging in to the Orchid Core VMS recording server itself:

• Storage device changed detected: Device tag not present:
  
  
  When Orchid Core VMS starts for the first time, it places a unique file or tag in the directory where it has been configured to store video archives.  If that tag disappears (due to, for example, a RAID volume becoming degraded or a network drive becoming disconnected), Orchid Core VMS will generate a storage error and enter its degraded state.  No new video archives can be recorded until the storage error is corrected.  This storage error can be resolved by:
• Restoring the storage device so that the file {configured-storage-path}/{orchid-id}/orchid-device.tag is again accessible.  This will typically involve fixing an offline RAID volume or reattaching a disconnected network drive.
• Resetting Orchid Core VMS's storage path so that it uses a new storage location.  Note that this option will effectively abandon any video archives that were written to the old storage location which is no longer accessible, and make it so that those video files can no longer be viewed in the Orchid Core VMS or Orchid Fusion VMS interfaces.  To reset Orchid Core VMS's storage path, change the value of archive.dir in the Orchid Core VMS server properties file and restart Orchid (see the Orchid Core VMS Administrator Guide for information on manipulating the properties file and restarting services).  Note that if you need to regenerate a new device tag in the same storage location that was used previously, it may be necessary to specify a temporary storage location (such as /tmp or C:\Temp), restart Orchid Core VMS, and then specify the final storage location, and restart Orchid Core VMS again.
  
  
• Invalid License:
  
  
  Each Orchid Core VMS server's license is tied to specific hardware characteristics of the server on which its license was activated.  If those hardware characteristics change beyond a certain threshold, the license will be marked as invalid and your Orchid Core VMS recording server will enter a degraded state.  Typically, this occurs when system hardware such as a motherboard or OS hard drives have been replaced.  This may also occur when changes are made to a VM host which is running Orchid Core VMS. When this occurs, you will need to either:
• Restore your Orchid Core VMS server's original hardware configuration, or
• Contact IPConfigure Technical Support who can reset your original license activation code so that it may be reapplied using your server's new hardware configuration. Note that IPConfigure Technical Support can and will reset your server license even if your support agreement is no longer valid.
  
  
• Time-Limited License Expired:
  
  
  Orchid Core VMS recording servers may be configured with time-limited licenses (listed on the Orchid Core VMS Activation Page as edition core-timed or fusion-timed).  These licenses are typically issued for testing and evaluation purposes, and cause Orchid Core VMS to enter a degraded state after they expire.  To correct this condition, contact your IPConfigure Salesperson to purchase a standard Orchid Core VMS license (which never expires).

Error: Server Video Files Missing

By default, Orchid Core VMS records video archives in one minute clips.  If an Orchid Core VMS recording server has not written a video file for a camera configured for continuous recording in over three minutes, a Server Video Files Missing notification will be triggered (if configured).

This condition most commonly occurs when cameras are experiencing significant packet loss over the network connection between the camera and the Orchid Core VMS recording server.  Cameras in this state will frequently show significantly reduced disk storage on the Orchid Core VMS System Report page when compared to other "healthy" cameras.  Even though video is being recorded, video frames are being dropped:

To correct this condition, system administrators should first verify packet loss between the camera and the Orchid Core VMS server using a third-party tool such as VLC.  This procedure is described in the Orchid Fusion VMS Administrator Guide under the section Advanced Topics -> Checking for Packet Loss.  Once packet loss has been identified, it can be typically be resolved by increasing the bandwidth between the affected camera and the Orchid Core VMS recording server, or by reducing the camera's resolution, framerate, and/or bitrate.

Error: Server Restart

This notification alerts administrators that the Orchid Core VMS service has restarted.  This may not always indicate an error, as the service may have restarted due to a software upgrade or hardware maintenance.  However, unexpected service restarts may occur for a variety of reasons:

• Clock change: If a server's clock changes significantly, the Orchid Core VMS service will restart in order to resynchronize itself with the timing information provided by the camera streams it is recording.  
• Memory exhaustion: If Orchid Core VMS requires more memory than is available, the service will be terminated by the operating system.  This condition typically occurs when a server's hardware is insufficiently matched to the quantity, framerate, and/or resolution of camera it has been asked to record.  It may also occur when a very slow video storage volume forces Orchid Core VMS to buffer large amounts of video in memory while it waits to write them out to disk.  
• Library conflicts and other unusual issues: The Orchid Core VMS service may also unexpectedly restart due to library conflicts, corrupted installations, or a variety of other unusual situations.  The Orchid Core VMS server and operating system logs may indicate the source of the error, and customers with a support agreement are encouraged to contact IPConfigure Technical Support for assistance in troubleshooting unexpected service restarts.