Understanding Retention and Disk Utilization

Table of Contents

• Background
• Orchid Core VMS Retention Policy
• Factors Affecting Disk Utilization
• Insuring Sufficient Retention
• Common Misconceptions
  
  

Background

Orchid Core VMS's primary responsibility is to store video from surveillance cameras to a storage medium for later playback and analysis.  This storage medium may be a hard disk drive, a RAID volume, a network share, or something else entirely—any storage medium that can be written to using a normal file system path (e.g., X:\some\location\ on Windows or /mnt/some/location/ on Linux) can be used by Orchid.

When Orchid's configured storage medium no longer has enough space left for new video, Orchid deletes previously recorded video in order to make space for new video according to the rules described below.

Orchid Core VMS Retention Policy

By default, Orchid Core VMS attempts to retain the same duration of video per configured camera, regardless of much space each individual camera uses on disk.  Consider the example below:

Here we see a server with four cameras configured, and even though each camera takes up significantly different amounts of space on disk (between less than 2GB to over 24GB per day of recorded video), Orchid always deletes the oldest available videos when it needs to free up space.  In this way, every camera gets the same duration of retention.  This behavior, in which every camera receives the same duration, is Orchid's Automatic retention policy.

If you have a camera for which you must retain video longer than the estimated Automatic retention, you can set a per-camera retention override by clicking the "AUTOMATIC" link in the Estimated Retention table.  You can now set an override that allows a camera to record for a longer duration than it would ordinarily receive under the Automatic retention policy.  In order to satisfy this override, when Orchid needs more space it will delete video in the following order:

1. Delete video from cameras with a retention override that is older than both the "Desired Retention" and also older than the oldest available video for cameras using the Automatic retention policy.
2. Delete the oldest video for cameras with an Automatic retention policy.

As long as enough disk space is available to meet the desired retention on cameras for which a retention override is specified, the cameras with a retention override will always have the requested amount of video stored, while cameras with an Automatic retention policy will get an equal share (by duration) of the remaining storage space.

Compare the example below to the previous screenshot: the camera Cort's Deck now has a retention override of 50 days.  In this example, by requiring that Orchid always store 50 days of video from the camera Cort's Deck, the remaining available storage space is estimated to leave a duration of 5.7 days for the other cameras.

IMPORTANT: Setting a retention override on all cameras has no effect, and does not guarantee that all cameras will meet the specified retention.  Since there are no cameras configured for Automatic retention in this scenario, there is nothing Orchid can do to free up additional space to meet the desired retention.  See the section "Common Misconceptions" below.

Orchid's retention override mechanism is designed to accommodate scenarios in which a limited number of "high risk" or "high value" cameras need an extended duration of video.  For example, a small number of cameras covering a store's cash register and office safe could be set to a desired retention of 60 days, while there may be enough disk space available for the remaining cameras to have 30 days of video on disk.

Remember: Orchid's retention policy governs the order in which old video is deleted to make room for new video—it has no control over how much space each camera requires on disk.  For that, see the section below.

Factors Affecting Disk Utilization

Orchid Core VMS saves video to disk exactly as it receives that video from your cameras; this behavior is critical for the forensic integrity of surveillance data.  As a result, the amount of storage used on disk for a given minute of recorded video is controlled almost entirely by the camera configuration.  The following screenshot of the Orchid Core VMS settings dialog for an ONVIF camera highlights the settings which can affect the amount of storage used by a camera:

Observant readers will note that this includes "pretty much everything."  Generally speaking, if you have specific camera retention goals in mind, you should configure your camera settings according to values provided by your camera manufacturer or VMS vendor's camera/storage design tool.  However, in order to better understand the effect of each of these settings, see the brief summary below:

• Encoder: controls how your video is encoded.  For the best compression and compatibility, choose H264.
• Bit Rate: is in some ways the only setting that really matters, as it specifies how many kilobits per second (kbps) your camera uses when compressing video.  Note that this setting must be made with all the other settings, like framerate and resolution, in mind.  If your Bit Rate is aggressively low, your video will be heavily compressed and lacking in detail and quality.
  • Cameras typically treat Bit Rate in one of two ways: Constant Bit Rate (CBR) or Variable Bit Rate (VBR).  This behavior is often configurable through the camera's web interface.
    • Variable Bit Rate: mode causes the Bit Rate setting in Orchid to act as a maximum.  The actual Bit Rate produced by the camera will vary over time and may be less depending on scene complexity, lighting, etc. It will never go above your specified Bit Rate, however.  Retention estimate calculations made using VBR will tend to be overestimates (Orchid will tend to achieve more retention than was estimated).
    • Constant Bit Rate: mode causes the Bit Rate setting in Orchid to be exact, regardless of scene complexity.  Retention estimate calculations made using CBR will be smaller (less video retained) but more exact.
  • To achieve a rough estimate of a camera's disk usage, you can convert between kilobits per second ("Bit Rate") and Gigabytes per day by multiplying by 0.0108.  Note that in some circumstances the Bit Rate may be too low to be achievable, or so high that even with the minimum available compression the effective Bit Rate will be lower.
  • Certain cameras, such as in the example above, support a Bit Rate of "0".  This specifies a dynamic setting, in which the camera will increase the Bit Rate when, for example, there is a lot of activity or motion, and decrease the Bit Rate during inactivity.  For many users this is a valuable behavior, but it often means taking a "wait and see" approach to calculating the effect of camera settings on disk usage.
• Frame Rate: controls the the number of image frames per second in your video.  With all other settings being equal, your disk usage increases linearly with your frame rate.  For example, tripling frame rate will triple disk usage.
• GOV Length: governs the size of your camera's "Group of Video," so called because H.264 video is stored in a group of images consisting of a single "I-frame" and multiple "change-frames."  The I-frame is similar to a JPEG image, and describes an entire video frame.  Change-frames, on the other hand, describe only the differences in a video frame when compared to the most recent I-frame.  Since video images typically have similar background details when compared frame-by-frame, this "Group of Video" approach is responsible for much of the space-saving ability of H264.  Since I-frames are larger than change-frames, a large GOV Length will reduce the amount of disk space used by a camera.  
  • Orchid's Low-Bandwidth Mode displays only I-frames for live video.
  • Packet loss due to network errors will often result in video that's unusable until the next available I-frame. 
  • For the above reasons, we recommend that your GOV Length be set to a value between the Frame Rate and twice the Frame Rate (i.e., an I-frame every 1 to 2 seconds).
• Profile: describes the number of H264 compression "features" used by your camera.  Lower profiles are simpler to decode but result in larger video.  Given the power of modern computing hardware, Profile should always be set the highest available setting.
• Resolution: sets the size of a video in pixels.  Required storage space increases approximately linearly with the number of configured pixels.  For example, 1080p video is approximately 2 megapixels (1920 x 1080 ≈ 2 million), and 720p video is approximately 1 megapixel (1280 x 720 ≈ 1 million).  As a result, 1080p video requires around twice as much storage as 720p video.
• Quality: is a setting that varies by manufacturer.  For many manufacturers, Quality affects relative Bit Rate when the Bit Rate value is dynamic rather than fixed, as described above.
• Recording Style: is an Orchid-specific setting.  While all of the above settings apply equally for all Recording Styles, the "Recording only when motion detected" style will reduce required camera storage in a way that's unpredictable since it's governed by how much motion your camera sees.

Beyond the camera settings above, other factors can affect video size:

• Scene complexity: busy, high-activity scenes will consume more storage space than low-activity scenes.
• Sensor noise and light level: camera sensors produce added image noise at low light levels.  Some cameras designed exclusively for lighted, indoor use can generate extreme amounts of sensor noise.  Because sensor noise is indistinguishable from motion by image compression algorithms, low light levels and noisy camera sensors use more disk space.
• Manufacturer-specific settings: may enable additional compression features which reduce the size of video.

Insuring Sufficient Retention

If you need to meet specific camera retention requirements, use the following procedure:

1. Use the design tools provided by your camera and/or VMS manufacturer in order to estimate retention requirements, and match your camera settings to the settings provided in the design tool.  If both your camera manufacturer and VMS manufacturer provide a design tool, generate calculations from both and use the more conservative value (i.e., the higher required storage).
   
   See IPConfigure's Storage Design Tool at https://calculator.ipconfigure.com.
   
   
2. Use fixed bit rates in order to provide a reliable upper bound on the size of data coming from your cameras.  Aggressive bit rates can result in highly compressed and degraded video, so consult a manufacturer's design tool for a reasonable Bit Rate setting for your resolution, frame rate, scene complexity, etc. 
   
   Remember that Orchid Core VMS assigns Bit Rate in kbps (kilobits per second).  Use the following formulas to convert from Mbps (used by some design tools) to kbps (used by Orchid), and from kbps to Gigabytes per day:
   
   Mbps / 1024 = kbps
   kbps * 0.0108 = Gigabytes per Day 
   
   
3. If you've followed the steps above and Orchid Core VMS is still not achieving the retention you expect, look for outliers using the bar graph on Orchid's Retention Policy page:
   
   The bar graph above provides a graphical representation of the amount of space on disk used by each camera in your system.  In this example, the cameras indicated by the large pink and brown sections are using considerably storage more than their compatriots, and are reducing overall system retention.  This condition may be caused by cameras configured with bit rates, frame rates, or resolutions higher than the other cameras, or perhaps by noisy cameras in low light conditions.
   
   
4. Finally, consider the cost of not having older video available relative to the cost of additional storage.  The price of a larger RAID volume may be comparably low compared to the cost of insufficient surveillance retention.  Increase the estimated size of your storage requirements by 20 to 30% for additional assurance.

Common Misconceptions

• Misconception #1: Setting a Desired Retention on ALL cameras will insure I record as much video as I have specified.
  
  As explained above, the Desired Retention setting controls only the order in which older video is deleted.  If insufficient space is available to meet all of the Desired Retention settings, video newer than the Desired Retention must still be deleted to make room for new video:
  
  The Desired Retention settings above have no effect, as video will be deleted in the same order as if the Automatic retention policy were used.
  
  The Desired Retention provides no guarantee when applied to all cameras.  Even though the current Estimated Retention currently indicates that all cameras will achieve 12.5 days of retention, this may change in the future if, for example, camera settings are changed. 
  
  Cameras configured with a dynamic Bit Rate may also experience unexpected variability.  An outdoor camera that uses 6 GB/day during the summer months might use 8 GB/day during the winter months, when there is less daylight and the camera records more noisy nighttime video.
  
  
• Misconception #2: Orchid will update camera settings to achieve my retention goals.
  
  Orchid will never modify a camera's configured settings except when instructed by an administrator through the Camera Configuration page.
  
  
• Misconception #3: Orchid can record more video than I have space on disk.
  
  Orchid cannot turn water into wine, nor can it turn gigabytes into terabytes.  In the example below, the Retention Policy has been configured in a way that Orchid believes is impossible to fulfill:
  
  Insufficient space exists on disk to retain 1,000 days of video for the camera Cort's Deck.  Due to the order in which Orchid deletes old video, as described above, not only will the camera Cort's Deck not achieve 1,000 days of retention, video from cameras configured for Automatic retention will be deleted as soon as it is recorded.
  
  
• Misconception #4: Orchid under-reports my available disk space.
  
  Computers address their internal memory using binary numbers, and as a result the prefixes we use for computer data, like mega-, giga-, and tera-, have historically referred to binary rather than decimal prefixes.  For example, the binary prefix kilo- is equal to 2¹⁰ = 1024, while the decimal kilo- is equal to 10³ = 1000.  For smaller prefixes this difference tends to be unimportant, but for larger prefixes it can be significant: one binary terabyte = 2⁴⁰, while one decimal terabyte = 10¹²: a difference of nearly 10%.
  
  Today, most operating systems (such as Windows and Linux) and most software applications (such as Orchid Core VMS) use binary prefixes.  Hard drive manufacturers, however, use decimal prefixes.  So while a hard drive manufacturer sells a 10 (decimal) TB disk, both Windows and Orchid will report only 9 (binary) TB of space available.  They're both talking about the same number of bits, they're just representing them differently—similar to referring to the same distance as either 100 meters or (nearly) 110 yards.
  
  Unless explicitly specified, you should assume that the storage design tools provided by your camera and VMS manufacturer provide their estimates using binary prefixes.  If you need to purchase a hard drive or RAID system to accommodate a recommendation in terabytes, increase that recommendation by 10% to account for the difference between binary terabytes and decimal terabytes.