We have two syslog servers and use a F5 to load balance between the two. In total they receive around 45 million messages a day.We have around a dozen rules that forward messages onto a security appliance or splunk and it can take around 30 minutes before those messages arrive. It can also take 30 minutes for any emails to end up in a users mailbox.
As soon as we start the syslog service the message count on the buffer starts to climb and eventually the overflow queue increase. We haven't checked the stats for a while but one of the servers had a overflow queue count of 125,000! It is a VM server running Windows 2003, 2 CPU's and 4Gb RAM.
Here are the stats from the first hour of starting the syslog service
Kiwi Syslog Server [Licensed] Version 9.4.1
/// Kiwi Syslog Server Statistics ///
---------------------------------------------------
24 hour period ending on: Thu, 22 May 2014 09:03:09
Syslog Server started on: Thu, 22 May 2014 08:04:17
Syslog Server uptime: 0 hours, 58 minutes
---------------------------------------------------
+ Messages received - Total: 767628
+ Messages received - Last 24 hours: 767628
+ Messages received - Since Midnight: 767628
+ Messages received - Last hour: 0
+ Message queue overflow - Last hour: 0
+ Messages received - This hour: 767628
+ Message queue overflow - This hour: 0
+ Messages per hour - Average: 767628
+ Messages forwarded: 775368
+ Messages logged to disk: 767587
+ Errors - Logging to disk: 0
+ Errors - Invalid priority tag: 0
+ Errors - No priority tag: 602
+ Errors - Oversize message: 464
+ Disk space remaining on drive C: 3904 MB
Breakdown of Syslog messages by severity
+--------------------+------------+------------+
| Message Level | Messages | Percentage |
+--------------------+------------+------------+
| 0 - Emerg | 17 | 0.00% |
| 1 - Alert | 10 | 0.00% |
| 2 - Critical | 504 | 0.07% |
| 3 - Error | 26356 | 3.43% |
| 4 - Warning | 619384 | 80.69% |
| 5 - Notice | 61780 | 8.05% |
| 6 - Info | 58963 | 7.68% |
| 7 - Debug | 614 | 0.08% |
+--------------------+------------+------------+
Message Buffer Information
==========================
Message Queue Max Size: 500000
Message Queue overflow: 18858
Message Count: 500000
Message Count Max: 500000
Percentage free: 0
Any help would be appreciated
Thanks
John