Is it feasible to create a syslog server which writes to a client dataset?

Question

I'm creating a simple syslog server in Delphi XE2 using Indy's TIdSyslogServer. I've decided to make it dump into a TClientDataSet and display in a TDBGrid, but I'm skeptical about how well it would handle if the log got quite big, and what I should expect when it grows to millions of records. This application is for internal use and I don't intend to make any software from it, and just keep the code real simple.

The purpose of the application is for numerous IP surveillance cameras along with various other network based equipment to report their log to one place.

This is a simple application with just 1 form, all the code is directly in the form's unit. The actual application is a separate project (call this my SSCCE).

uMain.pas

unit uMain;

interface

uses
  Winapi.Windows, Winapi.Messages, System.SysUtils, System.Variants, System.Classes, Vcl.Graphics,
  Vcl.Controls, Vcl.Forms, Vcl.Dialogs,
  IdSocketHandle, IdBaseComponent, IdComponent, IdSysLogServer, IdSysLog,
  IdSysLogMessage, IdUDPBase, IdUDPServer,
  Vcl.StdCtrls, Vcl.Grids, Vcl.DBGrids,
  Data.DB, Datasnap.DBClient, MidasLib {to avoid requiring MIDAS.DLL};

type
  TForm1 = class(TForm)
    Server: TIdSyslogServer;
    DS: TClientDataSet;
    DSC: TDataSource;
    DBGrid1: TDBGrid;
    procedure FormCreate(Sender: TObject);
    procedure FormDestroy(Sender: TObject);
    procedure ServerSyslog(Sender: TObject;
      ASysLogMessage: TIdSysLogMessage; ABinding: TIdSocketHandle);
  private
    procedure PrepareDS;
  public

  end;

var
  Form1: TForm1;

implementation

{$R *.dfm}

procedure TForm1.FormCreate(Sender: TObject);
var
  H: TIdSocketHandle;
begin
  PrepareDS;
  Server.Bindings.Clear;
  H:= Server.Bindings.Add;
  H.IP:= '0.0.0.0';     //All IP's
  H.Port:= 514;         //syslog standard port 514
  Server.Active:= True; //Activate server
end;

procedure TForm1.PrepareDS;
begin
  DS.DisableControls;
  try
    DS.Close;
    DS.FieldDefs.Clear;
    DS.FieldDefs.Add('timestamp', ftDateTime);
    DS.FieldDefs.Add('pri',       ftInteger);
    //Need to convert the next 2 to string
    DS.FieldDefs.Add('facility',  ftString,   15);
    DS.FieldDefs.Add('severity',  ftString,   15);
    DS.FieldDefs.Add('hostname',  ftString,   15);
    DS.FieldDefs.Add('message',   ftString,   200);
    DS.CreateDataSet;
    DS.Open;
  finally
    DS.EnableControls;
  end;
end;

procedure TForm1.FormDestroy(Sender: TObject);
begin
  Server.Active:= False;
end;

procedure TForm1.ServerSyslog(Sender: TObject;
  ASysLogMessage: TIdSysLogMessage; ABinding: TIdSocketHandle);
begin
  DS.Append;
    DS['timestamp']:= ASysLogMessage.TimeStamp;
    DS['pri']:=       ASysLogMessage.Pri;
    DS['facility']:=  ASysLogMessage.Facility;
    DS['severity']:=  ASysLogMessage.Severity;
    DS['hostname']:=  ASysLogMessage.Hostname;
    DS['message']:=   ASysLogMessage.Msg.Content;
  DS.Post;
end;

end.

uMain.dfm

object Form1: TForm1
  Left = 354
  Top = 124
  Caption = 'Form1'
  ClientHeight = 400
  ClientWidth = 597
  Color = clBtnFace
  Font.Charset = DEFAULT_CHARSET
  Font.Color = clWindowText
  Font.Height = -11
  Font.Name = 'Tahoma'
  Font.Style = []
  OldCreateOrder = False
  OnCreate = FormCreate
  OnDestroy = FormDestroy
  PixelsPerInch = 96
  TextHeight = 13
  object DBGrid1: TDBGrid
    Left = 0
    Top = 48
    Width = 597
    Height = 352
    Align = alBottom
    Anchors = [akLeft, akTop, akRight, akBottom]
    DataSource = DSC
    TabOrder = 0
    TitleFont.Charset = DEFAULT_CHARSET
    TitleFont.Color = clWindowText
    TitleFont.Height = -11
    TitleFont.Name = 'Tahoma'
    TitleFont.Style = []
  end
  object Server: TIdSyslogServer
    Bindings = <>
    OnSyslog = ServerSyslog
    Left = 120
    Top = 168
  end
  object DS: TClientDataSet
    Aggregates = <>
    Params = <>
    Left = 168
    Top = 168
  end
  object DSC: TDataSource
    DataSet = DS
    Left = 200
    Top = 168
  end
end

I'm assuming that at some point I should at least make it dump to a file, then start fresh. That's an obvious feature I will need to add. Along with that of course a way to recall the saved logs. That all comes later, but I'm only worried how the client dataset will handle it when it gets very large, and really how I should determine the maximum before I dump it.

Answer 1

TClientDataset will work fine for this purpose, except for the "million of records". I don´t believe that any dataset will have a good response for such an amount of data, so I think you will have to work some kind of paging system and keep in the dataset only some hundreds or thoudsands of records, after all, it´s unhuman to handle millions of anything (;-)).

If you really need to keep millions of records, so it´s my opinion that you really need some kind of database system to stored it, not an in-memory structure.

I took a look on your code and noticed that you call Open just after calling CreateDataset. That´s unnecessary given CreateDataset already gets the dataset open.

By the way, you had and excelent idea on including MidasLib in you uses clause.