Release 0.4 is the new stable

The previously released version 0.4 was found to be working w/o known bugs with the latest stable FPC 2.6.0 (svn revison 19485). Now virtually rereleasing the RC1 as the "offical" stable.

heLib release 0.4

This is a RC1 of heLib for FPC 2.6.0 (soon to be released Free Pascal version). Compatible/tested with tags/release_2_6_0, rev 19814. Download source tarball here.

Expect a release version (say, version 0.5) soon after FPC 2.6.0 gets out if this RC1 is not going to work with the final FPC 2.6.0 and/or issues will be found and fixed in heLib 0.4.

Online documentation.

Status:

$ ./helib_test -a -p --format=plain
...
Number of run tests: 701
Number of errors:    0
Number of failures:  0

Heap dump by heaptrc unit
5844426 memory blocks allocated : 252487390/286727568
5844426 memory blocks freed     : 252487390/286727568
0 unfreed memory blocks : 0
True heap size : 294912
True free heap : 294912
$

Anybody out there?

The helib/heContnrs latest release (0.3.1) is once again dead for the time being due to a compiler internal error with the latest (as of today) trunk FPC.

This is no rant about the trunk version changing. That's actually a good thing - the dev team is continuously improving it. The problem is to play catch up with the trunk. It was my decision to base heContnrs on the trunk version. It was the only option in that time, the stable FPC version back in September last year (2.4.0) was not able to cope with some language features required for heContnrs.

I don't know if some users of heContnrs do exist at all. I've never received a bug report through mail, nor an issue was filled on the project site (it can be done by anyone - anonymously). It seems to me it's time to re-investigate the current stable FPC version (2.4.2) and figure out if heContnrs could be made compatible with it. Never tried that actually, maybe a miracle will happen and it will just work, although I don't expect that seriously.

As a hobby project as heContnrs is/are, created mostly to fill my own needs, this can take some time (an unbound amount). If someone has some project with a now broken dependency on any of the previous working versions of the lib then perhaps let me know (via a comment here of through email) to create some pressure on me. I can't promise anything, but I don't do too many things in FPC anymore (only because my professional life is now going on in a different programming language). But I feel guilty for publishing the work which is not usable now and if that created problems to others, I will try to be responsible for that, i.e. try to fix this sooner then if I would do that when it's needed by myself only.

Generic B+Trees (heLib release 0.3)

FPRB/heLib/heContnrs.pas has got a new container type - generic BTrees.

heLib release 0.2

FPRB/heLib/heContnrs.pas has got a new container type - generic lists.

heLib release 0.1

FPRB/heLib 0.1 has been released today:

• Generic vector containers with generic enumerators.
  
• Online documentation.
  
• Test suite.
  
• Permissive BSD-style license.
  
• Requires trunk FPC (2.5.1).
  

Resources:

• Source tarball downloadable directly from the project home page.
  
• Subversion access
  

Ideas, code reviews, testing, bug reports, code contributions etc. are welcome.

Containers with multiple enumerators

Paul Ishenin is, AFAIK, the author of the FPC enumerators support implementation. In his article on the FPC Wiki "FOR-IN LOOP", section "Proposed extensions/Select which enumerator to use", one can read (I hope it's allowed/legal/fair use to reproduce here the part of interest):

It is impossible to choose among different possible enumerators. For example you can traverse a tree using different orders. The well known algorithms are: preorder, postorder, inorder and breadth‑first traversals. Therefore it would be useful to have an ability to choose an enumerator. For example using the following syntax:

 
type
  TTreeEnumeratorType = (tePreOrder, tePostOrder, teInOrder, teBreadthFirst)

procedure TraverseTree(Tree: TTree);
var
  Node: TNode;
begin
  // Variant1. For the class instances we can call the method Tree.GetEnumerator(teInOrder). 
  // For the classes we can call a class method
  for Node in Tree using GetEnumerator(teInOrder) do
    Dosomething(Node);
 
  // Variant2. Or we can call the global function
  for Node in Tree using GetEnumerator(Tree, teInOrder) do
    Dosomething(Node);
 
  // Variant3. In the previous variant 'in Tree' is useless so the next code is a simplified form:
  for Node using GetEnumerator(Tree, teInOrder) do
    Dosomething(Node);
 
  // Variant4. We can try to avoid new context key-word 'using' by calling method:
  for Node in Tree.GetSomeEnumerator(teInOrder) do
    Dosomething(Node);
  // but this brings ambiguity to the compiler since Tree.GetSomeEnumerator(teInOrder) can be translated into
  // Tree.GetSomeEnumerator(teInOrder).GetEnumerator
  // This ambiguity might be resolvable by checking whether the class implements IEnumerator interface
end;

It is true that any type can have only one GetEnumerator method attached to it. But that's IMO not the same as "It is impossible to choose among different possible enumerators."

I guess, i.e. this was not tested and not even tried - it is already possible to have different enumerators on a type. The desired behaviour could be achieved presumably in two ways.

Using IEnumerator?

Iff the current FPC implementation checks for existence of a IEnumerator derived implemented interface of a type (i.e. as opposed to checking for existence of exactly only IEnumerator interface), then it should be possible to have multiple distinct IEnumerator derived interfaces implemented by a type (each one returning a different enumerator object) and then "select" the enumerator like this:

 
  // maybe parenthesis would be required bellow, i.e. (Tree as ITreeInOrder)
  for Node in Tree as ITreeInOrder do
    Dosomething(Node);
 
  for Node in Tree as ITreeBreadthFirst do 
    Dosomething(Node);
 

Using the enumerator operator?

Another possibility is to declare multiple enumerator operators with differently typed operands and then cast the real type to select the specific enumerator:

 
type
  TInOrderTree = class(TTree);
  TBreadthFirstTree = class(TTree);
 
operator Enumerator(Op: TInOrderTree): TTreeEnumerator;
begin
  // implementation
end;
 
operator Enumerator(Op: TBreadthFirstTree): TTreeEnumerator;
begin
  // different implementation
end;
 
//...
  for Node in TInOrderTree(Tree) do
    Dosomething(Node);
 
  for Node in TBreadthFirstTree(Tree) do 
    Dosomething(Node);
 

The choice

The interface based variant seems a bit heavy to me. The casting is maybe somewhat lighter (per opinion), but at the same time a bit dangerous as type casts in FPC are - and have to be for a reason - usually a way how to tell the compiler not to take care of any type checking no more (an oversimplification, I admit, but ~applies to the current task). Both variants do also require declaration of new types, but that's a minor issue.

Let's get back to Paul's original proposal and specifically the Variant4. The observation "but this brings ambiguity to the compiler since Tree.GetSomeEnumerator(teInOrder) can be translated into Tree.GetSomeEnumerator(teInOrder).GetEnumerator" is correct. IMO the compiler must interpret the code exactly in this and only this way, but that's another story. Variant4 is also interesting here as it was chosen to play with in an attempt to get a usable case/implementation/usage pattern.

It's a matter of only personal preferences, but I like distinctly naming the specific "GetEnumerators" more than parameterizing a single one, so the container's client code will be something like:

 
  for Node in Tree.InOrder do
    Dosomething(Node);
 
  for Node in Tree.BreadthFirst do
    Dosomething(Node);
 

It's a few characters shorter and IMO pretty readable also. Conceptually, one should choose which enumerating behaviour should be the default one, supposedly based on the estimated typical usage pattern, and implement that enumerator as the (single) .GetEnumerator method of such type. All the other enumerators, if properly named, would make the intention clear. For example a TSomeMap could have a "default" enumerator returning the key-value pairs and the the specific/special .Key and .Value enumerators:

 
type
  TKey = String;
  TValue = Integer;
  TPair = record
    Key: TKey;
    Value: TValue;
  end;
 
  TMyMap = class(specialize TSomeMap<TKey, TValue>)
  // ...
  end;
 
// ...
var 
  M: TMyMap;
  P: TPair;
  K: TKey;
  V: TValue;
// ...
  for P in M do
    ProcessPair(P);
  for K in M.Key do
    ProcessKey(K);
  for V in M.Value do
    ProcessValue(V);
 

The implementation

Bellow are excerpts from heContnrs.pas implementing both the default (0 .. Count - 1) and the .Reversed (Count - 1 .. 0) enumerators of a vector.

 
//...
interface
 
type
 
  { TheEnumerator }
 
  generic TheEnumerator<TIterator, TValue> = object
  public type
    TGetCurrent = function(const Iterator: TIterator): TValue of object;
    TMoveNext = function(var Iterator: TIterator): Boolean of object;
  private
    FGetCurrent: TGetCurrent;
    FIterator: TIterator;
    FMoveNext: TMoveNext;
    function GetCurrent: TValue;
  public
    procedure Init(const InitialIterator: TIterator; const Mover: TMoveNext; const Getter: TGetCurrent);
    function MoveNext: Boolean;
    property Current: TValue read GetCurrent;
  end;
 
  { TheEnumeratorProvider }
 
  generic TheEnumeratorProvider<TProvidedEnumerator> = object
  public
    FEnumerator: TProvidedEnumerator;
    function GetEnumerator: TProvidedEnumerator;
  end;
 
  { TheVector }
 
  generic TheVector<TItem> = class
  public type
    //...
    TEnumerator = specialize TheEnumerator<Integer, TItem>;
    TReverseEnumeratorProvider = specialize TheEnumeratorProvider<TEnumerator>;
  private type
    PItem = ^TItem;
  private
    //...
    function GetItems(const Index: Integer): TItem;
    function MoveNext(var Index: Integer): Boolean;
    function MovePrev(var Index: Integer): Boolean;
  public
    //...
    function GetEnumerator: TEnumerator;
    function Reversed: TReverseEnumeratorProvider;
  end;
 
implementation
 
uses
  SysUtils;
 
{ TheEnumerator }
 
function TheEnumerator.GetCurrent: TValue;
begin
  Result := FGetCurrent(FIterator);
end;
 
function TheEnumerator.MoveNext: Boolean;
begin
  Result := FMoveNext(FIterator);
end;
 
procedure TheEnumerator.Init(const InitialIterator: TIterator; const Mover: TMoveNext; const Getter: TGetCurrent);
begin
  Assert(Assigned(Mover));
  Assert(Assigned(Getter));
  FIterator := InitialIterator;
  FMoveNext := Mover;
  FGetCurrent := Getter;
end;
 
{ TheEnumeratorProvider }
 
function TheEnumeratorProvider.GetEnumerator: TProvidedEnumerator;
begin
  Result := FEnumerator;
end;
 
{ TheVector }
 
//...
function TheVector.GetItems(const Index: Integer): TItem;
begin
  Assert((Index >= 0) and (Index < Count));
  Result := FData[Index];
end;
 
function TheVector.MoveNext(var Index: Integer): Boolean;
begin
  Inc(Index);
  Result := Index < Count;
end;
 
function TheVector.MovePrev(var Index: Integer): Boolean;
begin
  Dec(Index);
  Result := Index >= 0;
end;
 
function TheVector.GetEnumerator: TEnumerator;
begin
  Result.Init(-1, @MoveNext, @GetItems);
end;
 
function TheVector.Reversed: TReverseEnumeratorProvider;
begin
  Result.FEnumerator.Init(Count, @MovePrev, @GetItems);
end;
 
end.
 

The example

This code ...

 
program project1;
 
{$mode objfpc}{$H+}
 
uses heContnrs;
 
type
  TStrVector = specialize TheVector<String>;
 
var
  V: TStrVector;
  S: String;
 
begin
  {$if declared(HaltOnNotReleased)}
  HaltOnNotReleased := True;
  {$endif}
  V := TStrVector.Create;
  try
    V.Add('foo');
    V.Add('bar');
    V.Add('baz');
    Writeln('for S in V ...');
    for S in V do
      Writeln('S = ''', S, '''');
    Writeln('for S in V.Reversed ...');
    for S in V.Reversed do
      Writeln('S = ''', S, '''');
  finally
    V.Free;
  end;
end.
 

... compiled and executed, produces this output:

$ ./project1
for S in V ...
S = 'foo'
S = 'bar'
S = 'baz'
for S in V.Reversed ...
S = 'baz'
S = 'bar'
S = 'foo'
Heap dump by heaptrc unit
9 memory blocks allocated : 892/912
9 memory blocks freed     : 892/912
0 unfreed memory blocks : 0
True heap size : 360448
True free heap : 360448
$

The credits

The authorship of the original ideas on multiple enumerators, leading to this post and the above implementation, should be fully credited to and only to Paul Ishenin. My best thanks to him for sharing his work.

Sampler: Generic enumerators for generic containers

unit Unit1; 

{ (c) 2010 bflm, contact: befelemepeseveze at Google's free mail

  Free sample code - copy, use, modify, distribute, sell, release under other license, ...

}

{$mode objfpc}{$H+}

interface

type

  { TEnumerator }

  generic TEnumerator<TValue, TIterator> = object
  public type
    TGetCurrent = function(const Iterator: TIterator): TValue of object;
    TMoveNext = function(var Iterator: TIterator): Boolean of object;
  private var
    FGetCurrent: TGetCurrent;
    FMoveNext: TMoveNext;
    FIterator: TIterator;
    function GetCurrent: TValue;
  public
    procedure Init(const InitialIterator: TIterator; const Mover: TMoveNext; const Getter: TGetCurrent);
    function MoveNext: Boolean;
    property Current: TValue read GetCurrent;
  end;

  { TVector }

  generic TVector<T> = object
  public type
    TData = array of T;
    TCompare = function(const A, B: T): Integer;
    TVectorEnumerator = specialize TEnumerator<T, Integer>;
  private var
    FData: TData;
    FLen: Integer;
  private
    function GetCap: Integer;
    function GetData: TData;
    function GetItems(const Index: Integer): T; inline;
    function GetLen: Integer; inline;
    function MoveNext(var Index: Integer): Boolean;
    procedure Sort(Left, Right: Integer; const Compare: TCompare);
    procedure SetCap(const AValue: Integer);
    procedure SetData(const AValue: TData);
    procedure SetItems(const Index: Integer; const AValue: T); inline;
    procedure SetLen(const AValue: Integer);
  public
    function GetEnumerator: TVectorEnumerator;
    function Add(const Value: T): Integer;
    procedure Clear;
    procedure Pack;
    procedure Sort(const Compare: TCompare);
    property Cap: Integer read GetCap write SetCap;
    property Data: TData read GetData write SetData;
    property Items[const Index: Integer]: T read GetItems write SetItems; default;
    property Len: Integer read GetLen write SetLen;
  end;

implementation

{ TEnumerator }

function TEnumerator.GetCurrent: TValue;
begin
  Result := FGetCurrent(FIterator);
end;

function TEnumerator.MoveNext: Boolean;
begin
  Result := FMoveNext(FIterator);
end;

procedure TEnumerator.Init(const InitialIterator: TIterator; const Mover: TMoveNext; const Getter: TGetCurrent);
begin
  Assert(Assigned(Mover));
  Assert(Assigned(Getter));
  FIterator := InitialIterator;
  FMoveNext := Mover;
  FGetCurrent := Getter;
end;

{ TVector }

function TVector.Add(const Value: T): Integer;
var NeedCap: Integer;
begin
  NeedCap := Len + 1;
  if NeedCap > Cap then
    Cap := 2 * NeedCap;
  Result := FLen;
  Inc(FLen);
  Items[Result] := Value;
end;

procedure TVector.Clear;
begin
  Cap := 0;
end;

function TVector.GetItems(const Index: Integer): T;
begin
  Result := FData[Index];
end;

function TVector.GetCap: Integer;
begin
  Result := Length(FData);
end;

function TVector.GetData: TData;
begin
  Result := Copy(FData, 0, Len);
end;

function TVector.GetEnumerator: TVectorEnumerator;
begin
  Result.Init(-1, @MoveNext, @GetItems);
end;

function TVector.GetLen: Integer;
begin
  if FData = nil then
    FLen := 0; // boot
  Result := FLen;
end;

function TVector.MoveNext(var Index: Integer): Boolean;
begin
  Result := Index + 1 < Len;
  if Result then
    Inc(Index);
end;

procedure TVector.SetCap(const AValue: Integer);
begin
  if AValue = Cap then
    Exit;
  if Len > AValue then
    FLen := AValue;
  SetLength(FData, AValue);
end;

procedure TVector.SetData(const AValue: TData);
begin
  FData := AValue;
  FLen := Cap;
end;

procedure TVector.SetItems(const Index: Integer; const AValue: T);
begin
  FData[Index] := AValue;
end;

procedure TVector.SetLen(const AValue: Integer);
begin
  if AValue = Len then
    Exit;
  if AValue > Cap then
    Cap := AValue
  else if AValue <= Cap div 2 then
    Cap := Cap div 2;
  FLen := AValue;
end;

procedure TVector.Sort(Left, Right: Integer; const Compare: TCompare);
var
  L, R: Integer;
  Pivot, Swap: T;
begin
 repeat
   L := Left;
   R := Right;
   Pivot := Items[(Left + Right) div 2];
   repeat
     while Compare(Pivot, Items[L]) > 0 do
       L += 1;
     while Compare(Pivot, Items[R]) < 0 do
       R -= 1;
     if L <= R then begin
       Swap := Items[L];
       Items[L] := Items[R];
       Items[R] := Swap;
       L += 1;
       R -= 1;
     end;
   until L > R;
   if Left < R then
     Sort(Left, R, Compare);
   Left := L;
 until L >= Right;
end;

procedure TVector.Pack;
begin
  Cap := Len;
end;

procedure TVector.Sort(const Compare: TCompare);
begin
  if Len > 1 then
    Sort(0, FLen - 1, Compare);
end;

end.

program project1;

{$mode objfpc}{$H+}

uses Unit1;

type
  TIntVector = specialize TVector<Integer>;

function IntCmp(const A, B: Integer): Integer;
begin
  Result := A - B;
end;

procedure Dump(const Vector: TIntVector);
var V: Integer;
begin
  Write(Vector.Len, ' item(s): ');
  for V in Vector do
    Write(V, ' ');
  Writeln;
end;

procedure Test(const N, X: Integer);
var
  Vector: TIntVector;
  I: Integer;
begin
  for I := 1 to N do
    Vector.Add(I xor X);
  Dump(Vector);
  Vector.Sort(@IntCmp);
  Dump(Vector);
  WriteLn;
end;

begin // main
  {$if declared(HaltOnNotReleased)}
  HaltOnNotReleased := True;
  {$endif}
  Test(0, 0);
  Test(1, 0);
  Test(11, 0);
  Test(13, $55);
  Test(17, $AA);
  Test(19, $FF);
end.

$ fpc -B -gh project1 && ./project1 
Free Pascal Compiler version 2.5.1 [2010/05/05] for x86_64
Copyright (c) 1993-2009 by Florian Klaempfl
Target OS: Linux for x86-64
Compiling project1.pas
Compiling unit1.pas
Linking project1
/usr/bin/ld: warning: link.res contains output sections; did you forget -T?
259 lines compiled, 0.1 sec 
0 item(s): 
0 item(s): 

1 item(s): 1 
1 item(s): 1 

11 item(s): 1 2 3 4 5 6 7 8 9 10 11 
11 item(s): 1 2 3 4 5 6 7 8 9 10 11 

13 item(s): 84 87 86 81 80 83 82 93 92 95 94 89 88 
13 item(s): 80 81 82 83 84 86 87 88 89 92 93 94 95 

17 item(s): 171 168 169 174 175 172 173 162 163 160 161 166 167 164 165 186 187 
17 item(s): 160 161 162 163 164 165 166 167 168 169 171 172 173 174 175 186 187 

19 item(s): 254 253 252 251 250 249 248 247 246 245 244 243 242 241 240 239 238 237 236 
19 item(s): 236 237 238 239 240 241 242 243 244 245 246 247 248 249 250 251 252 253 254 

Heap dump by heaptrc unit
11 memory blocks allocated : 840/840
11 memory blocks freed     : 840/840
0 unfreed memory blocks : 0
True heap size : 98304
True free heap : 98304
$