                      RANDOM NOTES ON CSL/CCL
                      =======================


(I bet that much of this will get out of data soon, but this version has
 been reviewed in February 2010 and then in August 2017)


The code tries to be in fairly conservative C++, but in quite a lot
of places it depends on "the spirit of C" rather than the letter of
the language standards. I am trying to make it as respectable as I feel
that I can, but the nature of a dynamic language like Lisp leads me
to needing to do many casts.

The code here is supposed to be buildable using any 32- or 64-bit C
compiler on almost any reasonable computer. Normal building requires
that you can install (or if necessary build from source) the GNU autoconf
tools and a "sh" shell. By far the most testing has been done using
gcc. The platforms most tested now are 64-bit Windows (building using
the cygwin shell and tools and x86_64-w64-mingw32-*, and also a version
using 64-bit cygwin's direct gcc), 64-bit Linux (mostly Ubuntu) running
on x86_64, and MacOSX (Sierra). I also build on a Raspberry pi and perhaps
less frequently using a variety of other systems running using virtualbox.

The GUI code at present relies on the FOX GUI toolkit, and that can be built
for X11 and for Windows. It runs on a Mac only by using the X11 sub-system.
There is the start of a conversion to use wxWidgets - if ever completed that
will provide native Macintosh support.

I have adopted an indentation and code layout style that I like. I am aware
that it is not the default one supported my emacs etc, but I have been using
it for a long while and feel very set in my ways. You mnay find it described
as the Allman, ANSI or "emacs/bsd" style, and it was (close to) the default
used in some versions of Visual Studio. Sometimes I reduce wasted page
space by using the style that Wikipedia calls "Horstman". My personal views
really date back to the late 1960s and the layout style for BCPL code that I
became accustomed to then, and hence predate C and hence all the authors
whose names have become associated with C layout styles! So really I probably
want to call my layout the "Martin Richards" style. I can use astyle with
my own set of configuration options to regularize things.

This Lisp uses an agressive garbage collector that MUST be able to identify
all root pointers reliably. C++ local variables are not marked by the GC, and
because the GC can relocate things a reference held in a simple C++ variable
is NOT SAFE across anything that could cause a GC. The macros push() and
pop() place data on a Lisp-safe stack. The coding conventions needed to make
GC safe are ones that take a significant time to get used to. They are not
coder-friendly. If you breach them then the code may work most of the time
but fail later in a messy way. I can only say "you have been warned"!

At some stage I MAY implement a conservtive mostly-copying garbage collector
but that will be a fairly disruptive re-write for all storage management. If
I ever do manage that then I can remove very many uses of push() and pop()
and things may be smaller, faster and more reliable.

This Lisp needs to control error recovery and general unwinding.
This is generally achieved using C++ exceptions, and the file lispthrow.h
contains various fragments of stylized code using them to try to keep the
world straight.

Common Lisp supports functions that return multiple values. Thus the normal
simple case MUST use the onevalue() macro to indicate that what is returned
is a singleton. Failure to do this can mess up error recovery or the
garbage collector.

The above issues lead to significantly ugly and delicate code. They
are easy for somebody who is not neurotic enough to overlook. The
trouble that they can lead to will often be deferred, only showing up
occasionally on garbage collection or when recovering from an unrelated
exception. This situation where errors can go undetected for a long while
makes the need for care over these issues especially high priority!

For several platforms it seems to be a really good thing to be able to
respond well to asynchronous events. These may be "^C", or mouse clicks.
Especially with windowed systems some form of threading would be very very
helpful. This seems quite hard wrt portability. Thus for essentially all
systems I use the idea of a "tick stream" of more or less regular events that
can be used to activate polling etc. Having tried genuine timer-based systems
and having had a LOT of trouble with them I mostly use a software count-down
scheme under the guard SOFTWARE_TICKS. If you feel tempted to discard this
in favour of real threads or of nothing then consider the issues of back-
porting to (eg) the Mac, or DOS where you must poll before even ^C is
noticed. I do not want this code to lose portability, please! But at present
response to signals is not well controlled, and recovery after stack
overflow is pretty pathetic.

The basic data representation here is that small integers (and characters)
are represented as value types, with some of the low 3 bits used as tags.
Cons cells, vectors, big-numbers etc are handled by reference. By arranging
that all items are stored aligned to an 8-byte boundary the low 3 bits
in a pointer are available for use as tags. More details can be seen in
"tags.h".

The Lisp system here was designed for delivery of code that had already
been developed and was basically working. It was not designed as a
development environment. This means that in some respects it is
inflexible! For instance there is a Lisp to C++ compiler, but it is
configured so that the generated C++ is statically linked into the Lisp.
The Lisp does not have very clever windowed interfaces, browsers etc., but
the GUI supported via a separate GUI toolbox may still be better than
nothing. Actually in many ways I find the extent to which this Lisp can
check for bad activity and produce aq half-plausible backtrace means that
it is not that bad as a development platform.

The Lisp compiler has its master version coded in RLISP (an Algol-like
surface syntax for Lisp, popularised by the REDUCE algebra system). A
yacc-based parser can translate this into raw Lisp, but enhancements would
be better made to the RLISP version.


[more random comments follow...]

PRESERVE optional args.  First specifies a restart function
to be called, the second is a banner to display when the image is reloaded.
Reduce now uses this mechanism to gets its banner displayed. Third arg can be
stuff to pass to the restart function - passing is via a text-string interface
so it is a bit delicate!

Lexical closures supported in the bytecode compiler (for variable reference
but not GO or RETURN-FROM) and 3 new opcodes (closure, loadlex, storelex)
introduced in bytes1.c to support same.

Line-length overflow checking for printing of big numbers implemented.

binary printing for numbers, and case fold up as well as down, supported.

modules ("fasl files") within heap images are portable across byte ordering
and machine word-length. Ditto heap images.

-------------------------------------------------------------------------------

DATA REPRESENTATION
===================

All objects in CSL are represented by 32- or 64-bit words, and each object in
memory is aligned at an 8-byte boundary. More pedantically the type LispObject
is a synonym for intptr_t. The bottom 3 bits of a word are used as tags, and
indicate how the rest of the word should be interpreted.  In some cases the
rest of the word is just direct information (eg small integers are handled
this way).  In other cases the word contains the address in memory of some
object. Many objects in memory have a header word at their start that
indicates their length and gives more details about their type. The encoding
of object lengths means that the largest possible single object in CSL would
be 4 Mbytes long on a 32-bit system, but there is no serious limit if
you are using a 64-bit machine. Also CSL memory is allocated in chunks
(typically 8 Mbytes) and that puts another limit on maximum object size,
but vectors can have a multi-level represetation that works around the
limit. This may matter for arrays and bignums.

The coding in the low bits of a word is as follows:

    x x x | x 0 0 0      cons cell (and NIL in Common Lisp mode)
    x x x | x 0 0 1      pointer to a Lisp vector of some sort (inc strings)
    x x x | x 0 1 0      characters, vec-hdrs, other oddities (immediate)
    x x x | x 0 1 1      a forwarding address used during garbage collection
    x x x | x 1 0 0      a symbol
    x x x | x 1 0 1      bignum, rational number, complex number
    x x x | x 1 1 0      pointer to boxed floating point number
    x x x | f 1 1 1      immediate integer or float (f discriminates)


Header words in the vector heap

See tags.h for the detailed coding of vector headers, because there are
rather a lot of cases. The basic format is
          ... | x x x x | x x y y | y y y z | z 0 1 0 |
where zz is
          00   header of a symbol, or a character object or a "SPID"
          01   Lisp vector containing pointers
          10   Lisp bit-vector
          11   other Lisp vector containing binary data
then yyyyy gives more information about the exact type of the vector
concerned, and ....xxxxxx is the length of the vector (including its header)
counting in 32-bit words. bit-vectors, byte vectors (inclkuding strings) and
hanfword-vectors use some of yyyyy to give the length information the extra
resolution that they require.
Every symbol object is the same size, so I do not store an explicit length
code in its header - the xxxxxxyyyyy bits are used for many symbol-specific
attributes.


-------------------------------------------------------------------------------

                  FAST PROPERTY ACCESS IN CSL/CCL
                  ===============================

The "fast-get" facility in CSL arranges that for a number of user-specified
property names (used with both GET and FLAGP) the access has constant cost.
Normally access to a property involves searching a property list, and if one
symbol has many properties this can be tedious - with a worst case in the
fairly common circumstance that the property is not found.

To speed up CSL I allocate 6 bits in a symbol header that influence how that
symbol is treated when used as a property name.  One code is reserved to
mean "use a normal property list", leaving 63 codes for fast access. The
first of these is reserved for a property called 'NONCOM because the built-in
ORDERP function wants to use this for compatibility with REDUCE(!!).

A tag must be marked as "fast" before any properties with that indicated are
set up.  If this constraint is not adhered to then properties can end up
stored in inconsistent or redundant forms.  The protocol is to select an
integer in the range 0 to 63 for each tag to be handled specially, and then
to make a call such as

    (symbol-make-fastget 'noncom           0)
    (symbol-make-fastget 'my_property_name 1)

    (flag '(a b c) 'noncom)
    (put 'x 'my_property_name 'y)
 or (setf (get 'x 'my_property_name) 'y)

The fast-get status of a symbol can be inspected using

    (symbol-fast-get 'x nil)

or reset to have no special treatment (please do not do this!)

    (symbol-fast-get 'x -1)

When a symbol is given a "fast" property a vector of length 63 is created
and a word in the symbol's header is made to point to it. The vector contains
either property values ir a marker that indicates that no property is
present.  A call (symbol-fastgets 'x) can retrieve the vector and can be
useful while debugging, maybe.

If you inspect the property list of an object (using the PLIST function) then
any fast properties are extracted from the vector and built onto the front
of the property list as returned. Of course this means that altering the
property list using RPLACx may not be useful, but I do not think anybody should
try that anyway!

The function GET retrieves a property. In Common Lisp mode the three-argument
version (GET a b c) allows c to be returned as a default value if the
property sought is not present. The two-argument version can not distinguish
between a property whose value is NIL and not having a property present
at all.  The function FLAGP returns T if a property is present (whatever
value is associated, including the case where the stored value is NIL). The
function FLAG just sets properties, giving them the value T in a somewhat
arbitrary way.

Clearly use of fast-gets consumes memory, but because many of the extra
vectors contain many NIL elements they compress well in image files, which
therefore do not grow too badly.  To decide which properties are most
critical you can run tests using the profiled version of the bytecode
interpreter (bytes.c rather than bytes1.c), and after a run call the
function (BYTECOUNTS).   You need to adjust your build sequence so that
the file bytes.c is compiled with RECORD_GET defined, e.g. by putting
                   -DRECORD_GET
among the flags passed to your C compiler.  Note that this option will slow
things down noticeably.  The output from BYTECOUNTS indicates which property
tags were used, and how many unsuccessful searches for each tag occurred. This
latter information is collected because unsuccessful searches are the worst
case for a traditional property list search.

Note that when a property is handled "fast" it is not stored on the regular
property list (the only information about it is in the vector). Thus if the
most commonly used properties have been handled this way the average length
of property lists will shrink and so access to all other properties is
slightly speeded up too.

By editing FASTGET_SIZE in the source file "tags.h" if would be possible to
make the symbol extension vectors smaller than 63 items long, so in
cases where a much smaller set of tags is important the system can
be configured to save some memory.

-----------------------------------------------------------------------

[This issue was responded to during the Axiom customisation process]
Subject: Interface for dynamic opening and closing of libraries

Arthur:

I've discussed the specification of the "dynamic libraries" interface with
Barry and this is what we think we need.  For a user's own code, we will
always know the full pathname of the library it is in, so we can load the
module explicitly.

1. A function that loads a particular module from a particular library, e.g.
    (load-module <module> <pathname>)
   The library need not be open for input.  You might wish to have separate
   open-library, load-module, and close-library operations if that is more
   efficient.  

2. A function that opens a library for output, e.g.
    (open-library <pathname>)
   The function need not be responsible for checking if the library is open 
   already.

3. A function that reads a lisp source file, translates it into byte codes, 
   and writes the results into a named library, e.g.
    (fasl-out <filename> <library>)
   This could either:
     (a) Overwrite an existing module with the same name (our preferred option),
         or
     (b) replace the entire library if it already exists.

4. A function that closes an open library and, if option 3a above is 
   implemented, does all the necessary tidying-up, compaction etc, e.g.
    (close-library <pathname>)

5. Facilities for setting the library search path from within Lisp:
    (a) deleting a library from the search path;
    (b) adding a new library to the front of the search path;
    (c) adding a new library to the end of the search path.


------------------------------------------------------------------------

This shows the proportionate use of the various byte opcodes
on one particular test run. The information was collected while I was
trying to optimise the bytecode design. If any elaborate bytecode or
combination of bytecodes was getting very heavy use I might provide
special-case variants etc. Of course with different test programs the
profile may end up somewhat different.

    7.3400  LOADLOC6  
    3.8160  LOADFREE  
    3.6256  CARLOC6   
    2.8646  LOADFREE2 
    2.7618  EXIT      
    2.6747  PUSH      
    2.6054  LOADLOC1  
    2.2990  LITGET    
    2.2175  LOADLOC3  
    2.0495  STOREFREE 
    2.0072  JUMPNIL_L 
    2.0046  CONS      
    1.8176  BUILTIN2  
    1.7415  JUMPNIL   
    1.7210  LOADLOC0  
    1.5494  JUMPST1NI 
    1.5250  BUILTIN1  
    1.4593  JUMPNIL_B 
    1.4564  CALL1     
    1.4206  STORELOC1 
    1.3783  PUSHNIL2  
    1.3783  LOSE2     
    1.3624  FREERSTR  
    1.3624  FREEBIND  
    1.2805  JUMPT     
    1.2565  JUMP_B    
    1.1196  CDR       
    1.0389  LOADLOC5  
    1.0384  VNIL      
    0.9265  STORELOC2 
    0.9265  STORELOC0 
    0.9200  JUMPNEQCA 
    0.8966  CAR       
    0.8812  LOADLOC10 
    0.8712  JUMP_BL   
    0.8607  LOADLOC2  
    0.8244  STOREFREE 
    0.7828  LOADLOC4  
    0.7735  GREATERP  
    0.7659  JUMPNFLAG 
    0.7408  JUMPLITNE 
    0.7264  CALL2_1   
    0.7089  LOADLIT   
    0.6966  ADD1      
    0.6903  CARLOC2   
    0.6749  LOADLOC7  
    0.6739  JUMPT_BL  
    0.6556  CADR      
    0.6501  JUMPNATOM 
    0.6448  JUMPFLAGP 
    0.6395  CALL2     
    0.6369  JUMPLITEQ 
    0.6357  LOSE      
    0.6287  STORELOC  
    0.6274  SWOP      
    0.6003  LOADLIT4  
    0.5577  STORELOC5 
    0.5397  APPLY1    
    0.5004  LOADLOC9  
    0.4910  CDRLOC2   
    0.4772  JUMPL2T   
    0.4630  CARLOC10  
    0.4404  LOADLIT1  
    0.4196  CARLOC0   
    0.4178  CDRLOC0   
    0.4173  JUMPNIL_B 
    0.4029  JUMPFREE3 
    0.3861  STORELOC4 
    0.3767  STORELOC3 
    0.3659  NCONS     
    0.3613  LOADLOC8  
    0.3589  PUSHNIL   
    0.3164  JUMPB2NIL 
    0.3108  JUMPFREEN 
    0.3030  JUMPATOM  
    0.2955  JUMPFREET 
    0.2879  PUSHNILS  
    0.2823  LOADLOC   
    0.2672  CALL1_3   
    0.2587  CARLOC1   
    0.2554  CDDR      
    0.2515  CARLOC5   
    0.2469  CALL1_1   
    0.2457  LOADLOC11 
    0.2298  LOSES     
    0.2275  PUSHNIL3  
    0.2109  JUMPLIT2N 
    0.2024  NUMBERP   
    0.2013  CALL3     
    0.2004  JUMPT_L   
    0.1952  JUMPL0T   
    0.1930  CALL2R    
    0.1910  JUMPLIT1N 
    0.1840  JUMPL1T   
    0.1835  LOSE3     
    0.1789  STORELOC7 
    0.1785  CALL2_4   
    0.1706  CALL1_2   
    0.1696  CAARLOC2  
    0.1673  POP       
    0.1645  JUMPL1NIL 
    0.1582  LOADFREE1 
    0.1533  JCALL     
    0.1516  JUMPNE    
    0.1512  CARLOC4   
    0.1463  LOADLIT2  
    0.1418  JUMPL1NAT 
    0.1414  CARLOC11  
    0.1382  CDRLOC1   
    0.1373  CDRLOC5   
    0.1352  JUMPL0NIL 
    0.1350  JUMPB1NIL 
    0.1251  LOADLIT3  
    0.1238  LOADLIT5  
    0.1180  LOADFREE3 
    0.1173  CDAR      
    0.1163  CARLOC7   
    0.1127  CAAR      
    0.1120  JUMPST0NI 
    0.1108  ACONS     
    0.1080  LOADLIT6  
    0.1077  XCONS     
    0.1061  CALLN     
    0.0986  JUMPT_B   
    0.0973  CALL1_5   
    0.0959  CARLOC9   
    0.0890  LOADFREE4 

----------------------------------------------------------------------
[some comments from when NAG were getting Axiom support fully up to
scratch: these are now probably only of historical interest!]

15 May [PAB]	sys*.c: get_truename should preserve '/' at end of string
		print.c: fix memory leak in Ltruename
21 May [PAB]	syscwin.c: use STARTF_USESTDHANDLES
17 Jun [MCD]    scandir.c: Changed the NT version of scan_files to check
                  for a trailing '\' before adding '\*.*'.
12 Aug [MCD]    all files: merged with latest version from ACN.
14 Aug [MCD]    fns3.c: Added C-coded version of subseq and auxilliaries.
22 Aug [TTT]    arith03.c arith11.c: changed quotib,Cremainder to treat 
                properly the case  (DIVIDE  -134217728 134217728)
23 Aug [TTT]    fns1.c: changed a comment to refer to *break-loop* not *break-function*
30 Aug [TTT]    machine.h: SYS_TIMES macro name changed because of collision
		with system-defined macro in HP/UX
		also affected :sys.h restart.c sysunix.c sysvms.c sysxwin.c
20 Sep [TTT]    fns3.c: added hashtable-flavour thingies
9  Oct [TTT]    sockets.c: added glnag calls for all platforms
		added  NANQ support	
28 Oct [TTT]    scandir.c: use realloc_wnull instead of realloc 
		to cure SunOS4.1.2 problems with realloc(0,...)
20 Nov [TTT]	read.c: typo Lrdfn Lddfn fixed              
20 Nov [TTT]    read.c: Lrdf4 : changed the Lopen to honour :if-does-not-exist
                 keyword
21 Nov [MCD]    All files: merged latest changes from Arthur.
 2 Dec [TTT]    gc.c :  added timnag_ license management call in reclaim
 2 Dec [TTT]    csl.c:  added init_lm call in ENTRYPOINT
 3 Dec [TTT]    sockets.c: changed product codes to AX???23NA
                added nullary init_lm function 
------1997
24 Jan [TTT]    sysunix.c: changed mkdir(filename,0770) to
                mkdir(filename,0777) to respect world setting of umask 
                in create_directory.

23 April [MCD]  externs.h restart.c csl.c fasl.c fns1.c fns2.c eval2.c
                Introduced mechanism (using symbol_protect_flag) to toggle
                protection of symbols in kernel.
10 July [MCD] csl.c Added initialisation of symbol_protect_flag to 1
              syscwin.c: Fixed bug in truename which stuck extra trailing 
                slashes on directory names, causing directoryp to fail.
19 Sep [MCD] csl.c : Disabled CCL break messages on interrupt
3 Dec [PAB] read.c, gc.c: Added call-counting mode to mapstore 
 (yes, it is a hack).

-------------------------------------------------------------------------
I have just manufactured a version of CSL that is (perhaps) easy to call
from other C code.  What follows is the main chunk of code that shows
what the interface is.  What this means wrt REDUCE is that I will want to
manufacture an alternative to (BEGIN) that reads-simplifies-prints just
one REDUCE expression/command at a time rather than the current single
function that gobbles things for ever....

/*
 * The next fragment of code is to help with the use of CSL (and hence
 * packages written in Lisp and supported under CSL) as OEM products
 * embedded within larger C-coded packages.  There is (of course) a
 * significant issue about clashes between the names of external symbols
 * if CSL is to be linked with anything else, but I will not worry about that
 * just yet.
 * The protocol for calling Lisp code from C is as follows:
 *
 *     cslstart(argc, argv, writer);allocate memory and Lisp heap etc. Args
 *                                  should be "as if" CSL was being called
 *                                  directly and this was the main entrypoint.
 *                                  The extra arg accepts output from this
 *                                  stage.  Use NULL to get standard I/O.
 *     execute_lisp_function(fname, reader, writer);
 *                                  fname is a (C) string that names a Lisp
 *                                  function of 0 args.  This is called with
 *                                  stdin/stdout access redirected to use the
 *                                  two character-at-a-time functions passed
 *                                  down.  [Value returned indicates if
 *                                  the evaluation succeeded?]
 *     cslfinish(writer);           Tidies up ready to stop.
 */
 
int execute_lisp_function(char *fname,
    character_reader *r, character_writer *w)
{
    Lisp_Object nil;
    Lisp_Object ff = make_undefined_symbol(fname);
    nil = C_nil;
    if (exception_pending()) return 1;  /* Failed to make the symbol */
    procedural_input = r;
    procedural_output = w;
    Lapply0(nil, ff);
    procedural_input = NULL;
    procedural_output = NULL;
    nil = C_nil;
    if (exception_pending()) return 2;  /* Failure during evaluation */
    return 0;
}

#ifdef SAMPLE_OF_PROCEDURAL_INTERFACE

static char ibuff[100], obuff[100];
static int ibufp = 0, obufp = 0;
static int iget()
{
    int c = ibuff[ibufp++];
    if (c == 0) return EOF;
    else return c;
}

static void iput(int c)
{
    if (obufp < sizeof(obuff)-1)
    {   obuff[obufp++] = c;
        obuff[obufp] = 0;
    }
}

#endif

int MS_CDECL main(int argc, char *argv[])
{
    cslstart(argc, argv, NULL);
#ifdef SAMPLE_OF_PROCEDURAL_INTERFACE
    strcpy(ibuff, "(print '(a b c d))");
    execute_lisp_function("oem-supervisor", iget, iput);
    printf("Buffered output is <%s>\n", obuff);
#else
    cslaction();
#endif
    my_exit(cslfinish(NULL));
    return 0;
}

...............................end


Arthur Norman.    2002, updated 2010 and 2017.

