Friday, February 28, 2025

Initiating New Projects via NixOS

I have a confession to make!  I'm kindof get delayed whenever I start a new project.  It can take a little while for me to set things up.

Several years ago, I worked at a cryptocurrency company -- and as a company, we had two goals.  The first is to find a "consensus algorithm" that would be able to approve transactions at about the rate that Visa and MasterCard can approve transactions -- because waiting a day for a transaction to complete (which is where Bitcoin was at the time) is kindof unacceptable for something that's supposed to be used as currency!  The second?  To explore other things that cryptocurrency can do -- and among them was the possibility of putting software on the cloud, compiling it, and trading it, all managed by cryptocurrency transactions and smart contracts -- and one package management system we were encouraged to investigate for inspiration was NixOS.  And I fell in love with it!

Now, just what is this NixOS thing?  It's actually several things:  a configuration language, a package manager, and an operating system, and maybe a thing or two besides.  As a language -- it is weird and complex, and a source of great headaches! -- but it's also where its power lies -- the language can specify exactly what you need, and customize things very precisely.  As a package manager, you can set it up on any Unix-like system and install NixOS packages that will work on the system (which, for Linux, means pretty much anything, because it's a Linux distribution, after all, but for Mac OS X, there's a *lot* of stuff available for installation!).  And as an operating system -- well, it's a Linux OS, after all, so you can install it on your computer.

I initially used NixOS as a package manager for Debian and for Mac OS X, but I have "graduated" to installing NixOS on a computer itself.  It was a bit of a challenge, but I don't regret it!  NixOS has  a solution for something that has annoyed me about other Linux distributions (and Mac OS X, too!):  whenever I have a "blank" operating system, I have to remember the applications I installed before -- and while I try to keep notes of what those applications are, sometimes I forget to update the list, and often I have to just "get to work" and run into a situation where I need Package X, but discover it's not installed, and take a moment to install it.

The only advantage to this approach is that, every time I have to install a new version, some of the older software I'm no longer using (often because it was a "one off" for an exploratory workshop or installed out of curiosity) "disappears" simply because I don't get around to re-installing it.

With NixOS, I can specify all the packages I want installed in a single configuration file -- or, if it gets complicated enough, I can break it up into several smaller ones -- and I can also include information on user accounts and preferences for each package!

Yet, even with NixOS on my system, I still insist on creating a little "shell.nix" whenever I start a new project.  Take my "HIVE" project, for example -- it's intended to be written in Common Lisp, but because it uses OpenGL, I need external GPU drivers and libraries installed as well -- that little "shell.nix" allows me to create a custom command line shell that installs SBCL and these libraries, and even sets up needed environment variables to make sure everything works.  I can specify the version of SBCL I'd like to use, the versions of the libraries, and anything else I might need -- and all these things are independent of the OS I'm currently running!

This is much like the "virtual environments" that computer languages like Python and Ruby use, so that you don't get stuck with the out-of-date operating system version -- or, if your operating system is updated to a newer version, you don't have to get stuck with a project that no longer works because of breaking language changes.  This is particularly valuable when you have a "legacy" project that you don't yet have time to update, and you're wanting to start a new project using a later version.

And this has, interestingly enough, also solved the "temporary package" problem I had before -- I can use a "shell.nix" file to temporarily install an application or two for a particular workshop -- or I can even do something like "nix-shell -p gimp" to drop me into a command line shell where Gimp is temporarily installed -- and once I close that shell, Gimp is no longer available.  (Well, technically, it kindof is still available -- NixOS doesn't automatically delete temporarily-installed applications -- so, assuming I want to use the same version of Gimp I used before, NixOS doesn't necessarily have to reinstall it the next time I use "nix-shell -p gimp".)

So, whenever I embark on a new adventure, one of the first things I do is create (or more likely copy) a "shell.nix" file, and start figuring out what I need for my project.  In the case of a "computer easel", I want to use FreeCAD, which needs Python -- and since I want to keep all the data for running FreeCAD "local" to the project, I had to take some time to figure out how to set up environment variables to inform FreeCAD where my "home", "data", and "tmp" directories were -- and I had to figure out where I wanted them.

But I have that working now -- and, as a bonus feature, if I wanted to share my FreeCAD configuration with others, I think I just have to share this "shell.nix" file and the above directories.  I'm not 100% sure about that, though, because I'm not 100% certain if I figured out where FreeCAD keeps all of its configuration.

Overall, though, this allows me to have complete-ish control over the setup.  While I have had some surprises over the years, even with NixOS, I have nonetheless appreciated having a single spot where I can maintain a project's dependencies, without having to figure out what's on the particular system I'm currently using!

Wednesday, February 26, 2025

Motivation!

So, for the past few days, I've been trying to overcome a combination of repeated interruptions by various errands and chores, and a strong sense of "Pervasive Drive for Autonomy" where apparently I cannot do something because someone is putting a demand on me, and mentally, I'm not prepared to fulfill people's demands -- even if that person making the demands is me, and the demands are work on one of those things you know you want to work on, darn it!

This post isn't about my motivation to do things, though.  This morning, as I was settling in to finally start a project, deciding to watch a couple of videos before actually starting it, thinking I'm not even sure if I'm going to write a blogpost (or even just post something) -- because I'm afraid that a blogpost takes away precious energy needed to work on projects -- and besides, I'm not addicted to videos, I can stop any time! -- I came across The Problem With Math Textbooks on Youtube Shorts that resonated greatly with me.

The TL;DW (too long, didn't watch -- wait, isn't this a short video? -- well, maybe it won't be there by the time you internet archaeologists get to this post) summary:

Pure Math textbooks delve right into the axioms, which is a problem, because students are left thinking that we could just pluck axioms from thin air, giving us infinite possibilities.  Where do these axioms come from?  We need to describe the motivation that led to these axioms!

This is, indeed, an approach I've been wanting to take with mathematics for years.  When I took a "Physics for Scientists and Engineers" class as an undergrad, my room mate was explaining that he was taking the Physics class that didn't use calculus -- and thus, the math was significantly harder! -- and this led me to the conclusion that both physics and calculus would benefit if they're taught as physics gives birth to calculus -- or, perhaps, rather, as both are given life as twins!

But I was initially at a loss as to how to find motivation for everything else -- when I realized I had answered this for myself years ago too!  The motivation comes from the history of mathematics.

  • Euclid's Geometry was motivated by an attempt to standardize the measurement of the Earth (hence the geo of geometry!) -- and its alternatives were motivated by attempts to prove that Euclid's axioms were the only alternative, only later to be discovered that they have their own physical analogs.
  • Calculus was motivated by physics, and each refinement to the idea by mathematicians like Euler, Riemann, Gauss, and Lebesgue, were done to address philosophical concerns, and to refine the techniques.
  • Modern Abstract Algebra was motivated by solving the classical Greek problems of trisecting the circle, doubling the cube, and squaring the circle, using only a straight-edge and compass.
As for everything else?  Well ... I'm not sure if I can tell you ... because I'm not as confident on the history as I'd like to be.  The problem the field of Mathematics has is that, as the math becomes more refined and purified, the older techniques are jettisoned -- and little to no effort is put into understanding the history!  A good example of this is in Calculus itself -- everyone who goes through the mathematics classes know about epsilon and delta proofs (My blog's nom de plume "Epsilon Given" is taken from this!) -- but far fewer know about Newton's fluxions or Euler's infinite and infinitismal numbers, among other approaches to the subject.

Indeed, my own understanding of the history of mathematics is a mixture of "The History and Philosophy of Math" class I took in my first year of college, and being self-taught.

To this end, I have spent some time trying to collect older mathematical works by early mathematicians, with the hope of exploring the more "intuitive yet unrefined" approaches to mathematics.  I have Euler's Elements, a work or two by Archimedes and another mathematician I can't remember, Euler's "Introduction to Algebra", a book of Leibnitz's works on calculus, and Sir Isaac Newton's work on calculus, A Treatise of the Method of Fluxions and Infinite Series, With its Application to the Geometry of Curve Lines -- wait, shouldn't that have been Principia?  Well, that was Newton's physics book, published in Latin, but using complex, difficult, and sometimes incorrect "simple" math, because Newton was highly jealous of calculus during his lifetime -- and thus, A Treatise of the Method of Fluxions was published posthumously in English (albeit translated from Latin).

Sometimes the motivation is simply "I don't know.  It seemed like an interesting problem at the time!"  And that, too, is good, because it's a reminder that sometimes we just have to play, and see where are games take us!

Monday, February 24, 2025

Thoughts on Practical Interface Design

Apple has some interesting notions about software interface design that I find amusing -- and also deeply irritating.  I appreciate their reasoning, but the axioms they use as a basis for their reasoning are deeply flawed!

One is "the five closest points to the mouse cursor are where the cursor is, and the five corners of the monitor".  This makes intuitive sense -- because you can "fling" a cursor to a corner, and it would come to a hard stop -- and this is the foundation for putting the menu of the active app on the top of the screen, rather than at the top of the window.

The problem with this, however, is that it only makes sense when the screen is the size of a postcard (which was literally true of the first Macintoshes -- ok, maybe my memory is skewed here, but they nonetheless were rather small) or maybe even the size of a VGA monitor -- but, as I discovered when one of my employers provided me a nice, giant, curved monitor, and a Mac laptop that I could plug into that monitor ... this entire dynamic changes!  When you're on the lower right corner, this principle puts the menu up in the upper-left corner, and when you can literally choose between "distance moved on screen" and "distance as the crow flies" to describe the distance one needs to travel ... and where it might even be reasonable to describe said distance in "yards" or at least "feet" instead of "inches" -- all of the sudden, this one principle requires me to pick up my mouse several times to reach the menu.

What's worse, I have come to appreciate a feature under Linux, where I could hover my mouse over a partially covered window, and it becomes "active" without pulling it to the top -- such a feature is useful to have a browser with helpful information over a command line window where you're trying to use that information -- and while this can be sortof emulated under Mac OS X, it can only be partially emulated, because if you had this feature, and needed to cross several windows to access the menu, the menu would have changed several times by the time you get to it!

Fortunately for me, when I was given a curved screen, my employer also provided me a desktop computer, and promptly had me install Linux, so I was able to install KDE, which gave me the Windows-style convention of "menu on each window".  It's tempting to say that this is Linux's style, or at least KDE's, but it's more accurate to say that the driving force behind Linux user interface design is flexibility.  Indeed, if I preferred MacOS's design decisions, I can easily find them in Gnome, an alternative to KDE.

Another principle is "we have studies that show it's faster to use the mouse than it is to use the keyboard to edit text, but everyone thinks it's faster to use the keyboard".  The "studies" they rely on involve asking random people to do mundane tasks like "go through this paragraph and replace every 'e' with an underscore '_' -- and, surprise, it's easier and faster to do this with "point and click" than it is to arrow down to each letter, and then make the replacement.

Of course, as an avid user of Vim, I cannot help but ask "Why not just visually select the paragraph, type 's/e/_/gc', and after highlighting all the matches, manually approve one or two of the search-and-replaces for a couple of of times, and then hit 'a' to change all other matches when you're satisfied with the result?"  The Apple response to this, though, is "We're designing an interface for the 'average' person, not the power user!" but the proper response to this is "Yes, it's nice to have simple-yet-painful interfaces for the 'average' person who is going to do anything only once or twice -- but we need to cater for the power users, too, because eventually, in at least some tasks, the 'average' person is going to want to cross over the line into 'power user'!"

So, yeah, I'm not entirely a big fan of Apple's user interface design principles.  They sound good in the abstract, but they have led the designers astray to produce some awful designs!

Since I'm in the process of trying to figure out how to create my own Dual-Quaternion based 3D CAD-like system, I've given some thought about how things ought to be designed ... and I think the guiding principle I'm most attracted to is flexibility:  Don't try to predict what any particular user is going to need, instead, provide the tools necessary for the user to create and customize their own interface!

The fundamental principles behind all these are perhaps the most important:  first, enable flexibility, second, put as many things under your finger tips as possible, third, it's nice to be able to select anything and everything and copy them, and fourth, don't clobber user data -- and everything is user data.  I also have a couple of principles than I don't yet know how I'll implement -- the fifth is take advantage of the strengths of every input method (for years, the default treatment of touch screens under Linux has been to treat the touch screen as another mouse, which is annoying when you touch the upper half of the touch screen, and it sends the information to the second monitor that isn't a touch screen), as well as a sixth, the user should always have complete control over the program (which will probably need me to figure out how to install and use Real Time Linux).

So, with these principles in mind, I have had the following generic thoughts about the user interfaces I'd like to try to implement:

  • Anything that a user can do, to the practical extent possible, should be captured in undo trees of some sort -- perhaps even made version-controllable,
  • All functions of an application ought to be available to the user, to be bound to any key, or touch, or mouse movement, or gesture, available to the user (as inspired by Emacs),
  • The command line is a special interface:  it allows us to describe what we are trying to do with text, and this enables scripting, as well,
  • For every workflow, it should be possible to work out a "language" that can translate to keys on a keyboard -- much like how Vim approaches text editing -- although that "language" might differ from team to team, or even individual to individual, or project to project,
  • There is only one place closest to the cursor, and that's where the cursor is at this given moment.  It should be simple to pop up a circular menu at that particular point, probably by right-click, which can then open up to other circular menus -- and every such menu should have a "computer" icon on top (at 12 o'clock), an "application" menu just to the right (at 11 o'clock), an "environment" level just to the right at that (10 o'clock), and an exit button at the very center,
  • Perhaps every "icon" should have a character point in UTF, to the extent possible, and every help text, warning, and possibly graphic should be selectable by mouse and copyable to the right medium,
  • It should be possible to create menu-ish panels that consist of easy-to-access functions, information to be watched or examined, and icons to access various things; every such panel should be "locked" into place, with a little "lock" icon, that must be explicitly unlocked before menu items can be added or removed, or the panel itself be moved or resized (I have been both impressed by Ansys's ability to manipulate "default" menu and information panels, and annoyed by how easy it could be to accidentally change them, without knowing how to undo the changes, or even having the option to undo them! which, to be fair, I think Ansys has, I just don't remember how to use it),
  • It should be possible to view all panels available at any given moment, even if some of them are hidden, via some sort of "explosion" mechanism that keeps everything in place unless someone moves them (much like MacOS and KDE has for windows, although both seem to randomize what they show, and neither provides ways to organize the windows on the screen in this mode, without mechanisms to preserve these changes between "explosions"),
  • Everything associated with a project should be kept in a structured file format that can be explored by command-line tools, and in particular, as text files where possible, and while automatic strategies can be provided for inserting new things into these files, any changes made by a user needs to be respected -- and any changes that a user might do, that would break the system (eg, syntax errors, system display changes, etc) needs to be handled gracefully by the system -- so that the user will feel free to experiment without fear of everything coming crashing down (to the extent possible -- we are dealing with complex systems, after all, and we cannot fully understand what we are doing!) -- in other words, as I currently envision it, I intend my projects to be "text editors" that can keep track and edit "non-text" information.
I think some of these are contradictory, some of them may prove to be impractical, and while I have given these notions a lot of thought over the years (well, some of them -- my menu and panel ideas are relatively new), I do not know how they will work out in practice.

But then, if I knew what I was doing, it wouldn't be research, now, would it?

Friday, February 21, 2025

Identity Management: Introduction to Molecules: How Atoms Come Together

So far, I have covered all the fundamental elements that are needed for the Calculus of Identity Management.  Recall that they are:
  • Randomness, because unpredictability is hard to break,
  • Universal Unique Identifiers (UUIDs) and Nonces, for identifying unique things, and for providing unique tokens,
  • Cryptographic hashes, to check that documents have not been altered, or to keep passwords from prying eyes,
  • Symmetric keys, to pass data between two people, keeping prying eyes from understanding it,
  • Asymmetric keys, to share symmetric keys in a safe-ish way, and to confirm other people's identities,
And, as an honorable mention,
  • Steganography, because it's sometimes fun to hide data under people's noses!  And, er, because it can sometimes be useful to "watermark" pictures!
None of these things, by themselves, provide identity management.  They need to be brought together to do interesting things!  And this is where things get interesting -- because while we only have a small number of tools to manage identities, they come together in rich and complex ways -- in much the same way that atoms are relatively boring, but come together to provide innumerable molecules that make the universe possible.

As I cover these patterns, I won't be doing it in any particular order (beyond my personal whimsy); indeed, as these are sufficiently complex, there might not be an "order" to them that makes sense!

Wednesday, February 19, 2025

Work in Progress: Debugging Sweet Expressions

This is intended to be a brief report, so I'm not going to explain what Sweet Expressions are or why I am interested in them (although such an explanation, does, indeed, call for a "Curious Treehouse Musing") -- but in the process of trying to get this library up and running, I ran into a bug: it turns outh that Sweet Expressions cannot process a comment that comes right after a regular sexpr.

In the process of trying to wrap my head around the Sweet Expressions source code and figure out where and how comments are processed, I created a little macro to help me see how functions are called:

				
(defmacro debug-defun (fn-name fn-args &rest fn-body)
  "Like defun but adds debugging info around the function definition."
  (let* ((first-line (car fn-body)) ;; because the first line sometimes has to be the first
         (has-declare (and
                        (listp first-line) (eq (car first-line) 'declare))) ;; declare check!
         (body-bits (cdr fn-body)))  ;; the remainder of the body
    `(defun ,fn-name ,fn-args
      ,@(if has-declare `(,first-line))  ;; (declare) form?  Put it here!
      (if *debug*
        (progn
          (push "  " *current-level*) ;; "increment" the level
          (format t "~{~a~}IN FUN ~a~%" *current-level* ',fn-name)))  ;; print out level/name of fn
      (let
          ((function-result (multiple-value-list (progn ;; We need to capture the return value
                                                  ,@(if (not has-declare) `(,first-line)) ;; Not declare form?  first-line goes here!
                                                  ,@body-bits))))  ;; put last of body here
        (if *debug* (pop *current-level*)) ;; "decrement" the level -- we're done calling
        (values-list function-result))))) ;; return the result!
				
			

This macro can easily be "deactivated", too, by commenting it out and uncommenting this macro:

				
(defmacro debug-defun (fn-name fn-args &rest body)
  `(defun ,fn-name ,fn-args
     ;(format t "Hello, ~a~%" ',fn-name)
     ,@body))
				
			

But it turns out that this macro probably isn't as helpful as I thought it would be: I still struggle to see what value each symbol has, or what the algorithm is doing, at any one moment. It is the byproduct of how I am used to debugging code: with "print" statements! In my years of using PHP on web servers, Python on Amazon Web Services, and JavaScript, it's very easy (and sometimes necessary! because you don't have run-time access to an Amazon "Lambda" instance! among other scenarios) to use print statements or logs to show the states of variables.

However, in my previous internship, we were using Qt in a Windows application -- and while I probably could have used logs to get my bearings, and I sometimes even modified labels to confirm that I was looking at the right spot, it was more convenient to put in a breakpoint, attach to the running process, and step through the code. This, in turn, made me wonder how I can do that in Common Lisp -- and even accept that this might be the easiest option for debugging this issue.

Nonetheless, I appreciate creating that macro, even if I'm not going to use it. It was an interesting learning experience, helping me to better understand the power of Common Lisp macros and how to write them!

So, today, I'm going to try to do a "deep dive" into Common Lisp breakpoints and stepping. I also need to figure out how to run the Sweet Expression tests, or figure out my own way to do so -- and I need to create a test or two that would cause "Sweet Expressions" to choke, and then fix the issue.

Also, at the Experimental Airplane Association, this week is Virtual Ultralight Days! So I am going to try to catch a webinar or two on flying ultralights. (I have a special interest in aviation, and have a particular soft spot for ultralights, because my Grandpa built and flew one.)


Fun fact: For this post, I had to figure out how to insert code into Blogger; I found this blogpost and this help forum to be useful, although I had to figure out on my own that I needed to add the CSS into the blog's HTML, because the "Add CSS" option recommended in the instructions didn't exist...

Tuesday, February 18, 2025

Of Mice, Keyboards, and Men

Back in the ancient days where Intel 486 and Pentium processors were state-of-the art, monitors were vacuum tubes, 3 1/2" floppy disks were the the best option for transferring data between computers, and the best options for connecting between computers were by dialing into BBSs (Bulletin Board Services) via telephone and shareware called "QModem", I had an interesting realization:  that someday, all this wonderfully advanced -- and quickly advancing! -- computing technology will mature to the point where most people wouldn't feel pressure to upgrade every six months, or lament over the expensive computer that became out of date a month after purchasing it (which always seemed to happen whenever anyone purchased anything).

How did I reach that conclusion?  I realized that keyboards and mice were already there, and it would only be a matter of time before everything else got to that point, too.  It took about two or three decades to do so, but we're at that point today!  Well, monitors can probably advance a little bit to Augmented Reality setups that would essentially make 5-monitor systems portable -- or, for physical monitors, to have full-color movie-capable electronic ink (and now that the patents are close to expiring, maybe we'll finally start seeing advances in that direction!) -- but we have somehow even reached this point for silicon chips!  While I expected this to happen for silicon eventually, we didn't quite reach that point for the reason I expected:  I thought that processors would just get "good enough" to the point where most people wouldn't care, but instead, via the magic of Moore's law, we pushed them to the limits of our knowledge of physics.  I don't think I fully appreciated the role that 3D video games in particular, and the heavy computing needed for engineering and meteorology in general, would push technology to this point.

So, naturally, knowing about this technology plateau, I've given a lot of thought in how I'd design a keyboard and a mouse.

Wait, what?!?  Why the heck would I spend so much time and energy on stale, mature technology???  Do I expect to significantly advance the state of the art of these input devices?

No, no I don't.  I am interested in keyboards and mice because I want to customize them -- and, perhaps, to make it easier for others to customize their input devices, too.  Granted, this isn't for the faint in heart -- anyone who is perfectly satisfied with their $15 squishy keyboard and plain $10 mouse probably won't be interested in pushing these things to their limits -- but it can be fun for those who wish to explore, and may even make using these things a little more comfortable, particularly for those of us who are using these devices for several hours a day.

When I started working at my last position, the IT Department (aka Joe) set up a nice computer with a GPU, two monitors ... and a spongy computer and simple mouse.  My wife (who was also a co-worker at the time) suggested I ask Joe for a better keyboard and mouse -- but I didn't do so, for two reasons.

First, I have a perfectly serviceable keyboard and mouse that I had chosen both for their features and for their portability.

My keyboard, something like this Keychron K8 with blue key switches, was "only" $50 when I purchased it several years ago -- at the time, I was expecting to pay $200 when I finally decided to pull the trigger on a mechanical keyboard -- and before I pulled that trigger, I purchased a "sampler" of twelve different kinds of keyboard switches, each with their own levels of "tactileness", "hardness", and "clickiness" -- ultimately, I chose the blue keys, because they had a nice internal tactile "click" that lets me know I made contact without having to bottom out, they required only the softest touch to be activated, and they had a nice loud "clickity-clack" sound while typing.

My mouse, a VicTsing Pioneer, "only" cost $35 -- this mouse no longer seems available, and as far as I can tell, but is similar to this Rapoo MT760 mouse, which looks like it's $50 -- and it was chosen for its a nice horizontal thumb scroller I use regularly, a couple of thumb buttons I wish I could figure out how to use better, and overall was pleasant for me to use except when it froze up (which, unfortunately, it does fairly regularly, particularly when the power is getting low).

Second, to the degree I'm interested in getting a different keyboard and mouse, I want to get something significantly different!  I want a split keyboard with staggered columns instead of staggered rows, with each thumb having half a dozen small keys available (as opposed to just a giant spacebar and a couple of hard-to-reach modifiers), lots of layers, and the ability to attach several devices via Bluetooth and USB so that it could pass everything on to the computer the keyboard is attached to -- essentially, making it as if the keyboard, mouse, and headphones all shared a single dongle or Bluetooth connection -- and I want a mouse that has three push-button horizontal scrollers for my thumb, two buttons each for my two fingers, and a vertical scroll that could be tipped left or right as well as pushed straight down -- or, perhaps even have three vertical scrollers, each being tippable either direction -- and the mouse should have a wireless dongle that can fit into the keyboard.  Lately, I've been wondering if I could make all the buttons pressure sensitive -- allowing any one button to play the role of "Wacom tablet pen", but for mouse movement.

Oh, and I'd like to try a Space Mouse as well!  (Which was potentially relevant for my internship!)  I'm not entirely sure if I'd like it, but I like to experiment -- it's just that I haven't yet been sufficiently immersed in the 3D world to justify the expense!

So, yeah.  I opted to choose to bring my own keyboard and mouse, rather than ask for equipment I would like (and would have had to leave behind, anyway).  I had to buy a Bluetooth USB dongle to get my setup to work, but everything worked fine!  (Except when it didn't.)  I want too much customization, and for what little customization I had, I was comfortable with what I have found so far.

Incidentally, I used to hate Bluetooth, but it's grown on me.  While I would still rather use a USB wireless dongle than Bluetooth, I'm to the point where anything I make that has a dongle, is going to have Bluetooth, too.

Monday, February 17, 2025

Transcendental Computer Languages: Or How I Learned to Stop Worrying and Embrace the Lisp

When it comes to computer languages, there is a mantra popular among software engineers.  "Use the right tool for the job!"  For every project you might want to work on, there's a computer language just right for that project -- what's more, for every aspect of that project, there's the perfect language!  Every software developer should have a dozen or two languages in their toolbox, ready to go, to do do everything effortlessly using the best tool possible.

While computer languages seem to "specialize" for different "use cases" -- Assembly language is a language unique to a given processor; a Systems language is "close to the metal", requiring you to do "bookkeeping" chores by hand, but at least gives you "if/else" branches, "for" loops, and "functions"; and from there, higher level languages that manage memory for you, and have a lot of features built in or easily available as libraries.  If you're doing something that's "throw away after using once" or is so dominated by waiting for user input and internet communication times that the processor is mostly idle, you don't really care if you stop to automatically every fem moments to clear out some memory -- but if you're working at the inner core of the operating system, you'll want to use a Systems language where every bit of memory is managed by hand, and every moment is timed perfectly ... and where you can "dip into" Assembler if you're doing something particularly esoteric.  This naturally creates a hierarchy of languages, from "low-level" to "high-level", with the most manual at the bottom, and the most hand-holdy at the top.

So you wouldn't use a Systems language when writing an operating system, right?  And you wouldn't use a high-level language when working on the operating system kernel.  And you'd want to use a language specializing in web development for creating and managing websites, or a data processing language to do statistics and analysis, or a UI language for designing user interfaces, or if you want to, well, ....

Yeah, about that ... for all this talk about "The right tool, darn it!" there's an awful lot of "settling" into just a handful of tools.  C for systems programming, PHP and HTML/CSS/JavaScript for web programming, MySQL for databases (put aside that PostgreSQL may be a bit better), Python for data crunching, C++ for some applications ... and I'm not at all convinced that some of these are really the "right tools".  JavaScript, in particular, dominates its space not because it's the world's greatest language for capturing interface design and web interaction, but because it's pretty much the only option offered it its space.  C and C++ dominate where they do mostly because of a mix of inertia and Tradition, tradition!, particularly with languages like Rust on the horizon, promising to offer systems programming with the safety of scripted languages, somewhat.

The funny thing about most of those languages, is that most of them are "Turing complete" -- which is a fancy way of saying that every language can do anything any other language can do -- and because it can take so much time to master a language, and that language environment becomes a comfortable place to live, and it's such a hassle to download and install a new language ... all too often, the "best tool for the job" is simply "the one the project's already written in" or "the one I'm most comfortable with" (or, in the case of companies, "the three or four languages we think most of our software engineers are comfortable with") ... and because many languages are "general purpose", they can do everything you need to do, even if some of what you do is "against the grain" of the language itself.

Yet, despite this, there are all sorts of reasons for learning new languages!  Many years ago, when I was starting out as a web developer and software engineer, I stumbled onto a weird "functional" language called Haskell.  At the time, my brother-in-law asked me "Why are you learning that?  Where are you going to use it?", and I don't think I answered the question well at the time (or even perhaps at all), nor do I think I can coherently answer it right now.  As vague as I have to be about the answer, however, I can at least provide a few suggestions:  it's a new paradigm that, if I understand it, can help me be a better programmer; it's something new and interesting, and I crave variety, because one can't live life solely dedicated to only things that are "useful" (although I probably crave variety more than most people); as a mathematician, I couldn't help but admire its pure mathematics roots (yet, as an exploratory software engineer, I simultaneously despise its pure math roots, figuring one shouldn't have a PhD in mathematics to comfortable programming!); it's weird, and I love weird things; who knows, maybe it will prove to be useful after all?

And besides, eventually, it did prove useful!  I spent a few months using Haskell to help implement a "consensus" algorithm for a cryptocurrency ledger.  Also, several years ago, I started seeing weird themes crop up in JavaScript meetups:  algebraic types -- for better and for worse, Haskell-ish things were making their way to the web!

But those use cases were well into the future.  In the meantime, as I was first learning Haskell, I gradually developed the notion of a "transcendental" language -- a language that could reach down into the lower levels of the system, and eek out performance there, yet stretch out to the highest levels of abstraction, and provide simplicity without worrying about all those "fiddly bits" that makes low-level programming such a pain.  Ironically, I would shortly conclude that Haskell isn't quite transcendental, as powerful as it may be:  it's too rigid, it requires too much understanding of higher mathematical ideas, and it's too "bulky" (someone once pointed out to me that the compiler is about a gigabyte in size) -- to really be "transcendental".

But my exposure to Haskell introduced me to a language that is transcendental.  While Haskell is the first pure functional language, the first functional language came out in 1958, "specializing" in "LISt Processing", created a year after a language specializing in "FORmula TRANslation" came on the scene.  Because Haskell is a relative newcomer compared to these languages, when Haskell enthusiasts extolled the virtues of "functional programming", they relied heavily on essays describing the power of LISP.  And thus, I started down the path of learning about Common Lisp and Scheme.

(I had technically encountered LISP before, in a book comparing four different languages, but the book was so focused on showing how you could do the same things in four vastly different languages, it somehow made the languages seem equally capable, so at the time, I wasn't convinced I ought to explore LISP.  Having said that, it's been years, and I may be misremembering -- or may have misunderstood -- the tone of the book.)

Unlike "FORTRAN", which really was a language that specialized in formula translation (and to the degree it departs from this, there's the saying that "No one knows what language scientists and engineers will use 100 years from now, we just know it's going to be called FORTRAN"), LISP didn't exactly specialize at all.  LISP introduced pretty much everything we expect in a modern language -- functions as first class citizens, garbage collection, symbols, variable and keyword arguments to functions, anonymous functions, native data structures and the means to create new ones -- well, maybe not everything we expect -- things like object-oriented programming and exception handling didn't come until later -- but it managed to literally absorb those things as they came down the pike -- and LISP includes, even now, a few things we don't have in computer languages, such as macros written in the language itself -- taking advantage of the fact that every program written in LISP is a list of lists, which are essentially trees, which are easy-to-manipulate data structures -- and thus, it's possible, even easy (or at least, easy-ish, because some problems are inherently hard) to create "mini languages" that are just as much a part of the language itself, as any other thing you might do for that language.

Thus, LISP has this ability to reach into the very bowels of a system, even resorting to Assembler if it has to, yet it can easily reach the highest heights of abstraction.  LISP is a transcendental language!

LISP isn't the only one, to be sure.  I wondered whether FORTH should be considered one, because it's low-level and doesn't have garbage collection, but a demo of FORTH on an Arduino with 2 kilobytes of memory convinced me it deserves to be included in the category.  I also figure Smalltalk, too, should be included, although it's not well known as something that can reach down to lower levels -- but it's not hard to see how that ability could be added to a flavor of Smalltalk, if it is needed.  Are there others?  I don't know.  I do know, however, that some of my favorite languages -- APL and J, in particular, and Elixir as well -- have ideas I appreciate, but nonetheless don't manage to transcend the differences between different levels of language.  (And for Elixir, I wonder if I'd change my mind if I understood the language better.)

It is interesting to observe that these three transcendental languages have been used to write operating systems -- this is only natural, considering their ability to reach down into the bowels of things -- and these operating systems are known for their flexibility, which is the result of their ability to abstract ideas.  This doesn't necessarily mean that only transcendental languages can be used to write an operating system -- BCPL, C, C++, PL/1, and even Pascal and FORTRAN are low-level counter-examples, and I'd suggest that Python and Ruby, while not really operating systems, can be counter-examples as well -- but overall, if a language truly is transcendental, it's going to have a stronger "gravity" towards becoming an operating system than other computer languages.

But then, this is likely a red herring, anyway.  At some point, I think I'll write a post explaining why all languages want to be operating systems!