CardiganBay

My new WikiEngine written in ClojureLanguage

See it on GitHub and on GitLab

We have now replaced the old ProjectThoughtStorms engine for ThoughtStorms. This public site is a static rendering from a wiki managed on my local machine using CardiganBay.

A Quick Summary : InstallingCardiganBay, WhyIsCardiganBayLikeThat?, CardiganBayFeatures

Cardigan Bay is a "tool for thought" / "personal knowledge management" / "dashboard" style tool. See CardiganBayAsToolForThought for an example of how you might set it up for that.

CardiganBay motivations

I need a new wiki-engine that can

• Be the engine to manage and maintain my long-running, publicly facing, and somewhat in need of cleaning up ThoughtStormsWiki. Especially to help with the "cleaning up" bit.
• Be an engine to manage a couple of smaller, more specific wikis and sites, with some specialized media types embedded in pages. (See CBZones for further discussion of this and the previous functionality)
• Be the engine to run my own private notebook of ideas. Replacing my long defunct SdiDesk and more recent OutlinerWithWikiLinking
• Be something I can use when travelling. To work on these notebooks even when I'm away from my main machine or off-line.
  • In particular be usable on tablet and phone and help to answer the question WhyIsMyPhoneSoUseless?
• Be a tool to help recapture and re-integrate thousands of answers I have written on Quora and posts I've made on blogs and other social media, into my personal DigitalGarden
• Be a platform not simply for writing and capturing information, but for making sense of, and finding meaning in it. See SenseMaking.
• Be a platform to help me DO things as well as write about them.
• Be a platform to explore new ideas and functionalities for wiki.

Goals

The wiki-engine should

• Be built with new technologies appropriate to 2020
• Nevertheless embody some of the classic wiki values and ideas that I've found valuable over the last 20 years.
• Be compatible with useful existing contemporary standards.
• And a viable citizen in an internet of the 2020s, in a world dominated by social media etc.
  • Support embedding to and from other social media.
  • But nevertheless help to promote other open / free standards
  • And collaboration with other wikis
• Support transclusion
  • from one page to another (to help reduce redundancy)
  • from one CardiganBay wiki to another
  • eventually from other wiki-engines
• Run on servers, local desktop machines, and mobile devices like tablets and phones.
• Be great for capturing new information. But equally useful for cleaning, refactoring and reworking it to keep it up-to-date and relevant.
• Manage several wikis from a single instance of the engine. And to be able to perform operations that cut across them. Eg. to migrate or copy pages or cards from one to the other, to do diffs etc.
• Be able to publish collections of pages as stand-alone sites.
• Be able, increasingly, to analyse and ask questions about the contents of the wiki. To help us classify, fix broken links, merge pages, discover interesting insights and overviews.
• Host executable code, making it an active notebook similar to JuPyter Notebook etc.
• Support user customization. Where possible, customization information should live within the wiki and be editable by the user.

Design decisions

• A classic wiki with named pages.
• Names of pages are ConcretePageNames. (Ie. the name is the id of the page) This allows for spontaneous discovery and link-making (when writing on one page, we can guess the names of relevant other pages without having to look them up.)
• Pages are made from a sequence of "cards"
• Each card has an explicit type. By default, Markdown, but other types for other media or data which are rendered (or even editable) differently. This is how we support embedding from YouTube, SoundCloud, BandCamp etc. There's an Embedded type which is used for embedded media from elsewhere. (EmbeddingExamples)
• Despite the emphasis on cards, full pages always have a canonical representation which is a single, plain-text or SmartAscii "source" file. And pages can be edited in their entirety in a single box. (This was my main beef with the SmallestFederatedWiki)
• Borrowing from SmallestFederatedWiki the wiki engine is intended to have a single user. The expectation is that collaboration happens when each user has their own fork of the wiki data.
• Unlike SFW we rely on external tools such as Git to manage and merge forks rather than trying to do this ourselves.
• The wiki engine is a personal tool rather than intended for a public server. The assumption is that public facing sites will be exported as either completely flat sites, or minimal engines without editing capabilities.

Technical Decisions

• The wiki is a Single Page App written in Clojure / ClojureScript.
• It uses Reagent (the ClojureScript wrapper for React) as its client-side framework.
• All UI components are therefore written in hiccup format
• Communication between client and server is mainly through GraphQL
• Markdown is the default markup type
• By default cards of other types that need to contain some structured data will use EDN
• Where possible embedding uses oembed.
• Pages are stored as simple text files in the file system
• Assume git for managing page history / version control, rather than writing our own.
• We capture information about the collection of pages in a Core.Logic database. And, as much as possible, use core.logic logic programming to query and reason about it. Eg. to find broken links, orphaned pages etc.
• As a Clojure program, the wiki engine runs on the Java Virtual Machine and can be distributed as an Uber JARFile, without the need for potential users to install or understand any dependencies.

Code Road Map

OliSharpe asked in the comments of one of my videos :

It's interesting to see that you've added some cards that are workspaces in which you can test out bits of code, but I'd be curious to know if you've got longer term plans to make particular cards 'callable' from other cards, as this would enable Cardigan Bay to support the cumulative writing of functionality.

I also suspect that if you are going to go down such a route, then you will have to start thinking about authentication and authorisation type issues - but admittedly I'm expressing the kinds of directions that I'd be interested to see this go, which may not align with your short term goals (even if your long term goals include everything ;-) ).

I imagine that your shorter term goals are to have a sophisticated wiki type engine that does exactly what you want such an engine to do.

My answer :

Thanks for the comments.

Yes. Very much so.

So you can kind of think of there being multiple "milestones" which are somewhere between "short-term items already on my todo-list" through to "impossible fantasy goals which we might evolve into a long way in the future"

And of course, it's inspired by things I've been thinking about (and I'm sure talking to you about) for a long time. Here's some older thinking from 15+ years ago : http://thoughtstorms.info/view/ProgrammingWithAndInWiki and 3 years ago http://thoughtstorms.info/view/SmalltalkNeedsANewUI

So right now, that Clojure interpreter is using https://github.com/borkdude/sci and isn't very useful because it doesn't have access to any libraries.

So first milestone is to figure out how to get some basic, standard libraries into it The target is that it should be possible to write some basic mathematical equation and produce a nice graph of it. All within a single card. So a kind of "graphing calculator" model.

The second is more like IPython Notebook / Jupyter (https://jupyter.org/) where each cell (or card) is cumulative within a page. In other words, data or functions defined in one card are visible and available to cards further down the page.

Alternatively, yes, rather than automatically use card sequence, have a system where you can explicitly reference other cards in the same page. Eg. I've thought of adding a :labelleddata or :labelledcode card type which provide extra name-spaces that can be imported into Workspace cards.

Next milestone from that would be to allow you to build up a library of code over multiple pages. Or even multiple servers. And import from anywhere.

The main "security" idea I have so far is that cards have an id / reference which is the hash of their content. That will be used in any transclusion or import. So if I decided to pull in a library from Bob's server. I use an ID that depends on what that code is. If Bob changes the code later, then my reference immediately breaks. I, as owner of my server, have to go and look at Bob's new code to decide if I still want to use it. It's not perfect, but it means Bob can't sneakily change what his library does without me knowing.

The next milestone after that is to ask how much of the front-end (in browser) system can actually be built in the scriptable layer itself. Ie. available in user-editable code. This is the Smalltalk inspiration, of course. To have the whole environment built in itself and available to the user. In contrast, my client UI is built in ClojureScript. But it's mainly ClojureScript that gets compiled to Javascript back at compile-time. So not amenable to being inspected and customized by the user.

There are a few Smalltalks that run entirely in the browser these days. And I was thinking of adopting one for my front end. The thing about Smalltalkers, though, is that they are very wedded to their desktop GUI. Whereas I'm now describing myself as "militantly one-dimensional". I didn't want the usual Smalltalk WIMP GUI. I want my nice 1D sequence of cards. Earlier this year I spent some time looking into https://newspeaklanguage.org/ because that looked like it was a completely dynamic "live" environment that had moved beyond the traditional Smalltalk GUI framework and gone native on the web.

And in the very long term fantasy world ... perhaps there might be a whole new front end based on something like that.

But right now ... I know and like Clojure a hell of a lot. And it's just easier to build the client side in ClojureScript and reagent (which is React behind the scenes). Even NewSpeak turns out to be a kind of, I won't say "kludge", but a port of something that underneath is more like the Smalltalk GUI framework, with the web-stuff as a special rendering layer on top, rather than stripping out that GUI framework and making something simple for the browser.

So I'm still keeping an open mind, and an eye on that. Smalltalk is another great language. And I believe that if someone did do what I was suggesting in that page on SmalltalkNeedsANewUI, then it would be fantastic. But if I ever go there, I'll go there incrementally via this.

So it's really first couple of milestones I'm focused on now. Firstly, making sure some useful libraries are available for code running in the workspace. Then figuring out how to build larger code across multiple cards within a page.

I think by the time I do that, I'll have a better idea as to whether it would be possible to migrate more of the client itself to this scriptable layer.