Tim Disney

Stuff I Enjoyed in 2025 - Publications & Writing

2026-01-27T00:00:00Z

Continuing my (delayed) 2025 reflection series with the publications and articles I most enjoyed reading in 2025. Previously: Stuff I Enjoyed in 2025 - Books.

My religious read for the year was Garbage Day. They do a great job of contextualizing the wild and frustrating events in an entertaining way. Also great is Today in Tabs with such sharp writing.

The (semi)political writers I followed included John Ganz, Henry Farrell, and Dave Karpf.

The Digital Antiquarian has also been a really fun read going through the archives of old games. The Space Sim's Last Hurrah and Alpha Centauri in particular were great. Alpha Centauri and Freespace 2 were two of my favorite games growing up and essentially no one remembers them.

To make sense of all the AI developments Simon Willison's blog was indispensable. Also great was Cognitive Resonance putting things in a cognitive science framework, AI Snake Oil/AI as Normal Technology and their seminal AI as Normal Technology article, Ethan Mollick from One Useful Thing (taken with a grain of salt), and Venkatesh Rao for seeing someone explore going all in without being a complete grifter.

Favorite articles:

(note: I'm not necessarily cosigning the claims or ideas of any of these articles. They just got me thinking the most in 2025.)

We're getting the social media crisis wrong

The fundamental problem, as I see it, is not that social media misinforms individuals about what is true or untrue but that it creates publics with malformed collective understandings.

This reframing is just great.

Be Slightly Monstrous

The real conceit here is about consciously surrendering to a machinic reshaping of your patterns of cognition. Your ways of seeing. Your ways of feeling into reality.

He's exploring something interesting here. Not sure I like it but it's interesting.

Revenge of the Dilettantes

The future is bespokeness.

The claim is something like AI is in the process of driving the cost of production so low that there won't be any reason not to spin up bespoke productions for basically any creative output.

Why Some Students Learn Faster

Researchers divide intelligence into two parts: crystallized intelligence and fluid intelligence. Crystallized intelligence refers to how much you know: knowing things makes you smarter. [...] Fluid intelligence is a broad term for how well we can think and reason when we don’t have crystallized intelligence to rely on.

I hadn't come across the distinction between crystallized and fluid intelligence before so this was a bit of a revelation for me.

The Bookmaker

But what Silver willfully ignores is that the successful players in this world aren’t the bettors. They are the bookies and casino owners—the house that never loses.

Gambling is bad.

My AI Skeptic Friends Are All Nuts

Some of the smartest people I know share a bone-deep belief that AI is a fad — the next iteration of NFT mania. I’ve been reluctant to push back on them, because, well, they’re smarter than me. But their arguments are unserious, and worth confronting. Extraordinarily talented people are doing work that LLMs already do better, out of spite.

My read is that the consensus has definitively shifted after the release of Opus 4.5 at the end of the year to acknowledge coding agents are good actually but during the year there was a lot of discourse. This was my favorite piece trying to convince the skeptics.

The Zizians and the Rationalist death cults

Feeling comfortable with your own epistemological position, even if you know it’s flawed, is not the preferred mode for Rationalist development, but it’s pretty foundational to building a stable sense of self. By the same token, the ability to dismiss an argument with a “that sounds nuts,” without needing recourse to a point-by-point rebuttal, is anathema to the rationalist project. But it’s a pretty important skill to have if you want to avoid joining cults.

I think this is a sharp insight into one of the flaws inherent to the Rationalist community.

AI as Normal Technology

This was probably the most important article about AI in 2025. A well thought through middle path between "the singularity is coming in 2027" and "AI is a fad like NFTs".

Skyreader: A RSS Reader on the AT Protocol

2026-01-20T00:00:00Z

I've just released Skyreader, a RSS reader built on the AT Protocol.

login with your bluesky account
share articles like Google Reader but on the AT protocol
your feeds and shares are stored on your server, not locked in my app

RSS is neat #

You should be using an RSS reader.

RSS readers are simple. They manage a list of websites you care about and show you new articles when they're posted. No algorithm deciding what you should see.

What makes Skyreader different #

My favorite feature from the late, great Google Reader was article sharing. You could share articles to your followers in one click. It was a low-effort way to see what your friends were reading and share what you found interesting.

That feature died with Google Reader. Sure, you can post links in group chats or on social media now, but that collapses too many contexts. A dedicated feed of articles from a friend hits different.

Other RSS readers added social features after Google Reader shut down (like the old reader), but they all had the same problem: the social graph was locked into their platform. RSS itself is client-agnostic but article sharing was never part of the protocol, so it stayed locked in closed ecosystems.

The AT Protocol fixes this. AT Protocol is the underlying protocol for Bluesky and other decentralized apps. It's a kind of social filesystem that lets apps interoperate.

Skyreader writes the feeds you follow and the shares you make to the AT Protocol, so anyone could build a RSS reader that shows your shares. Your data is yours and the social layer is as portable as the feeds themselves.

I wrote Skyreader to match how I think about feeds. That might not work for you; the sheer number of different RSS readers suggests we all think about this differently. The neat thing about building on the AT protocol is we can all share the underlying data (feeds, shares) and move between readers as we like.

Take a look at Skyreader and let me know what you think. If you find any bugs let me know, but if you want it to work differently just make your own! RSS readers are simple, you can probably get Claude to knock one out in an afternoon (that's how long the initial Skyreader prototype took me). The code for Skyreader is available on Github if you want a starting point with the lexicon being the main thing you'll need to interoperate.

It's time to read like it's 2010!

Flint January Updates: Markdown Live Preview & Search

2026-01-18T00:00:00Z

Flint now supports live preview for markdown syntax!

This should make viewing your notes much nicer. I'd been reluctant to implement live preview before now because I figured it would be tricky to implement and also I like plain markdown just fine. But, my husband's first round of feedback when I made him try Flint was that raw markdown sucks (he said it gently of course lol). Getting it working was way easier than I'd made it out to be so I really should have gotten to it earlier. Ah well.

Relatedly, I've also added a bunch of UX affordances to make formatting easier and more discoverable.

Now when you select text a formatting window pops up so you can apply your bold formatting choices.

We also now have a "+" button in the gutter (alternatively toggled by typing /) that allows you to insert block level syntax like headers and quotes.

Search improvements #

Up until now search results have been limited to showing matched titles and small snippets of note content. Now Cmd+Shift+Enter (or Ctrl+Shift+Enter on Windows/Linux) opens an expanded search result screen that shows matches across all notes. Searches can also be saved in the sidebar too so you can refer back if needed.

So that's what's changed recently. Take a look and as always let me know what you think.

Stuff I Enjoyed in 2025 - Books

2026-01-16T00:00:00Z

This is part of my 2025 yearly roundup covering the books I read. tbh the middle of January is a bit late for an annual reflection but whatever, this is my blog and I can do what I want lol.

I didn't read as many books in 2025 as I would have liked so one of my 2026 goals is to fix that. Part of my issue is that I read multiple books at the same time. Not ideal so maybe in 2026 I can try and focus on one book at a time.

Books actually finished #

The Unaccountability Machine

The most interesting book I finished all year, will be thinking about this one for a while. It's about "the industrialisation of decision-making" and how accountability (or unaccountability as the case might be) now works (or doesn't).

we began to get into the habit of ignoring the fact that every year, more of the decisions that affect our lives are made not by people but by systems. Strange, alien intelligences with desires and drives quite different from our own. They’re taking over the world – and not only that, for it seems that some of them are going mad.

By "strange, alien intelligences" he mostly means a corporation and related kinds of decision-making organizations rather than AI but this line is becoming even more true as we restructure the world with LLMs.

Deep Response

My brother wrote a book! It's about figuring out how to deal with The Predicament. Sort of? Maybe?

Abundance

Interesting in the abstract but I wouldn't necessarily recommend it. A book for an alternate timeline.

More Everything Forever

Required reading for our current technological/political moment if you haven't been following the thinking of "those guys" (MacAskill, Altman, SBF, Kurzweil, Yudkowsky, etc.) for a while.

Termination Shock

A classic Neal Stephenson book for both good and bad.

Frankenstein

Picked it up after watching the Guillermo del Toro movie version. Thought I had read it before but apparently not. It's good!

Started but not finished #

Language Machines

Most interesting book I started but didn't finish. The basic claim is that structuralism has something interesting to say about LLMs.

The Notebook

Not the Nicholas Sparks novel. Instead a history of note-taking and actual notebooks. Obviously relevant to my interests with Flint.

The Tainted Cup

Just a really fun fantasy novel about detectives solving a murder in a empire that has to deal with the occasional leviathan invasion.

If Anyone Builds It, Everyone Dies

I don't agree with the conclusion or arguments but worth reading to get a sense of how the LessWrong crowd thinks (without wading through that site). Pair with More Everything Forever if you must read it.

New Year, New Flint

2026-01-03T00:00:00Z

Big changes with Flint! Over the holidays I completely reworked just about every part of Flint.

The UI has been totally refreshed, making every view cleaner and easier to navigate.

Under the hood, the way Flint stores data has also changed. Previously we used a local SQLite database with a two-way sync to markdown files. This worked well enough (though the sync code was gross and had unresolved bugs) but the architecture made it hard to implement features I really wanted like versioning and syncing.

So, the new Flint now uses Automerge to store notes, while still keeping the two-way markdown file syncing of the previous version. Automerge is a CRDT library that makes it possible to sync data across devices without needing to store all your data in a central server. The current rework doesn't including syncing yet but guess what feature I'll be working on soon 😆.

The other capability Automerge gives us is the ability to easily make a web app that stores all your note data locally in the browser. You can try it out here to see the new interface and features without installing anything. The only capabilities that aren't included at the moment are markdown file syncing, review mode, and the AI agent.

I think Flint is really starting to come together. More to come soon but take a look and let me know what you think!

Flint December Beta Updates (Open source and so many features)

2025-12-08T00:00:00Z

Progress on Flint has been going incredibly well! I'm recently back home from a great trip to Japan and with the release of Claude Opus 4.5 I've kinda been going beast mode on Flint development. Here's a roundup of all the features I've managed to ship in the past couple of weeks.

Flint Beta is Open (Source) #

First up though, the Flint beta is now open to anyone who wants to kick the tires. You can download it here.

Not only is it free to download but I've also committed to open source. Flint's code is now free and available on GitHub.

Core to my vision of Flint is that you own your thinking and for that to be true you must also have control over the means of (note) production. So open source it is.

On to the actual features.

Review Mode #

Review mode is a key piece of the "resurface" design pillar I've written about previously, which is all about helping you meaningfully re-engage with your notes over time.

The way it works is any note you click "Enable Review" on is added to your review queue in the new "Review" system view. When you start a review the agent reads your notes and generates a question to get you to meaningfully engage with the material.

You can then write a response and mark how well your engagement with the note went. Based on your response the note is scheduled for review sometime in the future.

I think this mode is a pretty big deal. Spaced repetition is a well-studied and powerful technique for learning but creating effective prompts has always been a challenge. I think integrating the prompt and scheduling interface into your note-taking system unlocks something special.

EPUB/PDF/Web Viewers with Highlighting #

To help create more material for you to review, you can now add EPUBs, PDFs, and web pages to Flint. Each file type has its own optimized viewer and each viewer has support for making highlights. And, of course, the agent can read your books too.

Decks (aka Queries/Databases/Bases) #

To help you keep track of all the notes you will now be creating I've also added Decks. Decks allow you to build dashboards of notes that match specific filters (a bit like Notion's Databases or Obsidian's Bases). Use it to track things like reading/watch lists, todo lists, etc.

For example, if you create a "movie" note type with properties like "rating" and "status" you can create a deck to track your favorites (type = movie AND rating >= 8) and another deck to track your watchlist (type = movie AND status != "seen"). No need to memorize the query syntax though, the UI lets you easily build complex filters and the agent is also very capable at building decks for you.

Each row in a deck is live—you can directly edit the title and properties to quickly keep your notes up to date.

Workspaces #

Workspaces are another feature to help you keep track of all your notes. They allow you to create focused groups of Pinned/Recent notes and quickly switch between them with keyboard shortcuts. Workspaces are great for creating dedicated groups of notes for managing different projects. As an example, I currently use the following spaces:

Default (unorganized, in-the-moment notes)
Flint (my notes about Flint development)
Writing (blog post drafts and supporting material)

Full Text Search #

The search bar now performs full text search across your notes:

Previously search only matched note titles. It now prioritizes note titles to make switching between notes easier but will also search note content.

Note Archiving #

To keep your vault clean you can now archive notes. Archived notes are not deleted but will no longer show up in search or link autocomplete lists. All existing links to archived notes will still work, you just can't make any edits until you unarchive.

Image Embedding #

You can now embed images in your notes:

Just drag and drop in image files from the finder/file explorer.

Agent Suggestions #

You can now have the agent generate suggestions for how to improve a note with comments along the side:

Suggestions are only generated on demand so you need to click the "Generate Suggestions" first to see them:

Next Up #

I've got big plans for more features as I work towards getting Flint ready for a 1.0 but the next few releases should be focused on polishing and responding to feedback. So try things out now and let me know how things are working (or not) for you.

We now have a bunch of places to drop feedback:

GitHub issues for reporting bugs and discussions for general questions or suggestions
Discord server
Email: tim@flintnote.com
Bluesky: @disnetdev.com

I'm excited to hear what you think!

How Flint Helps You Externalize

2025-11-30T00:00:00Z

As I wrote about in a previous post, a core design goal for Flint is to make externalization, the process of getting ideas out of your head, super simple. Easy note creation results in more notes and this is critical because it supports the other phases of deep knowledge (internalizing and resurfacing) by creating the raw material for you to work with (pure thoughts are hard to manipulate in an app!). Flint tries to accomplish this goal by minimizing the decision of where to start capturing ideas and deferring organization decisions that kill note-taking momentum.

So how does this work in practice?

It starts with always having a zero-decision place to begin writing: the Daily view. When you open up Flint with a brilliant idea to capture, just open today's daily note (Cmd-2/Ctrl-2) and start typing.

The daily view displays all your other entries for the week at a glance, making it easy to reflect on what you've been thinking about recently while you write about today.

While capturing in the daily note, you'll start to identify concepts that are more concrete and deserves their own note. When that happens you can create a new note via the [[title of note]] link syntax while you're already typing or with the new note button (shortcut Cmd-Shift-N/Ctrl-Shift-N).

Flint doesn't force you to think about how to organize new notes up front. The note is just created immediately and you can keep typing. You can decide what kind of note it should be or what metadata it needs later. You don't even need to pick a title; write first and organize later.

When you make a new note two other things happen: the note shows up in the "Recents" list on the left sidebar and it's added to the Inbox (notice the badge on Inbox).

The Recents and Pinned section of the sidebar are how you manage your open notes. Notes are added to the Recents list when you click on a link or create a new note. The Pinned list is for keeping track of notes you are working on. These lists are manually sorted (via drag and drop), which engages your spatial memory to help you keep track of where things are.

This design pattern isn't new, it draws inspiration from how Arc Browser manages tabs. The fundamental design concept is managing different time horizons for ephemeral items (recents) and items you keep for a longer period of time (pinned).

The Inbox is another way to help you manage notes across time. Each newly created note goes into the Inbox and has a button to mark it as "Processed", which removes it from the list.

The Inbox allows you to create a bunch of notes without interrupting your flow to organize immediately. Once you've done your initial externalization you can find all the notes you just created in the Inbox and then decide what to do with them later. The Inbox gives you a little routine to make small organizational decisions and prevents notes from falling through the cracks. It gives you confidence to make lots of notes and know that you'll get back to them when you are ready.

What I'm trying to accomplish with the design of Flint is the feeling of being confident to externalize. Writing things down should always feel effortless. From knowing where to start writing to feeling confident you'll be able to organize things later, nothing should slow you down.

If any of this sounds interesting definitely check out Flint, the beta is now open and you can download it here.

What's the point of a note-taking app?

2025-11-14T00:00:00Z

Note-taking apps are a bit weird. Most productivity apps you use every day have an obvious and tangible output that you spend all your time working towards. For example, the point of using Photoshop is to make an image. You spend all your time futzing with an image and eventually you get an image to use somewhere.

But the notes in a note-taking app are not really an output you directly use. Some people and systems approach them as if they were the goal but I think this is a mistake and explains why people often have bad experiences with note-taking apps.

So if not notes then what are we actually trying to achieve with a note-taking app? In a single phrase: deep learning (for humans, not the ML kind). The output we are looking for is the mental change that occurs via the process of note-taking that deepens our understanding about the things we care about: a topic of interest, ourselves, the world around us. By spending time taking notes we become smarter; our crystallized intelligence grows.

This means that a note-taking app is a cognitive tool, not a document manager. Most apps are built as the latter, which I think is why many people struggle with them.

So if the primary goal of a note-taking app is to deepen our understanding of the things we care about, how do we go about designing for that? Well, first we need to understand how learning works, which I think can be broken down into three main components:

Externalizing: Getting ideas out of your head and into a trusted system
Internalizing: Synthesizing what you've externalized into new insights and understanding
Resurfacing: Reinforcing your understanding through spaced retrieval and repeated engagement over time

These three processes feed on each other. Externalizing produces the raw material to internalize which sparks new ideas to be externalized while resurfacing what you've externalized and synthesized deepens your understanding and sparks new insights. Without all three you can get stuck. If you don't externalize, ideas are stuck in your head. If you only externalize, you're left with a bunch of notes that serve no one. If you don't resurface and review what you've externalized, your understanding is shallow and quickly fades.

Deep learning requires all three components working together in a virtuous cycle.

Where things go wrong for most note-taking systems is too much friction with one or more of these components. Friction during externalization keeps thoughts locked in your head. This might be due to an organization system that is too complex (too many options for creating a new note) or too simple (you don't trust the organizational system to find the note again). If you can't easily find and connect old notes, you miss the synthesis that deepens understanding. If the system doesn't support resurfacing those notes over time, your knowledge fades away. Designing a note-taking app around deep learning means minimizing friction at each of these stages.

While the goal of a good note-taking app should be to minimize the friction of the note-taking process, you can never completely eliminate it because some friction is learning. Having an LLM write your notes for example would be counterproductive, you skip the necessary friction that causes learning to occur. Actual learning requires the friction of articulating vague thoughts into precise language, the friction of recognizing connections between notes, the friction of intentional recall that reinforces and strengthens your memory.

A well-designed note-taking system minimizes accidental friction that gets in the way while preserving the productive essential friction that results in deep learning. It should feel natural and freeing to quickly capture thoughts as they come, knowing the system will help you find, organize, and revisit them later. It should resurface old insights to strengthen your memory and reveal new connections. It should amplify rather than replace your thinking.

This is the kind of system I'm trying to build with Flint, which is designed around the idea that note-taking is about deep learning and tries to minimize the accidental friction you encounter by:

using higher level abstractions in the UI than files and folders
making linking effortless
using Arc-style tabs that help you organize notes with your spatial memory
using a LLM agent to help with organization
using a LLM agent to resurface old notes to cement your memory

Over the next few weeks, I'll dive deeper into how Flint's design reduces friction and contributes to each of the components that enables that deep learning cycle in practice. But the core idea is simple: a good note-taking app should get out of the way while keeping you engaged with the friction that matters.

Introducing Flint

2025-10-10T00:00:00Z

Flint is a new note-taking/thinking-system I’m working on that fixes a lot of the frustrations I’ve had with traditional note-taking apps (or at lest it will try to fix those frustrations 😅)

I’ve long been obsessed with note-taking systems in both paper and digital forms because I love the idea of a system that helps me think better.

At the start of this summer (after freeing up a bunch of my time) I started experimenting with how to best improve my personal note taking system using LLMs. One of the outcomes of those experiments was flint-note-mcp an MCP server that gave LLMs the ability to write and manage a directory of structured notes. It was an attempt to see just how far you could push note taking away from a UI. Just chat with any LLM and have the AI write your notes.

The MCP server was an interesting experiment but ultimately it made me realize two things. First, LLMs are magic and have serious potential but also they have a tendency to take over your thinking, which is the opposite of what you want in a good note-taking system. Second, I really needed a good UI.

So, I started working on Flint, an attempt to make a good UI that uses LLMs in a way that won’t harm your thinking.

I’ll write a bunch more about all of this but the main philosophy behind Flint is this:

You own your notes. Flint is plaintext-first. No lock-in and no complex formats that make migrating to a different system a pain. Flint just makes normal markdown files on your hard drive. Eventually I might add an optional sync system but your notes will always start as just normal text files on your computer that you control.
A UI that is humane. The UI should surface appropriate abstractions that respects the way you think. This means you mange notes rather than files and folders. The UI should give you a workspace where you can make a mess and also give you tools to clean it up.
Writing is thinking and you do the thinking. LLMs are incredible bits of magic but they can easily damage your thinking by automating your writing. Flint will incorporate LLMs and AI agents in a way that allows you to be ambitious with your thinking system without replacing the writing and thinking work that only you can do.

I’ve got a bunch more thinking, writing, and building to do but if this sounds interesting to you, go subscribe to the Flint newsletter where I’ll send out links to the beta when it's ready. If you have any questions just shoot me an email at tim@flintnote.com.

GPT-5 Thoughts

2025-08-07T00:00:00Z

I watched the OpenAI livestream introducing GPT-5 today and had some thoughts.

First, the presentations were all oddly stilted and awkward. Most of the presenters seemed to be mimicking Sam Altman's affect. It was weird, like I'm not expecting them to be as polished as an Apple keynote (even then Apple has a spectrum of polish) but like maybe they could have done a few more run-throughs?

Second, they had blindingly obvious errors in some of their graphs. They had a bar chart where 52 was taller than 69, which was the same height as 30. Clear case of automation bias since I'm sure they had GPT-5 check for errors and it didn't catch it so the humans didn't either.

Third, one of the demos they had was of GPT-5 explaining the Bernoulli principle in relation to how flight works. It made a web visualization showing how lift is created based on the principle and how air travels over a wing. The wild thing is that the Wikipedia page on the Bernoulli principle calls out how this is a common misconception. Now, maybe the details of the demo were actually correct but it was hard to tell from just watching (Hacker News had a good time nerdsnipping themselves over it). Strange that they went with this example because it's so fraught with misconceptions and easy to get (subtly) wrong.

Also, not a huge point, but they kept saying how GPT-5 was like having a PhD in your pocket. As a PhD haver who has worked with plenty of other PhD havers, this is just so funny. Like I get what they are trying to communicate (it's so smart!) but it's not nearly as impressive as it sounds (in the same way that Bruce Banner having 7 PhDs is supposed to read as smart but is completely absurd). People really need to understand that a PhD does not make you smart.

Building a Personal GTD System in Plain Text with AI Agents

2025-07-02T00:00:00Z

I've been experimenting with trying to build a plain-text GTD system with Flint. I started by just creating a todo note type with a couple of simple metadata fields (e.g. status, due date, etc.) with agent instructions to keep things concise and action-oriented.

Creating and updating todos then happens conversationally:

Me: make a todo to schedule a dentist appointment today

Agent: Done.

Later in the day:

Me: mark dentist appointment done

Agent: Updated.

My favorite part of this approach is you can then ask the agent "what should I be working on today" and it's generally smart enough to look through all your todos but also other kinds of notes (daily logs, project notes, etc.) and give you a pretty intelligent response.

This is fine but gets awkward to update multiple items at a time. What I wanted was a way of seeing and updating multiple todos at a time while still keeping the benefits of text and agent smarts. So I just asked Flint to make me a special note that it would keep in sync:

Me: create a todo note that is special called "today", it should:

list all the other todo items that should be worked on today

format items like - [ ] todo item.

update agent instructions to update this note when updating other notes

when the note has items checked off, synchronize with corresponding note status

have a section called "Quick Add New Tasks" to easily add new items

each todo should be in one of three states: now (I want to focus on working on it today), upcoming (ready to be worked on soon), someday (a backlog of stuff to plan out eventually)

the today note should group by status but not show all the someday tasks (just show a count)

items in the now section should be written like - [ ] + todo item and items in the upcoming sections are written like - [ ] todo item.

when syncing the today note look for changes to the + to move items from the now state to the upcoming or vice versa

Flint then made a "todo" note that looks like this:

# Today - [Current Date]

## NOW (Focus on today) 🎯
- [ ] + Finish quarterly report draft
- [ ] + Review team feedback on project proposal
- [ ] + Schedule dentist appointment
- [ ] + Follow up on client contract (OVERDUE)

## UPCOMING (Ready to work on soon) 📋
- [ ] Plan next week's team meeting (Due: Friday)
- [ ] Update budget spreadsheet (Due: End of month)
- [ ] Research new project management tools

## Quick Add New Tasks
*Add new todo items below - use + for NOW, - for SOMEDAY, no symbol for UPCOMING*
- [ ]
- [ ]
- [ ]

Now I can easily check off (e.g. change [ ] to [x]) multiple items and then ask Flint to sync the note and it will update each todo note as completed and refresh the today note.

A day's interaction could look something like this:

Morning (9:00 AM):

User opens Today note, sees:
- [ ] + Write project proposal
- [ ] + Call insurance company
- [ ] Review budget numbers (Due: Thursday)

User adds + symbol to "Review budget numbers":
- [ ] + Review budget numbers (Due: Thursday)

System automatically:
- Updates individual todo note workflow_state to "now"
- Moves task to NOW section in Today note

Midday (1:00 PM):

User checks off completed task in Today note:
- [x] + Write project proposal ✅

System automatically:
- Updates individual todo note status to "completed"
- Adds completion timestamp
- Moves to "Recently Completed" section

Evening (6:00 PM):

User adds new task to Quick Add section:
- [ ] + Book flight for conference

System automatically:
- Creates new individual todo note: "Book flight for conference"
- Sets workflow_state to "now" (because of + symbol)
- Adds to NOW section in Today note
- Clears the Quick Add entry

I think this is pretty cool, you can build an approximation of a full-fledged GTD app like Things or OmniFocus in plain text just by writing down a few simple instructions.

Another neat thing is I built this workflow conversationally. My example above defined the today note in one instruction but I actually went back and forth with the agent a bunch to narrow down how the today note should best be structured.

This bidirectional sync between plain text and AI agents feels like a fundamental building block. You get the benefits of structured data and automation while keeping the flexibility and ownership of plain text. I suspect this pattern will be useful beyond task management.

The Note-Taking Trap - Why AI Could Save (or Destroy) How We Think

2025-06-30T00:00:00Z

Update, October 2025: This post introduced an initial experiment with using MCP for note-taking that has morphed into something bigger. Check out this post for more details.

I'm obsessed with note-taking systems. I've been obsessed with them for most of my computer-using life. Evernote, Notion, Roam, Obsidian—each promised to be my "external brain". I spent hours building out workflows to help me think and act on the things that mattered to me. But inevitably I would hit the same wall when my system became unmanageable and I had to start over from scratch.

Here's the problem: your thinking changes. Your projects shift. Your interests expand. But most note-taking systems lock you into structures that become prisons over time. Maybe you organize by project phases—"Research", "Planning", "Execution", "Review". Then you start working on iterative projects that don't follow linear phases, and your structure crumbles.

You either stick with categories that no longer fit, or you abandon the system entirely and start over.

Now AI promises a solution: just dump everything into a smart assistant and let it handle the thinking for you. Sounds perfect, right?

Maybe, but I think that approach is dangerous.

When we fully outsource our thinking to AI, our thinking atrophies. Think about the Google Maps effect: before GPS, we mentally mapped neighborhoods, remembered landmarks, and built spatial reasoning. After GPS, we can't navigate familiar areas without turn-by-turn directions. The same thing happens with AI-assisted thinking.

The messy work of organizing ideas, making connections, and structuring knowledge isn't busy work—it's how we actually think and learn. Automate that away, and you're automating away your own intelligence.

Plus, you're handing over your most valuable asset—your accumulated knowledge and thinking—to systems you don't control.

So here's what I want: an AI thinking system that I control and that augments my thinking instead of replacing it.

Instead of throwing everything into an AI void, imagine an agent that works with you based on your explicit preferences for how you want to think and organize. You tell the system what kinds of notes you work with—daily journals, project logs, book summaries, meeting notes, research captures—and what workflows matter to you: weekly reviews, concept mapping, progressive summarization, or building arguments over time.

For example, you specify that you keep "insight notes" and "reference notes" separate, and prefer connecting ideas through explicit linking rather than tagging. The AI then notices when you write about "flow state" in an insight note, remembers you mentioned "deep work" in a book reference from 6 months ago and "maker's schedule" in a meeting capture, then suggests creating a synthesis document that fits your preferred workflow—maybe a new insight note that builds on these connections, or links to add to your existing concept map.

Or when your project logs increasingly reference the same underlying problems across different domains, the AI suggests a reorganization that aligns with how you actually think—perhaps shifting from department-based folders to problem-solution frameworks, but only because you've indicated you prefer outcome-oriented organization over categorical filing.

The AI handles the friction—the tedious reorganization, automatically tagging notes with concepts and people mentioned, the structural updates—while you focus on the actual thinking. The goal isn't to eliminate the work of thinking. It's to eliminate the work that gets in the way of thinking.

This vision sounds great in theory, but how do you actually build it? The traditional approach would be to create another proprietary platform, which is exactly the kind of lock-in I want to avoid. Instead, we need a way for AI models to work with your note-taking system while keeping your notes completely under your control.

MCP Is Neat #

This is where I think Model Context Protocol (MCP) unlocks something special. Unlike proprietary systems that lock you into a platform, MCP unlocks the potential for any AI model to work with your data while you stay in control.

Consider the difference: In traditional systems, your notes are locked in Notion's proprietary format. Want to switch to Obsidian? Technically there is an export but the export breaks formatting and loses connections. Company gets acquired? Your thinking system disappears.

With an MCP-enabled system, your notes can just be files in a folder. The AI agent reads and writes through the MCP protocol, but you can switch AI models, sync services, or editing tools without losing anything. Your 10-year+ knowledge base outlasts any single company or platform.

But unlike traditional file-based systems like Obsidian where you're on your own for organization, the AI agent becomes your active partner in managing, connecting, and evolving this information.

This separation of concerns (intelligence versus data storage) enables something that wasn't possible before: true agent-first design where the AI can be deeply integrated with your thinking process and you control your data.

This isn't some distant future vision. It's what I'm building right now with Flint Note.

How Flint Works #

Flint is an MCP server that organizes your notes as a vault of ordinary markdown files with semantic note types. Each note type defines its purpose, custom agent instructions, and metadata schema. Let me show you how this works in practice by walking through setting up a vault with three common note types.

Setting Up Your Vault #

A vault starts as just a directory of markdown and YAML config files:

my-notes/
├── .flint-note/
│   └── config.yml
├── daily/
│   └── _description.md
├── todos/
│   └── _description.md
└── reading/
    └── _description.md

The magic happens when you define what each note type means and how the AI should behave when working with it.

Creating Note Types Through Conversation #

Let's start by defining our note types:

You: lets create three note types: daily, todo, and reading

Agent: I'll create all three note types for you: daily, todo, and reading. Let me set each one up with thoughtful agent instructions and useful metadata schemas.

[Creates daily/_description.md, todo/_description.md, reading/_description.md]

Perfect! I've created all three note types for your vault:

📅 Daily (Journal & Reflection)

Purpose: Daily journal entries for reflection, planning, and personal growth

Features: Mood tracking, energy levels, gratitude lists, key insights, and tomorrow's focus

Agent Behavior: Structures entries with morning intentions, daily events, and evening reflection while identifying patterns and celebrating progress

✅ Todo (Task Management)

Purpose: Task management with priority, deadlines, and progress monitoring

Features: Priority levels, categories, completion tracking, due dates, and project connections

Agent Behavior: Formats tasks as checkboxes, organizes by priority, identifies dependencies, and suggests productivity improvements

📚 Reading (Books & Articles)

Purpose: Reading notes for books, articles, and other materials with insights and connections

Features: Author tracking, reading status, ratings, key takeaways, and topic organization

Agent Behavior: Structures notes with summaries and key ideas, identifies connections to other materials, and encourages critical thinking

Your vault is now ready for comprehensive knowledge management! You can start creating notes of any of these types. Just say something like:

"Create today's daily note"

"Add a todo list for this week's tasks"

"I want to create a reading note for [book/article name]"

What would you like to create first?

The resulting daily/_description.md looks like this:

# Daily

## Purpose

Daily journal entries for reflection, planning, and personal
growth tracking

## Agent Instructions

- Structure daily notes with clear sections: Morning
  intentions, daily events, evening reflection
- Extract and highlight key insights, lessons learned,
  and personal growth moments
- Identify patterns in mood, energy, productivity,
  and relationships
- Suggest connections to previous daily entries when
  similar themes emerge
- Encourage reflection questions: What went well? What
  could be improved? What am I grateful for?
- Track recurring themes and suggest deeper exploration
  through follow-up questions
- Notice and celebrate small wins and progress toward
  personal goals
- When appropriate, suggest creating separate notes for
  significant insights or ideas that emerge

## Metadata Schema

Expected frontmatter or metadata fields for this note type:

- `mood`: Overall mood for the day (optional, select,
  options: [`energized`, `content`, `neutral`, `tired`,
  `stressed`, `excited`, `reflective`, `grateful`])
- `energy_level`: Energy level (1-10 scale) (optional,
  number, min: 1, max: 10)
- `gratitude`: Things you're grateful for today (optional, array)
- `key_events`: Significant events or moments from the
  day (optional, array)
- `insights`: Key insights or lessons learned (optional, array)
- `tomorrow_focus`: Main focus or intention for
   tomorrow (optional)

Since the description is just a plain text markdown file, you can have the agent update it or edit it yourself with a text editor.

Adding Notes Through Natural Conversation

Now let's see the system in action:

You: Add today's daily note

Agent: [Creates daily/2025-06-27.md with structured content]

Perfect! I've created your daily note for today, June 27, 2025. Following the agent instructions for daily notes, I've structured it with clear sections to guide your reflection:

Your Daily Note Structure #

Morning Intentions (Space to set your focus for the day)

Daily Events (Capture key moments, interactions, and accomplishments)

Evening Reflection (Process how the day went with guided questions)

Gratitude Section (Practice appreciation)

Key Insights & Lessons (Extract learning and growth moments)

Tomorrow's Focus (Set intention for the next day)

The note is ready for you to fill in! As you use it, I'll help you:

Identify patterns in your mood, energy, and productivity

Connect themes to previous daily entries

Celebrate your small wins and progress

Extract insights that might be worth exploring in separate notes

You can also add metadata like your mood, energy level (1-10), and specific gratitude items if you'd like more structured tracking. How would you like to start filling it in?

You: My energy is about a 7, pretty good energy

Agent: Great! I've updated your daily note to capture that energy level of 7. That's solid energy for the day!

With that good energy level, it sounds like you're having a productive day. What's been contributing to that energy? Any particular events, accomplishments, or moments that have stood out so far today?

I can help you capture those in the daily events section, or if you're reflecting on the day, we could work on the evening reflection part. What would you like to add to your daily note?

Adding a reading note:

You: Add a reading note for "The Unaccountability Machine" by Dan Davies. I've finished it, here's a great quote:

"An organisation does things, and it systematically does some things rather than others. But that's as far as it goes. Systems don't make mistakes – if they do something, that's their purpose."

Agent: Excellent! I've added that powerful quote and updated your reading note to show you've completed the book. That's a really striking insight – the idea that "systems don't make mistakes, if they do something, that's their purpose" cuts right to the heart of how we think about organizational accountability.

Since you've finished the book, I'd love to help you capture more of your insights:

What was the main thesis that resonated most with you?

How does this quote connect to the broader argument about unaccountability?

Are there specific examples from the book that illustrate this "systems don't make mistakes" principle?

Does this change how you think about organizations you've worked with or observed?

This seems like the kind of concept that could warrant its own separate note if it becomes a recurring theme in your thinking about systems and organizations. What other key takeaways stood out to you from the book?

[Creates reading/the-unaccountability-machine.md with the quote and reading status]

Evolving Your System

The real power emerges when you want to adapt how the system works:

You: Update my daily notes to also ask about tomorrow's top priority

Agent: I'll modify your daily note instructions to include planning tomorrow's main focus.

[Updates daily/_description.md to add tomorrow planning prompt]

Updated! Now when you create daily notes, I'll also ask: "What's your top priority for tomorrow?" This will help bridge your reflection with forward planning.

You: Actually, make it three priorities, not just one

Agent: Good adjustment! Three priorities gives you more flexibility while still maintaining focus.

[Updates the instruction again]

Done! I'll now ask for your top three priorities for tomorrow in each daily note.

The Result: Intelligent, Adaptive Organization

What you end up with is a system that understands the different contexts of your thinking:

Daily notes become reflective conversations that build self-awareness over time
Todos match tasks to your energy and connect to bigger projects
Reading notes capture insights with precision and link to your active work

But unlike rigid templates, every aspect can evolve through natural conversation. The AI learns your preferences and adapts its behavior, while your data remains simple markdown files you can edit anywhere.

The system grows with you, becoming more intelligent about your specific needs while keeping your information completely portable and under your control.

Why This Matters

We're at an inflection point where AI can finally deliver on the vision of an external brain that truly enhances thinking. But only if we're willing to rethink our assumptions about how knowledge systems should work.

Most AI-enhanced note apps are still designed for the old world where humans do the organizing and AI occasionally helps out. Flint flips that: the AI is your thinking partner from day one, and the system is designed to make that partnership as natural and powerful and flexible as possible.

Since all your notes are just markdown files, there's no lock-in. You can work on your notes with Flint, or fire up vim or Obsidian or iA Writer or any other local markdown editor you want. You can sync or backup with whatever file syncing service you already use (git, Dropbox, iCloud, Obsidian Sync, etc.).

But unlike other markdown-based systems, Flint is architected from the ground up to be agent-first. Every design decision (from the vault structure to the metadata schemas to the conversation patterns) optimizes for AI collaboration rather than human maintenance.

Getting Started

If this sounds interesting, you can try Flint by adding it to your MCP client of choice. You'll need Node.js installed first:

{
  "mcpServers": {
    "flint-note": {
      "command": "npx",
      "args": ["@flint-note/server@latest"]
    }
  }
}

I've also created system prompts to help guide your model's interactions: system_core.md for intelligent models like Claude 4 or ChatGPT 4, and simple_models_detailed.md for simpler or local models.

The project is still early, but if you try it out I'd love to hear what you think. The system is designed to evolve based on how people actually want to interact with their knowledge.

Check out the GitHub project to dive deeper, or just give it a try. The future of knowledge work is agent-first, and Flint is my attempt to build part of that future today.

MCP is neat

2025-06-16T00:00:00Z

Jotting down an aha! moment I had about how MCP can augment the utility of LLMs in some of the things I do every day.

The background is I've been trying to be more intentional about committing things to memory using Anki and spaced repetition. I was inspired years ago by Michael Nielsen's article about using SRS to read and remember ideas from papers so that's my main goal: commit interesting ideas from papers to memory.

When I first read that article I tried to build an Anki routine but fell off—the activation energy required to add cards was always too high (also writing good prompts is hard).

LLMs are really good at summarizing and restructuring text so I wondered if they could help with writing Anki cards. I'm not sure if this is the right approach long-term since writing cards by hand would almost certainly aid in the learning process. That said, it turns out I couldn't sustain the manual effort so I think it's worth experimenting to see if a different approach could help.

So my approach is to use an LLM to help write the cards. MCP then lets me easily connect the LLM directly to Anki and even further reduce the activation energy required for authoring cards.

I'm using Raycast, but the basic idea works in any MCP-capable LLM client (e.g. Claude desktop). The MCP server I'm using is anki-mcp-server, which requires Anki and the AnkiConnect plugin. Wiring up the Anki MCP server to Raycast is super straightforward and took me all of five minutes.

For articles, there are a few kinds of prompts that seem to work. If the article is well-known and in the model's training set, you can simply say:

suggest some anki cards for the article "As We May Think" by Vannevar Bush

Claude 4 responds with something like:

Basic Information Cards

Card 1:

• Front: Who wrote "As We May Think" and when was it published?

• Back: Vannevar Bush, published in The Atlantic Monthly in July 1945

Core Concepts

Card 2:

• Front: What is the "Memex" as described by Vannevar Bush?

• Back: A hypothetical electromechanical device that would store books, records, and communications, allowing users to retrieve and link information through associative trails

...

For newer or less well-known articles, you can paste the entire article or a URL (if the model supports web browsing). Not every suggested card will be useful, sometimes the model suggests trivia I don't care about, but it's easy to go back and forth with the model to pick the best ones. You can also add some pre-prompt instructions to guide the model toward the kinds of cards that are most helpful, but honestly, it's quick enough to just pick out the good ones.

Once I have the cards I like, it’s a simple instruction to add them to Anki:

now add those cards to @anki

(Raycast uses the @ notation to add an MCP server to a chat, but you can also provide it as a chat preset.)

Raycast also provides a nice feature to wrap commonly used prompts in AI commands. For example, I’ve added an “ankify” command with a prompt like:

from the article in {browser-tab}, suggest some anki cards to help me remember the salient points. when referencing the article use a short version of its title and link to it

This does what it sounds like: it provides the currently focused browser tab as context to the model. I can then check the suggested cards and follow up by telling it to add them to my Anki deck.

I think it's pretty neat that thanks to MCP we can now wire up these powerful capabilities to our LLMs in a matter of minutes.

New Things

2025-06-03T00:00:00Z

Big changes happening in my life. After almost a decade, today was my last day working at Shape Security / F5.

It was a really good run. I had the chance to work with incredible folks building wild things on the web and I think we make it a little safer and more convenient for everyone.

Now it's time for something completely different. Not entirely sure what that is yet but I'll figure out something fun soon.

Large AI Models Are Cultural and Social Technologies

2025-03-19T00:00:00Z

Large AI Models Are Cultural and Social Technologies. The way LLMs are often framed is as "intelligent agents" but maybe this is the wrong framing. Rather we should view them as "cultural and social technologies" (like the printing press or markets) that are "allowing humans to take advantage of information other humans have accumulated".

Not as snappy or exciting in a science fiction sort of way but more accurate without being dismissive of their impact on society.

Zed Edit Prediction Feature

2025-02-13T00:00:00Z

Zed now predicts your next edit with Zeta, our new open model. Played around with Zed's edit prediction feature today and it worked surprisly well.

I like this feature because it sits on the spectrum betwen "just chat with Claude" on the one hand and "describe what Cursor should do" on the other. It augments your process rather than trying to do it entirely.

Hypermedia Controls - From Feral to Formal

2025-01-25T00:00:00Z

Hypermedia Controls: From Feral to Formal. An interesting paper that tries to locate and formalize a set of core primitives in hypermedia systems as expressed in HTMX. It identifies a "hypermedia control" as consisting of four mechanisms: (1) an element that (2) responds to an event trigger by (3) sending a network request and (4) placing the response in at some position in the viewport. By enhancing a hypermedia system with primitives that allow you to manipulate each of those mechanisms you can declaratively extend the system with your own hypermedia controls.

An example they give:

<button hx-trigger="click" hx-post="/clicked" hx-target="#output">
  Issue a request
</button>
<output id="output"> </output>

When the user clicks on the button the system will issue a network request to /clicked and place the response in the <output id="output"> element.

This is interesting in so far as it goes but I'm not convinced that the "hypermedia maximalist" approach is really all that great of a way to develop systems.

We're Getting the Social Media Crisis Wrong

2025-01-16T00:00:00Z

We're Getting the Social Media Crisis Wrong. It's not about misinformation, it's about groups with collective misunderstandings:

The fundamental problem, as I see it, is not that social media misinforms individuals about what is true or untrue but that it creates publics with malformed collective understandings.

I really like this subtle shift in perspective. It aligns with my distrust of social media as distorting people's behavior around chasing engagement but he actually identifies a more specific issue:

The more important change is to our beliefs about what other people think

Our beliefs and opinions about the world are influenced by what we think other people think and social media is a (distorted) machine that tells us what other people think.

Decentralized Systems Aren't

2025-01-11T00:00:00Z

Decentralized Systems Aren't. Centralized systems will always layer on top of decentralized systems unless you figure out how to fix the underlying economic problem of increasing returns to scale.

To actually get a permissionless decentralized system you need:

a business model that has decreasing returns to scale
a way to prevent Sybil attacks without massive cost
a way to prevent collusion between independent nodes

Never Forgive Them

2025-01-07T00:00:00Z

Never Forgive Them:

I will never forgive these people for what they’ve done to the computer, and the more I learn about both their intentions and actions the more certain I am that they are unrepentant and that their greed will never be sated. I have watched them take the things that made me human — social networking, digital communities, apps, and the other connecting fabric of our digital lives — and turned them into devices of torture, profitable mechanisms of abuse, and find it disgusting how many reporters seem to believe it's their responsibility to thank them and explain why it's good this is happening to their readers.

Last year Ed Zitron became my favorite critic of the tech industry's rot. He's incisive and angry and it's cathartic to read him.

Some Games I Enjoyed In 2024

2025-01-01T00:00:00Z

Palworld #

Pokemon but with guns.

Was briefly very into this game. Wrote a bit about it at the time.

Quickly dropped off because it was very early access and those kind of engagement hooks have a very short half-life on me. Might circle back at some point but probably not.

Final Fantasy XIV #

WoW but make it good.

I finished the story content up through Endwalker this year and very much enjoyed it. I'd been working through the main story of FF14 off and on since 2021 (got interested in the game off the hype for the initial Endwalker release).

As much as I enjoyed my time I think I'm done with FF14. The story could be incredibly affecting but working through the MSQ too often felt like a slog.

Persona 5 Royal #

Persona but make it the fifth one.

I've been playing this one off and on for years. Unless it's an automation game I find it difficult to really stick with a game beyond twenty hours or so and Persona 5 Royal is much longer than that. Finally pushed through to the end and really glad I did.

Destiny 2: The Final Shape #

Destiny but make it good finally (finally 🪦).

I've been a casual Destiny enjoyer off and on ever since D1 came out in 2014 so it was nice to see everything really come together after a decade. I dropped off hard after finishing the campaign though and don't expect I'll be coming back.

Elden Ring: Shadow of the Erdtree #

Elden Ring but even more.

Consort Radahn was BS. 10/10

Metaphor ReFantazio #

Persona but make it fantasy and extremely relevant to our current moment.

Only half way through but having a great time. Has been more consistently compelling to me than Persona 5 (which took forever to finish). Better story pacing, refined job system, and unique world building helps a lot. Also doesn't hurt that I "get" persona-type games better now after having played a few.

Factorio: Space Age #

Factorio but make it space.

Automation games in general and factorio in particular are extremely my jam. I'll be playing this one for a while. I'm taking my time, sixty hours in and I've only managed to get to one of the new planets so far (Vulcanus).

UFO 50 #

Retro games but make them modern and alt-history and 50 of them.

I was surprised how much I enjoyed this since I'm not usually much of a retro gamer. My nostalgia goggles aren't corrective enough to look past the frustrating game designs of the past.

But modern game designers interpreting the games of old through the conceit of alt-history works really well. While a few of the games in the collection are intentionally frustrating in the style of actual 80s games, most are a clever injection of modern sensibilities under 8-bit constraints. Party House, Pilot Quest, Campanella 2, Rail Heist, and Avianos are particular standouts for me.

Don't Get Burned

2024-12-31T00:00:00Z

How I’m Trying to Use BlueSky Without Getting Burned Again. I really like the principle of assuming the platform/service you are using will go away in three years. That's a long enough timeline to derive value without overcommitting.

Some of the Apps I used in 2024

2024-12-31T00:00:00Z

Browser: Arc

Still a transformative browser UX but I'm worried about its future considering their recent pivot to a new "AI first" browser. We'll see how things go in 2025.

Notes: Obsidian

Continues to run my life.

Read it Later: Readwise Reader

Still the best thing ever. I have highlights sync to Obsidian, which is the best thing ever.

Task Management: OmniFocus

I tried to switch to Things last year but it didn't stick. Back to OmniFocus and it's great although the v4 update was a bit rough but it's mostly evened out now.

Code Editor: Zed

Zed is incredible. Fast, great VIM mode, AI integration is great (though I need to check out Cursor at some point too).

AI Assistant: Claude

This was the year I started to actually get productive with AI assistants. They aren’t always good but when they are they really are. I mostly use Claude at this point though I don’t really do extensive comparisons.

Launcher: Raycast

A bunch of my one-off AI assistants use is actually via Raycast. Super helpful to have it always a keystroke away.

Calendar: Fantastical

Continues to be great.

Podcasts: Overcast

Continues to be great

Email: Mail.app

Boring but I’ve come around on that being what I like out of email.

Finance: YNAB

Continues to be great.

AI Scaling Myths

2024-12-29T00:00:00Z

AI Scaling Myths. As 2024 went on, more and more AI discourse started to come around to the idea that AI scaling was probably about to end. One things I learned from this article is that "scaling laws" have a more precise meaning than what I'd known.

I had a general understanding of "scaling laws" to mean that the bigger the model the more "capabilities" were supposed to pop out. But the original meaning in AI research is about the "decrease in perplexity" (perplexity being jargon I also learned from this article meaning a measure of word prediction uncertainty). A decrease in perplexity is not quite the same thing as new capabilities.

Division of Labor: How Palworld's Mechanics Hooked Me

2024-01-28T00:00:00Z

I've been playing a bunch of Palworld recently (along with everyone else apparently) and been having a great time. It's open-world survival crafting (think Valheim) mixed with monster collecting (Pokémon) mixed with some very light (at least in the first ten or so hours, might get more complicated later) automation (a little bit of Satisfactory).

I'm a little surprised how much it has hooked me since I've always bounced off the survival crafting and monster collecting genres. I understand their appeal but the tedium inherent in both have been too much for me (also the combat in Pokémon just straight up sucks, don't @ me). That said, I've sunk hundreds of hours into automation games (every few months I circle back to either Satisfactory, Factorio, or Dyson Sphere Program), and those games definitely have plenty of tedium baked into their design.

So what makes Palworld click for me?

I think the key is the way that Palworld uses monster collecting to relieve the survival crafting tedium. The pals you collect are not just used in combat, you put them to work in your base to gather resources and craft items. The tedium is a motivator to find new and better Pals. There is a virtuous cycle at work here; you collect Pals to build a better base and craft better items that you use to find better Pals that you use to build a better base and so on.

This ever-building cycle is exactly the hook that pulls me so deep into automation games, the initial tedium is a motivator to build more complicated factories that you use to build yet more complicated factories and so on.

I'm only ten or so hours into Palworld so I'm curious to see if its hook can sustain itself to an endgame. I suspect not, automation games have another endgame hook that Palworld seems to be missing: player expression. As you build out more complicated factories in a game like Satisfactory the player expresses their design sensibilities in their builds (how they prioritize aesthetics, compactness, modularity, etc.). At least so far, I don't see an equivalent space for player expression in Palworld (I'm not counting base building because it's not tied back into the core mechanics, bases are mostly cosmetic).

In any case, Palworld is very enjoyable and I'm interested to see how it unfolds both as I play more of it and as the developers evolve it during the early access period.

Neat simple LLM use for text manipulation

2024-01-06T00:00:00Z

I had a neat little interaction with an LLM this week where I had a markdown note in Obsidian I wanted to convert some of its formatting. The note had a bunch of external markdown links:

- [Note1](app://obsidian.md/path/to/Note1)
- [Note2](app://obsidian.md/path/to/Note2)
- ...

that I wanted to convert to internal links:

- [[Note1]]
- [[Note2]]
- ...

Previously to do this I'd either reach for something like a vim macro or write a little regex. Realistically for most cases the activation energy would be too much and I'd just not do the reformatting at all.

But with a LLM (in this case I used the TextGen plugin) you can just show an example and tell the LLM to do it and it just works. The prompt was something like

- [Note1](app://obsidian.md/path/to/Note1)
- [Note2](app://obsidian.md/path/to/Note2)
- ...

---

convert the list above from `[Note](path/to/Note)` to `[[Note]]`

It's pretty cool that we basically get a kind of programming by example for "free" with LLMs.

Some of the Apps I used in 2023

2023-12-26T00:00:00Z

Browser: Arc (it's soooo good, love the clever rethinking of tabs)
Notes: Obsidian (it's now running my life, so so good. I'd tried so many solutions over the years, this one is really sticking)
Read it later: Readwise Reader (basically what I've wanted for years, captures everything, handles article processing workflows like I want, syncs highlights to Obsidian)
Task Management: Things (switched from OmniFocus after > 10 years, OF 4 was not sparking joy)
Code: VS Code (meh, it's fine)
LLM for coding: Copilot (I find it hit or miss, but scary neat when it hits)
Task Launcher: Raycast (its ChatGPT integration is pretty great)
Mastodon Client: Ivory (for iPhone, love the feel) and Elk (for desktop Web, love the threading of conversations)
Calendar: Fantastical (continues to be great)
Podcasts: Overcast (continues to be great)
Email: Fastmail (great, starting to move from gmail), Spark (meh, slightly better than Mail.app)
Finance: YNAB (continues to be great)
RSS Reader: Feedbin and NetNewsWire (good apps but my RSS consumption is sporadic)

Using Obsidian with Readwise, dataview, and periodic notes

2023-07-09T00:00:00Z

This is a writeup of how I use Obsidian to organize my personal knowledge base (PKB).

My goal in building out a PKB is to capture and reflect on the most relevant thoughts, ideas, and events in my life. I've tried a bunch of different tools over the years and have found that it is critical to decide on the right level of structure to impose. Too much structure is overwhelming and nothing gets written down. Too little structure leads to sprawling write-once read-never snake nests.

As a disclaimer, PKBs are very, well, personal so this approach might not work for you.

At a high level, the kinds of notes I use are:

project notes
- notes to collect links, ideas, and brainstorms about a particular project
evergreen notes
- chunked concepts that I find particularly relevant (e.g. "Beware of automation bias")
daily, weekly, and quarterly notes
- the focus for this post

On any given day I use a daily note to capture whatever comes up. Daily notes serve as a kind of scratchpad to capture random thoughts, drafts of other documents, and any other bits of ephemera.

The only structure I impose in a daily note is for capturing meeting notes where I'll use a heading and link to the related project or person. For example, a 1/1 meeting will have a heading like # [[Person Name]] 1/1. The links in the heading to [[Person Name]] provide backlinks so that in the future I can quickly see all the topics we've covered.

For a while I tried to get by with just the daily note level of structure but it wasn't enough. It was too easy for something I wrote in a daily note to get lost. I needed a structured way to reflect on a longer timeline than a day.

So, I use the periodic notes plugin to create both weekly and quarterly notes.

The weekly note is more structured than a daily note and I use a template to fill in headings that I expand on throughout the week.

The first section is for reflection on how the week went. It uses a retro format I've used for years at work with simple "happy" and "sad" sections.

# Reflect

## Happy

- good progress on [[Project X]]
- three day weekend!

## Sad

- [[Twitter]] is at it again
- stalled discussion on [[Project Y]]
    - need to follow up with [[John Smith]]

The next section is to track the media I consumed that week. Any movies, shows, or games.

# Media Consumed

- [[Silo]]
- [[Factorio]]

I don't bother rating or writing down much else since the point is just to remember what and when I encountered various bits of media.

The final section is the Readwise highlights from the week. I use Readwise Reader for almost all my reading so when I make a highlight in Reader it gets automatically synced to Obsidian via the Readwise plugin. Having highlights synced is great for preservation but realistically I never looked at them again (outside of the Readwise spaced repetition emails). What I wanted was a way to see my recent highlights and have a chance to reflect on them again.

Enter the dataview plugin:

list
 from "Readwise"
 where
  file.ctime > date("{{date:YYYY-MM-DD}}")
  and file.ctime < date("{{date:YYYY-MM-DD}}") + dur(7 day)
sort file.ctime

That query pulls back a list of all the Readwise notes created during the week. At the end of the week I look over them and create any evergreen notes that might make sense.

The quarterly notes are similar to weekly notes but with a few more sections to encourage me to reflect a bit deeper.

Quarter notes start with a Happy/Sad reflection section just like weekly notes however this isn't meant to be inclusive of all the reflections from every week, just the most salient items from the quarter.

The other sections are aspects of my life that I want to spend time reflecting on and should be fairly self-explanatory:

Life Events
Professional Events
Tech Events
World Events
Media Consumed
Health
Finances

I try to spend a bit more time on each of these sections than I would in a weekly note, writing full sentences and paragraphs rather than just listing out bullet points.

So the way I use my PKB is essentially jotting down whatever is happening in the moment in a daily note, on the weekend filling out the weekly note and reflecting on highlights from that week, and then at the end of the quarter spending a bit more time reflecting on what happened in a number of different areas over the past few months.

I've struggled over the years trying to find the right cadence and tools to effectively capture and reflect on my "personal knowledge", but this combination of daily, weekly, and quarterly notes has been working pretty well for me so far.

Announcing ParserLang

2019-06-30T00:00:00Z

I'm excited to announce a new project I've been working on called ParserLang!

🎉🎉🎉

ParserLang is a parser combinator library for JavaScript with declarative superpowers. In addition to defining parsers in JavaScript, ParserLang allows you to use declarative syntax via template literals:

import { lang } from "parser-lang";

let { calc } = lang`
  num = /[0-9]+/ > ${ch => parseInt(ch, 10)};

  addExpr = num '+' multExpr > ${([left, op, right]) => left + right}
          | num ;

  multExpr = addExpr '*' multExpr > ${([left, op, right]) => left * right}
           | addExpr ;
  
  calc = multExpr ;
`;

calc.tryParse("1+1*2");
// 3

A thing that has surprised me about JavaScript in the post-ES2015 era is how few projects have made use of tagged template literals even though they are incredibly powerful. This is my attempt to rectify that situation.

Sweetening syntactic abstractions

2017-04-09T00:00:00Z

Version 3.0 of Sweet.js has just been released! 🎉

To set expectations, keep in mind that I consider Sweet to still be an experiment and under heavy work. The version number is a reflection of semver (we've made breaking changes) not project maturity. Since the big redesign last year tons of progress has been made, but I expect at least one or two more breaking changes (major version bumps) before things start to get baked and really ready to use. That said, if you are excited about the idea of true syntactic abstractions in JavaScript please dive in!

So what's new?

Custom Operators #

Custom operators are back! In the old pre-1.0 days we had the ability to define new operators with custom precedence and associativity but it was dropped in the redesign.

Operators are defined with the new operator keyword:

operator >>= left 1 = (left, right) => {
  return #`${left}.then(${right})`;
}

fetch('/foo.json') >>= resp => { return resp.json() }
                   >>= json => { return processJson(json) }

That expands to:

fetch("/foo.json").then(resp => {
  return resp.json();
}).then(json => {
  return processJson(json);
});

The implementation of custom operators is pretty experimental at the moment but give it a whirl and let us know if you run into any problems. More details are in the tutorial.

Note: the technical underpinnings for custom operators comes out of Jon Rafkind's dissertation on the Honu language.

Modules #

We've been steadily adding ES module support over the past few releases. The interaction between macros and modules is fairly complex so this is an ongoing process.

We currently have the ability to import macros from another module:

// foo.js
'lang sweet.js';
export syntax m = // ...

// main.js
'lang sweet.js';
import { m } from './foo';

m // ...

Note the use of the 'lang sweet.js' directives. These directives are currently required in any module that uses macros. It allows the Sweet compiler to avoid needlessly expanding modules that don't contain any macros. At present the directive is "just" an optimization but soon we'll be using it for some pretty cool stuff.

You can now also import modules into compiletime code (macro definitions) by using the for syntax form of import:

// log.js
'lang sweet.js';

export function log(msg) {
  console.log(msg);
}

// main.js
import { log } from './log.js' for syntax;

syntax m = ctx => {
  log('doing some Sweet things');
  // ...
}

We're taking the Racket approach of dividing everything up into phases. Runtime syntax is in phase 0 and compiletime macro definitions are in phase 1. Importing for syntax allows you to phase shift your code around. Phases greater than phase 1 happen when you import for syntax a macro that uses another macro that was imported for syntax. This gives rise to an infinite "tower of phases" which sounds complicated but turns out to be pretty straightforward in practice.

Still to come are better support for implicit runtime imports, finer grain support for phases that let you import for a specific phase, and an equivalent to Racket's begin-for-syntax.

Note: the technical underpinnings of modules and macros comes out of the Racket approach set forward by Matthew Flatt in his "You want it when?" paper.

Readtables #

While macros allow you to extend how syntax is parsed, sometimes you also need to extend how source text is lexed. The lexing extension approach we are taking is called readtables and @gabejohnson has been doing some amazing design and implementation work. Sweet now uses readtables internally and will soon be exposing them to users.

Internals and helpers #

During expansion Sweet constructs several intermediate representations of syntax that can be manipulated and eventually turned into a Shift AST. The exact representation we want to use is under flux but unfortunately it is exposed to macro authors inside macro definitions. Exposing what should be internal details is bad so to move away from that Sweet now provides a helper library for macro authors:

import * as H from 'sweet.js/helpers' for syntax;

syntax m = ctx => {
  let v = ctx.next().value;
  if (H.isIdentifier(v, 'foo') {
    return H.fromString(v, 'bar');
  }
  return H.fromString(v, 'baz');
}
m foo; // expands to 'bar'

Macro authors should only use the helper library to inspect and manipulate syntax objects rather than rely on the current representation of syntax. Eventually we will document and freeze a intermediate syntax representation but until then just use the helpers.

What's next? #

The current plan is to get Sweet to a solid and stable place where we can start building declarative conveniences on top its foundation. In particular, the current macro definition syntax is intentionally low-level and not convenient to work in. We've got some ideas about what this might look like but first we're going to make sure the base is solid.

If any of this excites you, please jump in! We'd love to have you!

Announcing Sweet.js 1.0

2016-03-28T00:00:00Z

Sweet.js, the hygienic macro system for JavaScript, just got a shiny new release: the magical 1.0! 🎉

If you’ve been following the development of Sweet for a while you might have noticed a dearth of activity. Part of that was life (I recently finished grad school and started a new job) but part of that seeming lack of activity was actually a giant rewrite in progress.

Yes I know, that’s a thing you should never do.

But I swear it made sense this time. The collection of bad decisions I made early in development had finally made forward progress all but impossible. And so, Sweet has been completely rewritten.

So what’s good about the rewrite? Lot’s of stuff, but two I want to call out specifically here.

First, a proper parser is now integrated in the expansion pipeline. Previously, Sweet had a half-baked pseudo-parser that kinda-sorta built up an AST during expansion and then immediately flattened it back to tokens to let our crazy fork of esprima build up a proper AST before doing codegen.

Yeah, totes reasonable.

This insane pipeline had tons of bugs especially related to proper ES2015 support.

Now we have a proper parser modeled after the Shift Parser and producing a Shift AST with Babel as an optional backend for great ES2015 support everywhere. It’s amazing.

Second, Sweet now has much more reasonable macro binding forms. Previously, you could define a macro in two ways, the recursive and non-recursive declaration forms:

// recursive form
macro foo { /* ... */ }

// non-recursive form
let foo = macro { /* ... */ }

The recursive form binds the macro name inside the macro definition while the non-recursive form does not. This was gross and confusing because standard let in ES2015 does not work like this at all. Now, Sweet uses syntax and syntaxrec:

// non-recursive form
syntax foo = function (ctx) { /* ... */ }
// recursive form
syntaxrec foo = function (ctx) { /* ... */ }

These are more symmetric and fit in better as a compiletime extension of the var/let/const binding forms of ES2015.

While all these changes are great, there are a few items from the pre-1.0 days that have not been re-implemented yet. Custom operators and infix macros in particular are not currently supported. However, the foundation provided by the rewrite will make adding these features back pretty straightforward so expect them to be available soon.

So should you dive in with the new Sweet? Well, to be honest it really depends on how risk averse you are. Since so much has changed with the rewrite there are bound to be bugs we still need to shake out. Maybe don’t go running Sweet over your production code just yet.

But if you’d like to help out by putting Sweet through its paces we would love to have you! The best way to get started is to familiarize yourself with all the new syntax by reading the tutorial. If you’ve got questions or need help, head on over to gitter or #sweet.js on irc.mozilla.org.

Have fun sweetening your code!

Hygiene in sweetjs

2013-09-27T00:00:00Z

The most important feature of sweet.js is hygiene. Hygiene prevents variables names inside of macros from clashing with variables in the surrounding code. It's what gives macros the power to actually be syntactic abstractions by hiding implementation details and allowing you to use a hygienic macro anywhere in your code.

For hygiene to work sweet.js must rename variables. Recently several people have asked me why sweet.js renames all the variables. Wouldn't it be better and cleaner to only rename the variables that macros introduce?

The tl;dr is "because hygiene" but let's unpack that a little.

Hygiene Part 1 (Binding) #

The part of hygiene most people intuitively grok is keeping track of the variable bindings that a macro introduces. For example, the swap macro creates a tmp variable that should only be bound inside of the macro:

macro swap {
  rule { ($a, $b) } => {
    var tmp = $a;
    $a = $b;
    $b = tmp;
  }
}

var tmp = 10;
var b = 20;
swap (tmp, b)

Hygiene keeps the two tmp bindings distinct by renaming them both:

var tmp$1 = 10;
var b$2 = 20;

var tmp$3 = tmp$1;
tmp$1 = b$2;
b$2 = tmp$3;

This is the point where most people say "why bother renaming the variables outside of the macro"? Can't you just rename the bindings created by the macro? Wouldn't it be cleaner for the expansion to just be something like:

var tmp = 10;
var b = 20;

var tmp$1 = tmp;
tmp = b;
b = tmp$1;

Hygiene Part 2 (Reference) #

The complication comes in with variable references. The body of a macro can contain references to bindings declared outside of the macro and those references must be consistent no matter the context in which the macro is invoked.

Some code to clarify. Let's say you have a macro that uses a random number function:

var random = function(seed) { /* ... */ }
let m = macro {
    rule {()} => {
        var n = random(42);
        // ...
    }
}

This macro needs to refer to random in any context that it gets invoked. But its context could have a different binding to random!

function foo() {
    var random = 42;
    m ()
}

Hygiene needs to keep the two random bindings different. So sweet.js will expand this into something like:

var random$1 = function(seed$4) { /* ... */ }
function foo() {
    var random$2 = 42;
    var n$3 = random$1(42);
    // ...
}

Note that there is no way for hygiene to do this if it only renamed identifiers inside of macros since both random bindings were declared outside of the macro. Hygiene is necessarily a whole program transformation.

(ps if this sort of feels like a closure you're on to something: one of the early techniques that led to modern hygiene algorithms was called syntactic closures)

Strictly speaking the hygiene algorithm is still conservative. Variable bindings declared outside of a macro that are never referenced by a macro don't really need to be renamed. However, modifying the hygiene algorithm to only rename exactly what needs to be renamed seems pretty difficult (especially to do so efficiently). If anyone knows techniques for this definitely let me know (or even better submit a pull request).

How to read macros

2012-12-20T00:00:00Z

In my last post I gave a little overview of sweet.js the hygienic macro system I built over the summer. Today I want to write a little bit about what makes sweet.js possible and why we haven't really seen a macro system for JavaScript before now. I gave hints at some of this in my intern talk but now we can finally do a deep dive!

Basics #

First, let's take a look at compilers 101:

The traditional way you build a compiler front end is to first write a lexer and a parser. Code (a string) is fed to the lexer which produces an array of tokens which gets fed to the parser which produces an Abstract Syntax Tree (AST).

For example,

if (foo > bar) {
    alert("w00t!");
}

gets lexed into something like:

["if", "(", "foo", ">", "bar", ")", "{" ...]

The lexer is basically responsible for throwing away unnecessary whitespace and grouping identifiers, strings, and numbers into discrete chunks (ie tokens). The array of tokens is then parsed into an AST that might look something like this:

// output of esprima
{
    "type": "Program",
    "body": [
        {
            "type": "IfStatement",
            "test": {
                "type": "BinaryExpression",
                "operator": ">",
                "left": {
                    "type": "Identifier",
                    "name": "foo"
                },
                "right": {
                    "type": "Identifier",
                    "name": "bar"
                }
            },
            "consequent": {
                "type": "BlockStatement",
                "body": [{
                        "type": "ExpressionStatement",
                        "expression": {
                            "type": "CallExpression",
                            "callee": {
                                "type": "Identifier",
                                "name": "alert"
                            },
                            "arguments": [{
                                    "type": "Literal",
                                    "value": "w00t!",
                                    "raw": "\"w00t!\""
                                }]
                        }
                    }]
            },
            "alternate": null
        }
    ]
}

The AST gives you the structure necessary to do code optimization/generation etc.

So where can we fit macros into this picture? Which representation is best for macros to do their stuff?

Well, by the time we get to an AST it's too late since parsers only understand a fixed grammar (well, technically there is research on adaptive/extensible grammars but that way leads to madness!). Obviously the raw code as a string is too unstructured for macros so how about the array of tokens produced by the lexer?

Tokens are fine for cpp #define style macros but we want moar power! And, as it turns out, just normal tokens aren't going to cut it for us. Consider this simple macro that provides a concise way to define functions:

macro def {
    case $name $params $body => {
        function $name $params $body
    }
}
def add(a, b) {
    return a + b;
}

which should be expanded into:

function add(a, b) {
    return a + b;
}

Critically, note that the macro needs to match $params with (a, b) and $body with { return a + b; }. However, we don't have enough structure to do this with just the token array ["def", "add", "(", "a", ",", "b", ...]: we need to match the delimiters ((), {}, and []) first!

If you remember your compilers class (or went to wikipedia), delimiter matching is what separates context-free languages (what parsers recognize) from regular languages (what lexers recognize).

This is one of the reasons why macros are big in the lisp family of languages (scheme, racket, clojure, etc.). S-expressions (with (all (those (parentheses)))) are already fully delimited so it becomes almost trivial to do delimiter matching. Some people say this is due to homoiconicity but as Dave Herman pointed out, homoiconicity isn't really the point. It's not that the lisp family is homoiconic but rather that the nature of s-expressions makes it easy to implement read which is necessary for macros.

read is the crucial function that gives a little bit more structure to the array of tokens by matching up all those delimiters. Now instead of just a flat array of tokens we are going to get a read tree:

["def", "add", {
    type: "()",
    innerTokens: ["a", ",", "b"]
    }, {
    type: "{}",
    innerTokens: ["return", "a", "+", "b"]
}]

Note this doesn't have all the structure of a full AST, it just knows about tokens and delimiters not expressions and statements. So now our def macro pattern variables will match up like so:

$params -> {
    type: "()",
    innerTokens: ["a", ",", "b"]
}
$body -> {
    type: "{}",
    innerTokens: ["return", "a", "+", "b"]
}

Great! Now our pipeline looks something like:

The Devil in the Details #

Ok, so all well and good but then why haven't we seen people implement read and build a macro system for JavaScript before now?

It turns out that there's this really annoying token (for potential macro implementers) in JavaScript: /.

Here's the problem, depending on context / can mean two different things: the divide operator or the beginning of a regular expression literal.

10 / 2              // 5
/foo/.test("foo")   // true

(Well technically I guess / can also mean the start of a comment but this is always easy to figure out (since // always means line comment))

So how do we disambiguate between divide and regex? It turns out that the way a normal parser (like esprima for example) does it is by running the lexer and parser together and resolving the ambiguity via the current parsing context. In other words, as the parser is working through each production, it calls out to the lexer with a flag saying what context it is in. Depending on that context the lexer will either lex / as a divide token or as a regular expression literal.

But, we can't use the parsing context in read because we don't have any parsing context yet!

So, we somehow need to separate the lexer/reader from the parser.

Now you might think we could get away with just leaving / as an ambiguous token (say a divOrRegex token for example) to be handled by the parser once all the macros have been expanded away but consider this code fragment we might want to read:

... { /foo}bar/ ...
// as a token array this would be
// [... "{", "/", "foo", "}", "bar", "/", ...]

Remember that the entire point of read is to do delimiter matching, so should we match the } with the opening { or as part of a regular expression literal (remember /}/ is a valid regex that matches a single })? It completely depends on our interpretation of /!

Therefore, in our lexer/reader we must disambiguate the meaning of / without the help of the parser. So how do we do that?

This is the hard technical problem that Paul Stansifer (he also designed the Rust macro system) solved this summer, unlocking the power of JavaScript macros for us all!

The basic idea is when you see a / as you are reading, just look back a couple of tokens and a small fixed set of tokens will determine unambiguously if / should be a divide or the start of a regex literal. To figure out exactly how far back and which tokens to look for requires working through all the different cases in the JavaScript grammar which is hard but done!

A snippet of this algorithm goes something like:

if tok is / if tok-1 is ) look back to matching ( if identifier before ( in "if" "while" "for" "with" tok is start of regex literal else tok is divide ...

For example, if we have:

if (foo + 24 > bar) /baz/.test("foo")

When we see the / we note that the previous token was ) so we find its matching ( and note that the token before that was if so / must be the beginning of a regular expression.

What's really cool here is that when we need to disambiguate / we've already been reading up to that point so (foo + 24 > bar) is a single token (the () token with inner tokens foo, +, 24, >, and bar) and checking the token before the parens is literally as simple as tokens[idx-2] === "if". By creating the read tree as we go along we don't need to carry lookbehind state in a complicated way; in fact, in the worst case, we only have to look back 5 tokens.

If you want to read more about how this works, I've got the entire algorithm pseudo-coded up here and the actual JavaScript implementation in these relatively short two functions.

Hygienic Macros for JavaScript

2012-10-14T00:00:00Z

Been slow to embloggen this (start of the quarter etc.) but my summer intern project was released a little while ago at sweetjs.org. Sweet.js is a hygienic macro compiler for JavaScript that takes JavaScript written with macros and produces normal JavaScript you can run in your browser or on node. It's an experiment to see if macros can work well for JavaScript.

Macros allow you to extend the syntax of JavaScript by writing macro definitions that perform syntax transformations, thus allowing you to do cool things like add sweet syntax for var destructuring or even haskell-like do notation.

The idea is to provide a middle ground between no syntax extension and having to build an entire compile-to-js language like CoffeeScript, which can't compose with other syntax extensions.

I talk a bit about the motivation and design in this presentation I gave at the end of my internship.

The language that most directly inspires sweet.js is, of course, scheme. I'm not really much of a schemer myself but have always admired the fancy work going on in that world and macros are pretty fancy. At the start of the summer I was still somewhat of a macro newbie but being at Mozilla was fantastic since I was able to draw on and learn from two well-versed scheme macrologists Dave Herman (who's PhD thesis was on macros) and Paul Stansifer (who has been developing the Rust macro system).

Sweet.js is still in very early stages, lots of bugs and missing features, but I think it shows some real promise. Let me know what you think!

Memory Checking in Low-Level JavaScript

2012-07-18T00:00:00Z

So this past month I've been helping out on the Low-Level JavaScript (LLJS) project. Bit of a change for me since I usually hide out at as high a level as I can possibly get :)

LLJS is an experiment in adding low-level features like pointers and manual memory management to a dialect of JavaScript. These low-level features are pretty cool and can get you some nice performance wins, but they also come at a cost since you are always one null pointer dereference or memory leak away from oblivion.

One way C/C++ programmers have handled this danger is by using an analysis tool like Valgrind to detect memory errors that happen while the program is running. Since memory checking has proven to be useful in the C world, we figured it might be helpful for LLJS. So, I've hacked up a valgrind-like memory analysis for LLJS. It can detect four kinds of memory errors:

reads/writes to unallocated memory (pointers we haven't malloc'ed)
reads from undefined memory (malloc'ed memory we haven't written to)
bad frees (calling free on memory that hasn't been malloc'ed)
leaks (any unfreed memory at the end of the program run)

How to use #

Go grab the latest version of LLJS from its github repo. Then add the following snippet to the end of your script:

let m = require('memory');
console.log(m.memcheck.report());

This pulls in the memory module and prints out the result of the memory checking (this snippet is for node.js, so use load if you're using SpiderMonkey or grab memory off the global object if you're in the browser).

Then compile your LLJS code with the -m flag:

bin/ljc -m -o your_file.js your_file.ljs

and run it:

node --harmony-proxies your_file.js

As you can see the memory checker uses Proxies, which node/V8 hides behind a flag (no flag needed on SpiderMonkey).

For example, the following script:

extern console;

function test_unallocated() {
  let uint *x;
  // not allocated yet!
  return *x;
}
test_unallocated();

function test_leak() {
  // leak!
  let uint *leak = new uint;
}
test_leak();

function test_free() {
  let uint *x = new uint;
  delete x;
  delete x;
}
test_free();

let m = require('memory');
console.log(m.memcheck.report());

will print out the following report:

Unallocated: 0 at: test_unallocated:3:0

Undefined: 0 at: test_unallocated:3:0

Bad frees: 8184 at: test_free:16:0

Leaks: 8200 at: test_leak:10:0

How it works #

Memory allocation in LLJS is implemented as a big typed array where pointers are really just array indexes into the array.

To do memory checking we create a second shadow memory typed array that stores the allocation status of each byte of real memory (if the byte has been allocated, initialized, etc.). Calls to malloc and free change the allocation status of the bytes in shadow memory. The real memory array is wrapped in a Proxy that intercepts each get and set operation and checks the shadow memory for a possible error.

For example, say we we allocate an int and try to do some pointer arithmetic and dereference unallocated memory:

let int *x = new int;
*(x+1)

When the dereference of x+1 happens, the Proxy wrapping real memory will check in shadow memory to see if x+1 has been allocated. Since it hasn't, an unallocated memory access error is recorded.

This approach is basically a simplified version of how Valgrind does memory checking.

Stable node.js now runs contracts.coffee!

2012-07-15T00:00:00Z

Took almost a year but now the stable version of node.js (0.8.0+) has a recent enough version of V8 to run contracts.coffee! The much needed proxy support is still behind a flag so you'll need to supply --harmony_proxies and --harmony-collections as usual.

Also, the latest version of contracts.coffee (0.3.1) and is now up on npm and collects a few bugs fixes.

Link Roundup

2012-05-13T00:00:00Z

*JS is a really cool (if ungoogleable) project that attempts to be "the bastard child of JavaScript and C" by giving you the low level tools to build fast programs in a high level language.

After the recent excitement surrounding Bret Victor's amazing talk, Jonathan Edwards has a sobering essay about the limits of IDE innovation. Makes the point that languages must co-evolve to get real innovation.

Wired has a nice writeup on the current state of computer security (spoiler: it sucks) and The New Yorker has a great article on the recent geohot vs. Sony craziness.

Link Roundup

2012-04-28T00:00:00Z

Lots of discussion around classes and other changes being considered for the future of JavaScript. Interesting how difficult it has been for the standardization committee (or the JS community in general) to come to a consensus on what classes should look like (or if they should even exist).

In what is almost becoming a weekly story, emscripten has been used to impressive effect again. This time SpiderMonkey (the JS engine in Firefox) has been compiled to JavaScript.

Finally, an explanation of why Lisp’s syntax is so important and useful that finally clicked for me. The key: "it's possible to read it without parsing."

The greatest language in the world has been made. PL is done, we can all go home now.

contracts.coffee now runs in Chrome!

2012-04-19T00:00:00Z

Chrome can now run contracts.coffee! I've been waiting a while for the proxy support in Chrome to get far enough along but it's finally here.

Chrome hides experimental JavaScript features like proxies behind flags so you'll need to go to about:flags and enable the experimental JavaScript flag.

Try it out and if you run into any issues let me know on github.

Updates to contracts.coffee

2012-04-16T00:00:00Z

A new version of contracts.coffee is up on npm. We're now synced with the latest release version of CoffeeScript (1.3.1) and have collected a few bug fixes. We also have experimental require.js/AMD support. Change log here.

Get it with npm install contracts.coffee.

Weekly Link Roundup

2012-04-08T00:00:00Z

Details surface about Google's project Glass. The future is now and all that. Rainbows End and Daemon/Freedom™ are now required reading.

I didn't realize when I first saw it but xkcd's April 1st comic was actually quite brilliant.

Some good advice on giving academic talks under the twin lenses of memory (your audience's resource — build context and be wary of details) and fear (your resource — desensitize and dissociate).

An interesting privacy hack for the web was announced and kickstarted. Amusingly enough they use Libya's ccTLD (.ir or .eg would also have been apropos). Hopefully they've thought about domain seizures.

And on a related note, here's a good read on the internet wars. In particular the forces of order (SOPA/PIPA, Iran, China, Egypt, etc.) vs. the forces of chaos (Anonymous, LulzSec, etc.).

Link Roundup

2012-03-26T00:00:00Z

Gizmodo has a surprisingly well-reasoned critique of Google. Far better than the usual knee-jerk Google bashing that too often passes for criticism online these days.

Kony 2012 has probably done more for raising the discourse on US "aid" than its stated goal (killing a guy... er, I mean "raising awareness"). Also, Žižek.

Looks like the web lost the video (H.264) battle. Lame.

A fantastic retrospective on BitC with some echoing comments from the Rust team.

Fear not! The proof is trivial.

Link roundup

2012-03-18T00:00:00Z

Crista Lopes bemoans the state of PL research with good responses from Matthias Felleisen and LtU (read moar Kuhn!).

SQLite compiled to JavaScript via Emscripten

Mark Lynas seems to think we can get the entire world out of poverty without breaking the planetary boundaries (think carbon, biodiversity, land use, etc.) by 2050. Cute.

Game of Thrones is available at long last online but who's complaining.

The NSA is apparently building a massive data center in Utah. This doesn't concern me at all.

Occupying Language Design at POPL

2012-02-02T00:00:00Z

At POPL last week I gave a student lightning talk on our Virtual Values paper. The format was four minutes with the slides advancing every 20 seconds. Very difficult to communicate the important idea in so short a time but it was really fun and I think turned out pretty well.

Check out the slides if you're interested. I added notes on each slide that corresponded more or less to what I was saying.

Updates to contracts.coffee

2012-01-04T00:00:00Z

An update to contracts.coffee (0.2.0) is now up on npm. I've fixed up a few bugs since the intial release and it's now based on the latest CoffeeScript (1.2.0) but the big update is a rework of how modules are handled.

Before this update we were handling module blame tracking by forcing the programmer to call .use() before a contracted value could be used. This did the work of figuring out who the server and client module.

This was an awkward way to enforce a module pattern since not only do you have to write more code but it also means you have to know which values in a module have a contract. This breaks abstraction which is bad.

So with this latest update we're completely dropping .use(). How we keep tack of the modules without it depends on whether you are using node.js or a browser.

If you are using node.js then the wiring is handled automatically by wrapping the require function and exports object. If you put a contract on an export value the correct server and client module names are recorded.

# file Math.coffee exports.square :: (Num) -> Pos exports.square = (x) -> x * x

#####

# file Main.coffee {square} = require './Math'

# vioation blaming Main.coffee square "a string"

You can also put a contract on a value that isn't on the exports object (for use inside the module), but the module name might get confusing since it's just the file name with "(value)" or "(caller)" appened. I will look at adding something like Racket's nested boundaries to clean this up at some point.

If you're using the browser, things are a bit more complicated. Since the browser environment doesn't currently have a standard way to do modules, we're stuck doing the wiring by hand. Details are in the documentation but the basic idea is that the library now provides Contracts.exports and Contracts.use which are used to construct contract aware modules.

Contracts.exports creates and names an exports object:

# Math.coffee

# create and name the exports object exports = Contracts.exports "Math"

exports.square :: (Num) -> Pos exports.square = (x) -> x * x

# put the exports object on the global object # for other modules to see and use window.Math = exports

And Contracts.use pulls in a module and names the user of the module.

# Main.coffee {square} = Contracts.use window.Math, "Main"

square 4 # 16 square "a string" # Contract Violation...blaming Main

I'll look at ways to automate this more in future updates.

Performance of contracts.coffee

2011-09-27T00:00:00Z

I've been meaning to get around to running some performance benchmarks on contracts.coffee, but I recently learned a valuable lesson: let the internet do your work for you! :-)

I woke up this morning to find @paulmillr had put together a nice performance benchmark of contracts.coffee.

Check it out for all the gory details but suffice it to say contracts makes things...slow. This is about what I'd expected since in addition to running the actual contract check we must wrap the contracted functions/objects in a Proxy which runs handlers for each function call or property set/get. Lots of stuff is in the way of the running code.

So does the slow performance of contracts make them unusable? Not necessarily. First off, the contracts.coffee compiler can emit JavaScript with contracts completely disabled. So, you can have a "slow" testing/development build with contracts enabled to help track down bugs and a production build with contracts disabled and no slowdown.

In addition, most code is IO bound not CPU bound. So if you want contracts enabled everywhere but still care about performance, be smart about where you apply them. Put them on module boundaries, but don't put them on tight loops for example.

In sum, contracts are another tool in the software engineer's toolbox that have advantages (bug squashing) and disadvantages (performance slowdowns when enabled). Use them wisely.

Contracts.coffee Works In Node.js!

2011-09-20T00:00:00Z

Contracts.coffee now works in node.js! V8 (the JavaScript engine used by node) has recently added support for Proxies which contracts.coffee requires.

You'll need a bleeding-edge version of node to get it running. Try either master (unstable) or 0.5.7. Build instructions are here.

Proxies are hidden behind a command line flag in V8 so you'll also need to supply --harmony_proxies to node when calling the compiler:

node --harmony_proxies bin/coffee -CL test.coffee

There are almost certainly node-specific bugs in contracts.coffee so don't trust it with anything important yet. You can get started playing though!

Report any bugs you run into on the issue tracker.

Duck-Typing Invariants In contracts.coffee

2011-09-05T00:00:00Z

So the response I've been getting for contracts.coffee has been absolutely incredible! There's been tons of positive feedback and some really great excitement about the project.

I've also heard a few worries that contracts are too much like static types and might not be such a good fit for a dynamic language like CoffeeScript or JavaScript. Now, I don't want to get into a huge debate over the merits of static vs. dynamic typing (that way leads to madness and holy wars!) but I would like to clarify a few points and show that contracts actually fit quite nicely into the dynamic ethos of CoffeeScript/JavaScript.

In dynamic languages like CoffeeScript and JavaScript you'll often hear about duck-typing where we don't care about the "type" of an object, just the stuff on the object we actually need to use. So the classic example is a "file-like" object where all we care about are the few properties the code actually uses (like open, read, close, etc.). As long as the properties exist we're good. If a property don't exist and we try to use it, just throw an exception so we can track down the bug.

Contracts allow us to extend this concept quite a lot without sacrificing the awesome flexibility we dynamic language users love about duck-typing. To start out, let's consider the simple case of a missing property. Let's first define a contract for a binary tree:

# A basic binary tree BinaryTree = ?(Null or { node: Num left: Self or Null right: Self or Null })

Pretty simple, a binary tree is either null or has a node and two children that are also binary trees. A function can now say it expects a BinaryTree:

f :: (BinaryTree) -> ... f = (tree) -> n = tree.node ...

And if we call f with something that isn't quite a binary tree:

broken_tree = nodee: 42 # misspelled! right: null left: null

f broken_tree

We get a very descriptive error message:

Contract violation: expected <{node: Num, left: self or Null,
                                right: self or Null}>,
  actual: "[missing property: node]"
Value guarded in: coffee-script.js:53
  -- blame is on: coffee-script.js:56
Parent contracts:
{node: Num, left: self or Null, right: self or Null}
Null or {node: Num, left: self or Null, righ : self or Null}
(Null or {node: Num, left: self or Null, right: self or Null})
  -> any

Now we're already winning since the error message is triggered as soon as the function is called with an object that is missing the required properties. If we weren't using contracts, the variable n in the example above would get the value undefined which might not surface as an obvious problem until much later (if ever!).

We're also not losing any of the flexibility of duck-typing since contracts allow us to specify just the object properties the function needs in order to work properly. We're just being explicit about how our ducks should quack!

In addition to an early failure, we get lots of information in the error message about what was missing, which contract was violated, and which module was to blame (a powerful notion I explain a little here but which really deserves a future blog post).

But we're just getting warmed up :)

Contracts on objects can also have arbitrary invariants that get checked whenever properties are modified. So we can have a contract for a binary search tree:

# A binary search tree is a binary tree where each node is # greater than the left child but less than the right child BinarySearchTree = ?(Null or { node: Num left: Self or Null right: Self or Null | invariant: -> (@.node >= @.left.node) and (@.node < @.right.node) })

And a red-black tree:

# A red-black tree is a binary search tree # that keeps its balance RedBlackTree = ?(Null or { node: Num color: Str left: Self or Null right: Self or Null | invariant: -> (@.color is "red" or @.color is "black") and (if @.color is "red" (@.left.color is "black" and @.right.color is "black") else true ) and (@.node >= @.left.node) and (@.node < @.right.node) })

Note that our contract for a red-black tree is exactly the same as a binary search tree with the addition of the color property and some related invariants. We have a kind of subtyping going on here: a function that expects a binary search tree will also work with a red-black tree but not vica versa.

Now our functions can "duck-type" the invariants on the objects they accept:

takesBST :: (BinarySearchTree) -> Any takesBST = (bst) -> ...

takesRedBlack :: (RedBlackTree) -> Any takesRedBlack = (rbTree) -> ...

bst = makeBinarySearchTree() rb = makeRedBlackTree()

takesBST bst # works fine takesBST rb # works fine

takesRedBlack rb # works fine takesRedBlack bst # might fail if the full # red-black invariants don't hold!

These functions not only specify that they require certain properties but also that certain invariants must hold. This is really powerful! The function that only requires the behavior of a binary search tree can accept a red-black tree just fine and the function that requires the behavior of a red-black tree but is given just a binary search tree will fail when the invariants are violated (just as we want it to).

So, contracts allow us to be explicit and fail early when things go wrong while still giving us the flexibility of duck-typing. And they work for both "missing-property" kinds of failures along with a more general "failed invariant".

Contracts.coffee - Contracts for JavaScript and CoffeeScript

2011-08-23T00:00:00Z

This summer I've been working at Mozilla looking at bringing the awesomeness that is contracts to JavaScript. If you aren't familiar with contracts, think of them as super powerful asserts.

Why contracts? Because they allow us to build better software! They allow us to express invariants (things that will always be true about our code) in JavaScript about JavaScript and check at runtime that these invariants hold. And when things break (like they always do) we get a precise error message pin-pointing the part of our code the broke the contract.

Contracts in programming languages were first popularized in the language Eiffel. In fact, the people behind Eiffel promoted an entire design methodology centered around contracts (called, appropriately enough, Design by Contract™) that encouraged thinking about invariants and interfaces between software component boundaries.

So if contracts are so great why don't we see them in JavaScript? Well, for the most part, the languages that have supported contracts in the past all share an important characteristic...no lambdas. Or more specifically in Java and Eiffel functions are not first class (you can't pass a function in a method call). Traditional contract systems fall down in a higher order setting (how do you check a function argument?).

Until, that is, the scheme people (err...I mean Racket people) figured out what to do a few years ago. Now, Racket has a very nice contract system.

But we haven't seen anything comparable to Racket's contracts in JavaScript so far. So this summer I looked into taking the work that was done in Racket and trying to fit it into JavaScript. As you probably know, JavaScript has a bit of scheme in it (just confused by some C/Java syntax).

The result of this exploration is a JavaScript libarary called contracts.js and a fork of the CoffeeScript compiler (which translates CoffeeScript directly to JavaScript) called contracts.coffee. The JavaScript library gives us the ability to apply contracts to our code and the CoffeeScript extension gives us some really pretty syntax.

So what does it all look like?

Here's some CoffeeScript code with contracts:

id :: (Num) -> Num id = (x) -> x

And the JavaScript translation:

var id = guard(
     fun(Num, Num),
     function(x) { return x; })

This code takes the identity function and wraps it in a function contract. Now every time id is called the contract library first checks that the argument being passed in satisfies the Num contract (which checks the the value is a number) and that the result of id also satisfies Num. If any of these checks fail, an error is thrown. For example:

id("string")

Error: Contract violation: expected <Number>, actual: "string"
Value guarded in: id_module.js:42
  -- blame is on: client_code.js:104
Parent contracts: (Number) -> Number

Pretty nifty right?

We can also have contracts on objects:

person :: name: Str age: Num person = name: "Bertrand Meyer" age: 42

And arrays:

loc :: [...Num] loc = [99332, 23452, 123, 2, 5000]

And various combinations thereof:

average :: ({name: Str, age: Num}, [...Num]) -> Str average = (person, loc) -> sum = loc.reduce (s1, s2) -> s1 + s2 "#{person.name} wrote on average #{sum / loc.length} lines of code."

You can find documentation, install instructions, and a bunch more examples for contracts.coffee at its website. Docs for the underlying contracts.js library is coming soon but for now just check out the github page.

This is still very much a work in progress and I would love feedback. Let me know if you find it useful!