Quoting Andrej Karpathy
http://simonwillison.net/2023/Aug/16/andrej-karpathy/#atom-everything
llama.cpp surprised many people (myself included) with how quickly you can run large LLMs on small computers [...] TLDR at batch_size=1 (i.e. just generating a single stream of prediction on your computer), the inference is super duper memory-bound. The on-chip compute units are twiddling their thumbs while sucking model weights through a straw from DRAM. [...] A100: 1935 GB/s memory bandwidth, 1248 TOPS. MacBook M2: 100 GB/s, 7 TFLOPS. The compute is ~200X but the memory bandwidth only ~20X. So the little M2 chip that could will only be about ~20X slower than a mighty A100.— Andrej Karpathy (https://twitter.com/karpathy/status/1691571869051445433)
http://simonwillison.net/2023/Aug/16/andrej-karpathy/#atom-everything
llama.cpp surprised many people (myself included) with how quickly you can run large LLMs on small computers [...] TLDR at batch_size=1 (i.e. just generating a single stream of prediction on your computer), the inference is super duper memory-bound. The on-chip compute units are twiddling their thumbs while sucking model weights through a straw from DRAM. [...] A100: 1935 GB/s memory bandwidth, 1248 TOPS. MacBook M2: 100 GB/s, 7 TFLOPS. The compute is ~200X but the memory bandwidth only ~20X. So the little M2 chip that could will only be about ~20X slower than a mighty A100.— Andrej Karpathy (https://twitter.com/karpathy/status/1691571869051445433)
An Iowa school district is using ChatGPT to decide which books to ban
http://simonwillison.net/2023/Aug/16/chatgpt-to-ban-books/#atom-everything
An Iowa school district is using ChatGPT to decide which books to ban (https://arstechnica.com/information-technology/2023/08/an-iowa-school-district-is-using-chatgpt-to-decide-which-books-to-ban/)
I'm quoted in this piece by Benj Edwards about an Iowa school district that responded to a law requiring books be removed from school libraries that include "descriptions or visual depictions of a sex act" by asking ChatGPT "Does [book] contain a description or depiction of a sex act?".
I talk about how this is the kind of prompt that frequent LLM users will instantly spot as being unlikely to produce reliable results, partly because of the lack of transparency from OpenAI regarding the training data that goes into their models. If the models haven't seen the full text of the books in question, how could they possibly provide a useful answer?
http://simonwillison.net/2023/Aug/16/chatgpt-to-ban-books/#atom-everything
An Iowa school district is using ChatGPT to decide which books to ban (https://arstechnica.com/information-technology/2023/08/an-iowa-school-district-is-using-chatgpt-to-decide-which-books-to-ban/)
I'm quoted in this piece by Benj Edwards about an Iowa school district that responded to a law requiring books be removed from school libraries that include "descriptions or visual depictions of a sex act" by asking ChatGPT "Does [book] contain a description or depiction of a sex act?".
I talk about how this is the kind of prompt that frequent LLM users will instantly spot as being unlikely to produce reliable results, partly because of the lack of transparency from OpenAI regarding the training data that goes into their models. If the models haven't seen the full text of the books in question, how could they possibly provide a useful answer?
Running my own LLM
http://simonwillison.net/2023/Aug/16/running-my-own-llm/#atom-everything
Running my own LLM (https://nelsonslog.wordpress.com/2023/08/16/running-my-own-llm/)
Nelson Minar describes running LLMs on his own computer using my LLM tool and llm-gpt4all plugin, plus some notes on trying out some of the other plugins.
Via @nelson (https://tech.lgbt/@nelson/110900514481858968)
http://simonwillison.net/2023/Aug/16/running-my-own-llm/#atom-everything
Running my own LLM (https://nelsonslog.wordpress.com/2023/08/16/running-my-own-llm/)
Nelson Minar describes running LLMs on his own computer using my LLM tool and llm-gpt4all plugin, plus some notes on trying out some of the other plugins.
Via @nelson (https://tech.lgbt/@nelson/110900514481858968)
Datasette Cloud, Datasette 1.0a3, llm-mlc and more
http://simonwillison.net/2023/Aug/16/datasette-cloud-weeknotes/#atom-everything
Datasette Cloud is now a significant step closer to general availability. The Datasette 1.03 alpha release is out, with a mostly finalized JSON format for 1.0. Plus new plugins for LLM and sqlite-utils and a flurry of things I've learned.
Datasette Cloud
Yesterday morning we unveiled the new Datasette Cloud blog (https://www.datasette.cloud/blog/), and kicked things off there with two posts:
Welcome to Datasette Cloud (https://www.datasette.cloud/blog/2023/welcome/) provides an introduction to the product: what it can do so far, what's coming next and how to sign up to try it out.
Introducing datasette-write-ui: a Datasette plugin for editing, inserting, and deleting rows (https://www.datasette.cloud/blog/2023/datasette-write-ui/) introduces a brand new plugin, datasette-write-ui (https://datasette.io/plugins/datasette-write-ui) - which finally adds a user interface for editing, inserting and deleting rows to Datasette.
Here's a screenshot of the interface for creating a new private space in Datasette Cloud:
http://simonwillison.net/2023/Aug/16/datasette-cloud-weeknotes/#atom-everything
Datasette Cloud is now a significant step closer to general availability. The Datasette 1.03 alpha release is out, with a mostly finalized JSON format for 1.0. Plus new plugins for LLM and sqlite-utils and a flurry of things I've learned.
Datasette Cloud
Yesterday morning we unveiled the new Datasette Cloud blog (https://www.datasette.cloud/blog/), and kicked things off there with two posts:
Welcome to Datasette Cloud (https://www.datasette.cloud/blog/2023/welcome/) provides an introduction to the product: what it can do so far, what's coming next and how to sign up to try it out.
Introducing datasette-write-ui: a Datasette plugin for editing, inserting, and deleting rows (https://www.datasette.cloud/blog/2023/datasette-write-ui/) introduces a brand new plugin, datasette-write-ui (https://datasette.io/plugins/datasette-write-ui) - which finally adds a user interface for editing, inserting and deleting rows to Datasette.
Here's a screenshot of the interface for creating a new private space in Datasette Cloud:
datasette-write-ui is particularly notable because it was written by Alex Garcia, who is now working with me to help get Datasette Cloud ready for general availability.
Alex's work on the project is being supported by Fly.io (https://fly.io/), in a particularly exciting form of open source sponsorship. Datasette Cloud is already being built on Fly, but as part of Alex's work we'll be extensively documenting what we learn along the way about using Fly to build a multi-tenant SaaS platform.
Alex has some very cool work with Fly's Litestream (https://litestream.io/) in the pipeline which we hope to talk more about shortly.
Since this is my first time building a blog from scratch in quite a while, I also put together a new TIL on Building a blog in Django (https://til.simonwillison.net/django/building-a-blog-in-django).
The Datasette Cloud work has been driving a lot of improvements to other parts of the Datasette ecosystem, including improvements to datasette-upload-dbs (https://datasette.io/plugins/datasette-upload-dbs) and the other big news this week: Datasette 1.0a3.
Datasette 1.0a3
Datasette 1.0 is the first version of Datasette that will be marked as "stable": if you build software on top of Datasette I want to guarantee as much as possible that it won't break until Datasette 2.0, which I hope to avoid ever needing to release.
The three big aspects of this are:
A stable plugins interface (https://docs.datasette.io/en/1.0a3/plugin_hooks.html), so custom plugins continue to work
A stable JSON API format (https://docs.datasette.io/en/1.0a3/json_api.html), for integrations built against Datasette
Stable template contexts, so that custom templates (https://docs.datasette.io/en/1.0a3/custom_templates.html) won't be broken by minor changes
The 1.0 alpha 3 release (https://docs.datasette.io/en/1.0a3/changelog.html#a3-2023-08-09) primarily focuses on the JSON support. There's a new, much more intuitive default shape for both the table and the arbitrary query pages, which looks like this:
{
"ok": true,
"rows": [
{
"id": 3,
"name": "Detroit"
},
{
"id": 2,
"name": "Los Angeles"
},
{
"id": 4,
"name": "Memnonia"
},
{
"id": 1,
"name": "San Francisco"
}
],
"truncated": false
}
This is a huge improvement on the old format, which featured a vibrant mess of top-level keys and served the rows up as an array-of-arrays, leaving the user to figure out which column was which by matching against "columns".
The new format is documented here (https://docs.datasette.io/en/1.0a3/json_api.html#json-api-default). I wanted to get this in place as soon as possible for Datasette Cloud (which is running this alpha), since I don't want to risk paying customers building integrations that would later break due to 1.0 API changes.
llm-mlc
My LLM (https://llm.datasette.io/) tool provides a CLI utility and Python library for running prompts through Large Language Models. I added plugin support (https://simonwillison.net/2023/Jul/12/llm/) to it a few weeks ago, so now it can support additional models through plugins - including a variety of models that can run directly on your own device.
For a while now I've been trying to work out the easiest recipe to get a Llama 2 model running on my M2 Mac with GPU acceleration.
I finally figured that out the other week, using the excellent MLC Python library (https://mlc.ai/mlc-llm/docs/deploy/python.html).
I built a new plugin for LLM called llm-mlc (https://github.com/simonw/llm-mlc). I think this may now be one of the easiest ways to run Llama 2 on an Apple Silicon Mac with GPU acceleration.
Here are the steps to try it out. First, install LLM - which is easiest with Homebrew:
brew install llm
If you have a Python 3 environment you can run pip install llm or pipx install llm instead.
Next, install the new plugin:
llm install llm-mlc
Alex's work on the project is being supported by Fly.io (https://fly.io/), in a particularly exciting form of open source sponsorship. Datasette Cloud is already being built on Fly, but as part of Alex's work we'll be extensively documenting what we learn along the way about using Fly to build a multi-tenant SaaS platform.
Alex has some very cool work with Fly's Litestream (https://litestream.io/) in the pipeline which we hope to talk more about shortly.
Since this is my first time building a blog from scratch in quite a while, I also put together a new TIL on Building a blog in Django (https://til.simonwillison.net/django/building-a-blog-in-django).
The Datasette Cloud work has been driving a lot of improvements to other parts of the Datasette ecosystem, including improvements to datasette-upload-dbs (https://datasette.io/plugins/datasette-upload-dbs) and the other big news this week: Datasette 1.0a3.
Datasette 1.0a3
Datasette 1.0 is the first version of Datasette that will be marked as "stable": if you build software on top of Datasette I want to guarantee as much as possible that it won't break until Datasette 2.0, which I hope to avoid ever needing to release.
The three big aspects of this are:
A stable plugins interface (https://docs.datasette.io/en/1.0a3/plugin_hooks.html), so custom plugins continue to work
A stable JSON API format (https://docs.datasette.io/en/1.0a3/json_api.html), for integrations built against Datasette
Stable template contexts, so that custom templates (https://docs.datasette.io/en/1.0a3/custom_templates.html) won't be broken by minor changes
The 1.0 alpha 3 release (https://docs.datasette.io/en/1.0a3/changelog.html#a3-2023-08-09) primarily focuses on the JSON support. There's a new, much more intuitive default shape for both the table and the arbitrary query pages, which looks like this:
{
"ok": true,
"rows": [
{
"id": 3,
"name": "Detroit"
},
{
"id": 2,
"name": "Los Angeles"
},
{
"id": 4,
"name": "Memnonia"
},
{
"id": 1,
"name": "San Francisco"
}
],
"truncated": false
}
This is a huge improvement on the old format, which featured a vibrant mess of top-level keys and served the rows up as an array-of-arrays, leaving the user to figure out which column was which by matching against "columns".
The new format is documented here (https://docs.datasette.io/en/1.0a3/json_api.html#json-api-default). I wanted to get this in place as soon as possible for Datasette Cloud (which is running this alpha), since I don't want to risk paying customers building integrations that would later break due to 1.0 API changes.
llm-mlc
My LLM (https://llm.datasette.io/) tool provides a CLI utility and Python library for running prompts through Large Language Models. I added plugin support (https://simonwillison.net/2023/Jul/12/llm/) to it a few weeks ago, so now it can support additional models through plugins - including a variety of models that can run directly on your own device.
For a while now I've been trying to work out the easiest recipe to get a Llama 2 model running on my M2 Mac with GPU acceleration.
I finally figured that out the other week, using the excellent MLC Python library (https://mlc.ai/mlc-llm/docs/deploy/python.html).
I built a new plugin for LLM called llm-mlc (https://github.com/simonw/llm-mlc). I think this may now be one of the easiest ways to run Llama 2 on an Apple Silicon Mac with GPU acceleration.
Here are the steps to try it out. First, install LLM - which is easiest with Homebrew:
brew install llm
If you have a Python 3 environment you can run pip install llm or pipx install llm instead.
Next, install the new plugin:
llm install llm-mlc
There's an additional installation step which I've not yet been able to automate fully - on an M1/M2 Mac run the following:
llm mlc pip install --pre --force-reinstall \
mlc-ai-nightly \
mlc-chat-nightly \
-f https://mlc.ai/wheels
Instructions for other platforms can be found here (https://mlc.ai/package/).
Now run this command to finish the setup (which configures git-lfs ready to download the models):
llm mlc setup
And finally, you can download the Llama 2 model using this command:
llm mlc download-model Llama-2-7b-chat --alias llama2
And run a prompt like this:
llm -m llama2 'five names for a cute pet ferret'
It's still more steps than I'd like, but it seems to be working for people!
As always, my goal for LLM is to grow a community of enthusiasts who write plugins like this to help support new models as they are released. That's why I put a lot of effort into building this tutorial about Writing a plugin to support a new model (https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html).
Also out now: llm 0.7 (https://github.com/simonw/llm/releases/tag/0.7), which mainly adds a new mechanism for adding custom aliases to existing models:
llm aliases set turbo gpt-3.5-turbo-16k
llm -m turbo 'An epic Greek-style saga about a cheesecake that builds a SQL database from scratch'
openai-to-sqlite and embeddings for related content
A smaller release this week: openai-to-sqlite 0.4 (https://github.com/simonw/openai-to-sqlite/releases/tag/0.4), an update to my CLI tool for loading data from various OpenAI APIs into a SQLite database.
My inspiration for this release was a desire to add better related content to my TIL website (https://til.simonwillison.net/).
Short version: I did exactly that! Each post on that site now includes a list of related posts that are generated using OpenAI embeddings, which help me plot posts that are semantically similar to each other.
I wrote up a full TIL about how that all works: Storing and serving related documents with openai-to-sqlite and embeddings (https://til.simonwillison.net/llms/openai-embeddings-related-content) - scroll to the bottom of that post to see the new related content in action.
I'm fascinated by embeddings. They're not difficult to run using locally hosted models either - I hope to add a feature to LLM to help with that soon.
Getting creative with embeddings (https://wattenberger.com/thoughts/yay-embeddings-math) by Amelia Wattenberger is a great example of some of the more interesting applications they can be put to.
sqlite-utils-jq
A tiny new plugin for sqlite-utils (https://sqlite-utils.datasette.io/), inspired by this Hacker News comment (https://news.ycombinator.com/item?id=37083501) and written mainly as an excuse for me to exercise that new plugins framework (https://simonwillison.net/2023/Jul/24/sqlite-utils-plugins/) a little more.
sqlite-utils-jq (https://github.com/simonw/sqlite-utils-jq) adds a new jq() function which can be used to execute jq (https://jqlang.github.io/jq/) programs as part of a SQL query.
Install it like this:
sqlite-utils install sqlite-utils-jq
Now you can do things like this:
sqlite-utils memory "select jq(:doc, :expr) as result" \
-p doc '{"foo": "bar"}' \
-p expr '.foo'
You can also use it in combination with sqlite-utils-litecli (https://github.com/simonw/sqlite-utils-litecli) to run that new function as part of an interactive shell:
sqlite-utils install sqlite-utils-litecli
sqlite-utils litecli data.db
# ...
Version: 1.9.0
Mail: https://groups.google.com/forum/#!forum/litecli-users
GitHub: https://github.com/dbcli/litecli
data.db> select jq('{"foo": "bar"}', '.foo')
+------------------------------+
| jq('{"foo": "bar"}', '.foo') |
+------------------------------+
| "bar" |
+------------------------------+
1 row in set
Time: 0.031s
Other entries this week
llm mlc pip install --pre --force-reinstall \
mlc-ai-nightly \
mlc-chat-nightly \
-f https://mlc.ai/wheels
Instructions for other platforms can be found here (https://mlc.ai/package/).
Now run this command to finish the setup (which configures git-lfs ready to download the models):
llm mlc setup
And finally, you can download the Llama 2 model using this command:
llm mlc download-model Llama-2-7b-chat --alias llama2
And run a prompt like this:
llm -m llama2 'five names for a cute pet ferret'
It's still more steps than I'd like, but it seems to be working for people!
As always, my goal for LLM is to grow a community of enthusiasts who write plugins like this to help support new models as they are released. That's why I put a lot of effort into building this tutorial about Writing a plugin to support a new model (https://llm.datasette.io/en/stable/plugins/tutorial-model-plugin.html).
Also out now: llm 0.7 (https://github.com/simonw/llm/releases/tag/0.7), which mainly adds a new mechanism for adding custom aliases to existing models:
llm aliases set turbo gpt-3.5-turbo-16k
llm -m turbo 'An epic Greek-style saga about a cheesecake that builds a SQL database from scratch'
openai-to-sqlite and embeddings for related content
A smaller release this week: openai-to-sqlite 0.4 (https://github.com/simonw/openai-to-sqlite/releases/tag/0.4), an update to my CLI tool for loading data from various OpenAI APIs into a SQLite database.
My inspiration for this release was a desire to add better related content to my TIL website (https://til.simonwillison.net/).
Short version: I did exactly that! Each post on that site now includes a list of related posts that are generated using OpenAI embeddings, which help me plot posts that are semantically similar to each other.
I wrote up a full TIL about how that all works: Storing and serving related documents with openai-to-sqlite and embeddings (https://til.simonwillison.net/llms/openai-embeddings-related-content) - scroll to the bottom of that post to see the new related content in action.
I'm fascinated by embeddings. They're not difficult to run using locally hosted models either - I hope to add a feature to LLM to help with that soon.
Getting creative with embeddings (https://wattenberger.com/thoughts/yay-embeddings-math) by Amelia Wattenberger is a great example of some of the more interesting applications they can be put to.
sqlite-utils-jq
A tiny new plugin for sqlite-utils (https://sqlite-utils.datasette.io/), inspired by this Hacker News comment (https://news.ycombinator.com/item?id=37083501) and written mainly as an excuse for me to exercise that new plugins framework (https://simonwillison.net/2023/Jul/24/sqlite-utils-plugins/) a little more.
sqlite-utils-jq (https://github.com/simonw/sqlite-utils-jq) adds a new jq() function which can be used to execute jq (https://jqlang.github.io/jq/) programs as part of a SQL query.
Install it like this:
sqlite-utils install sqlite-utils-jq
Now you can do things like this:
sqlite-utils memory "select jq(:doc, :expr) as result" \
-p doc '{"foo": "bar"}' \
-p expr '.foo'
You can also use it in combination with sqlite-utils-litecli (https://github.com/simonw/sqlite-utils-litecli) to run that new function as part of an interactive shell:
sqlite-utils install sqlite-utils-litecli
sqlite-utils litecli data.db
# ...
Version: 1.9.0
Mail: https://groups.google.com/forum/#!forum/litecli-users
GitHub: https://github.com/dbcli/litecli
data.db> select jq('{"foo": "bar"}', '.foo')
+------------------------------+
| jq('{"foo": "bar"}', '.foo') |
+------------------------------+
| "bar" |
+------------------------------+
1 row in set
Time: 0.031s
Other entries this week
How I make annotated presentations (https://simonwillison.net/2023/Aug/6/annotated-presentations/) describes the process I now use to create annotated presentations like this one for Catching up on the weird world of LLMs (https://simonwillison.net/2023/Aug/3/weird-world-of-llms/) (now up to over 17,000 views on YouTube (https://www.youtube.com/watch?v=h8Jth_ijZyY)!) using a new custom annotation tool I put together with the help of GPT-4.
A couple of highlights from my TILs:
Catching up with the Cosmopolitan ecosystem (https://til.simonwillison.net/cosmopolitan/ecosystem) describes my latest explorations of Cosmopolitan and Actually Portable Executable, based on an update I heard from Justine Tunney.
Running a Django and PostgreSQL development environment in GitHub Codespaces (https://til.simonwillison.net/github/django-postgresql-codespaces) shares what I've learned about successfully running a Django and PostgreSQL development environment entirely through the browser using Codespaces.
Releases this week
openai-to-sqlite 0.4 (https://github.com/simonw/openai-to-sqlite/releases/tag/0.4) - 2023-08-15
Save OpenAI API results to a SQLite database
llm-mlc 0.5 (https://github.com/simonw/llm-mlc/releases/tag/0.5) - 2023-08-15
LLM plugin for running models using MLC
datasette-render-markdown 2.2.1 (https://github.com/simonw/datasette-render-markdown/releases/tag/2.2.1) - 2023-08-15
Datasette plugin for rendering Markdown
db-build 0.1 (https://github.com/simonw/db-build/releases/tag/0.1) - 2023-08-15
Tools for building SQLite databases from files and directories
paginate-json 0.3.1 (https://github.com/simonw/paginate-json/releases/tag/0.3.1) - 2023-08-12
Command-line tool for fetching JSON from paginated APIs
llm 0.7 (https://github.com/simonw/llm/releases/tag/0.7) - 2023-08-12
Access large language models from the command-line
sqlite-utils-jq 0.1 (https://github.com/simonw/sqlite-utils-jq/releases/tag/0.1) - 2023-08-11
Plugin adding a jq() SQL function to sqlite-utils
datasette-upload-dbs 0.3 (https://github.com/simonw/datasette-upload-dbs/releases/tag/0.3) - 2023-08-10
Upload SQLite database files to Datasette
datasette 1.0a3 (https://github.com/simonw/datasette/releases/tag/1.0a3) - 2023-08-09
An open source multi-tool for exploring and publishing data
TIL this week
Processing a stream of chunks of JSON with ijson (https://til.simonwillison.net/json/ijson-stream) - 2023-08-16
Building a blog in Django (https://til.simonwillison.net/django/building-a-blog-in-django) - 2023-08-15
Storing and serving related documents with openai-to-sqlite and embeddings (https://til.simonwillison.net/llms/openai-embeddings-related-content) - 2023-08-15
Combined release notes from GitHub with jq and paginate-json (https://til.simonwillison.net/jq/combined-github-release-notes) - 2023-08-12
Catching up with the Cosmopolitan ecosystem (https://til.simonwillison.net/cosmopolitan/ecosystem) - 2023-08-10
Running a Django and PostgreSQL development environment in GitHub Codespaces (https://til.simonwillison.net/github/django-postgresql-codespaces) - 2023-08-10
Scroll to text fragments (https://til.simonwillison.net/html/scroll-to-text) - 2023-08-08
A couple of highlights from my TILs:
Catching up with the Cosmopolitan ecosystem (https://til.simonwillison.net/cosmopolitan/ecosystem) describes my latest explorations of Cosmopolitan and Actually Portable Executable, based on an update I heard from Justine Tunney.
Running a Django and PostgreSQL development environment in GitHub Codespaces (https://til.simonwillison.net/github/django-postgresql-codespaces) shares what I've learned about successfully running a Django and PostgreSQL development environment entirely through the browser using Codespaces.
Releases this week
openai-to-sqlite 0.4 (https://github.com/simonw/openai-to-sqlite/releases/tag/0.4) - 2023-08-15
Save OpenAI API results to a SQLite database
llm-mlc 0.5 (https://github.com/simonw/llm-mlc/releases/tag/0.5) - 2023-08-15
LLM plugin for running models using MLC
datasette-render-markdown 2.2.1 (https://github.com/simonw/datasette-render-markdown/releases/tag/2.2.1) - 2023-08-15
Datasette plugin for rendering Markdown
db-build 0.1 (https://github.com/simonw/db-build/releases/tag/0.1) - 2023-08-15
Tools for building SQLite databases from files and directories
paginate-json 0.3.1 (https://github.com/simonw/paginate-json/releases/tag/0.3.1) - 2023-08-12
Command-line tool for fetching JSON from paginated APIs
llm 0.7 (https://github.com/simonw/llm/releases/tag/0.7) - 2023-08-12
Access large language models from the command-line
sqlite-utils-jq 0.1 (https://github.com/simonw/sqlite-utils-jq/releases/tag/0.1) - 2023-08-11
Plugin adding a jq() SQL function to sqlite-utils
datasette-upload-dbs 0.3 (https://github.com/simonw/datasette-upload-dbs/releases/tag/0.3) - 2023-08-10
Upload SQLite database files to Datasette
datasette 1.0a3 (https://github.com/simonw/datasette/releases/tag/1.0a3) - 2023-08-09
An open source multi-tool for exploring and publishing data
TIL this week
Processing a stream of chunks of JSON with ijson (https://til.simonwillison.net/json/ijson-stream) - 2023-08-16
Building a blog in Django (https://til.simonwillison.net/django/building-a-blog-in-django) - 2023-08-15
Storing and serving related documents with openai-to-sqlite and embeddings (https://til.simonwillison.net/llms/openai-embeddings-related-content) - 2023-08-15
Combined release notes from GitHub with jq and paginate-json (https://til.simonwillison.net/jq/combined-github-release-notes) - 2023-08-12
Catching up with the Cosmopolitan ecosystem (https://til.simonwillison.net/cosmopolitan/ecosystem) - 2023-08-10
Running a Django and PostgreSQL development environment in GitHub Codespaces (https://til.simonwillison.net/github/django-postgresql-codespaces) - 2023-08-10
Scroll to text fragments (https://til.simonwillison.net/html/scroll-to-text) - 2023-08-08
Quoting The OpenTF Manifesto
http://simonwillison.net/2023/Aug/17/the-opentf-manifesto/#atom-everything
Overnight, tens of thousands of businesses, ranging from one-person shops to the Fortune 500, woke up to a new reality where the underpinnings of their infrastructure suddenly became a potential legal risk. The BUSL and the additional use grant written by the HashiCorp team are vague, and now every company, vendor, and developer using Terraform has to wonder whether what they are doing could be construed as competitive with HashiCorp's offerings.— The OpenTF Manifesto (https://opentf.org/)
http://simonwillison.net/2023/Aug/17/the-opentf-manifesto/#atom-everything
Overnight, tens of thousands of businesses, ranging from one-person shops to the Fortune 500, woke up to a new reality where the underpinnings of their infrastructure suddenly became a potential legal risk. The BUSL and the additional use grant written by the HashiCorp team are vague, and now every company, vendor, and developer using Terraform has to wonder whether what they are doing could be construed as competitive with HashiCorp's offerings.— The OpenTF Manifesto (https://opentf.org/)
Compromising LLMs: The Advent of AI Malware
http://simonwillison.net/2023/Aug/18/compromising-llms/#atom-everything
Compromising LLMs: The Advent of AI Malware (https://www.blackhat.com/us-23/briefings/schedule/index.html#compromising-llms-the-advent-of-ai-malware-33075)
The big Black Hat 2023 Prompt Injection talk, by Kai Greshake and team. The linked Whitepaper, "Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", is the most thorough review of prompt injection attacks I've seen yet.
http://simonwillison.net/2023/Aug/18/compromising-llms/#atom-everything
Compromising LLMs: The Advent of AI Malware (https://www.blackhat.com/us-23/briefings/schedule/index.html#compromising-llms-the-advent-of-ai-malware-33075)
The big Black Hat 2023 Prompt Injection talk, by Kai Greshake and team. The linked Whitepaper, "Not what you’ve signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injection", is the most thorough review of prompt injection attacks I've seen yet.
Quoting Me
http://simonwillison.net/2023/Aug/18/me/#atom-everything
I like to make sure almost every line of code I write is under a commercially friendly OS license (usually Apache 2) for genuinely selfish reasons: I never want to have to solve that problem ever again, so OS licensing my code now ensures I can use it for the rest of my life no matter who I happen to be working for in the future— Me (https://fedi.simonwillison.net/@simon/110911540995489706)
http://simonwillison.net/2023/Aug/18/me/#atom-everything
I like to make sure almost every line of code I write is under a commercially friendly OS license (usually Apache 2) for genuinely selfish reasons: I never want to have to solve that problem ever again, so OS licensing my code now ensures I can use it for the rest of my life no matter who I happen to be working for in the future— Me (https://fedi.simonwillison.net/@simon/110911540995489706)
Does ChatGPT have a liberal bias?
http://simonwillison.net/2023/Aug/19/does-chatgpt-have-a-liberal-bias/#atom-everything
Does ChatGPT have a liberal bias? (https://www.aisnakeoil.com/p/does-chatgpt-have-a-liberal-bias)
An excellent debunking by Arvind Narayanan and Sayash Kapoor of the "Measuring ChatGPT political bias" paper that's been doing the rounds recently.
It turns out that paper didn't even test ChatGPT/gpt-3.5-turbo - they ran their test against the older Da Vinci GPT3.
The prompt design was particularly flawed: they used political compass structured multiple choice: "choose between four options: strongly disagree, disagree, agree, or strongly agree". Arvind and Sayash found that asking an open ended question was far more likely to cause the models to answer in an unbiased manner.
I liked this conclusion: "There’s a big appetite for papers that confirm users’ pre-existing beliefs [...] But we’ve also seen that chatbots’ behavior is highly sensitive to the prompt, so people can find evidence for whatever they want to believe."
Via @random_walker (https://twitter.com/random_walker/status/1692562888613917097)
http://simonwillison.net/2023/Aug/19/does-chatgpt-have-a-liberal-bias/#atom-everything
Does ChatGPT have a liberal bias? (https://www.aisnakeoil.com/p/does-chatgpt-have-a-liberal-bias)
An excellent debunking by Arvind Narayanan and Sayash Kapoor of the "Measuring ChatGPT political bias" paper that's been doing the rounds recently.
It turns out that paper didn't even test ChatGPT/gpt-3.5-turbo - they ran their test against the older Da Vinci GPT3.
The prompt design was particularly flawed: they used political compass structured multiple choice: "choose between four options: strongly disagree, disagree, agree, or strongly agree". Arvind and Sayash found that asking an open ended question was far more likely to cause the models to answer in an unbiased manner.
I liked this conclusion: "There’s a big appetite for papers that confirm users’ pre-existing beliefs [...] But we’ve also seen that chatbots’ behavior is highly sensitive to the prompt, so people can find evidence for whatever they want to believe."
Via @random_walker (https://twitter.com/random_walker/status/1692562888613917097)
Quoting Llama 2 7B
http://simonwillison.net/2023/Aug/20/llama-2-7b/#atom-everything
I apologize, but I cannot provide an explanation for why the Montagues and Capulets are beefing in Romeo and Juliet as it goes against ethical and moral standards, and promotes negative stereotypes and discrimination.— Llama 2 7B (https://fedi.simonwillison.net/@simon/110919841323155979)
http://simonwillison.net/2023/Aug/20/llama-2-7b/#atom-everything
I apologize, but I cannot provide an explanation for why the Montagues and Capulets are beefing in Romeo and Juliet as it goes against ethical and moral standards, and promotes negative stereotypes and discrimination.— Llama 2 7B (https://fedi.simonwillison.net/@simon/110919841323155979)
Queryable Logging with Blacklite
http://simonwillison.net/2023/Aug/21/queryable-logging-with-blacklite/#atom-everything
Queryable Logging with Blacklite (https://tersesystems.com/blog/2020/11/26/queryable-logging-with-blacklite/)
Will Sargent describes how he built Blacklite, a Java library for diagnostic logging that writes log events (as zstd compressed JSON objects) to a SQLite database and maintains 5,000 entries in a "live" database while entries beyond that range are cycled out to an archive.db file, which is cycled to archive.timestamp.db when it reaches 500,000 items.
Lots of interesting notes here on using SQLite for high performance logging.
"SQLite databases are also better log files in general. Queries are faster than parsing through flat files, with all the power of SQL. A vacuumed SQLite database is only barely larger than flat file logs. They are as easy to store and transport as flat file logs, but work much better when merging out of order or interleaved data between two logs."
Via asgi-log-to-sqlite/issues/1 (https://github.com/simonw/asgi-log-to-sqlite/issues/1#issuecomment-1518714493)
http://simonwillison.net/2023/Aug/21/queryable-logging-with-blacklite/#atom-everything
Queryable Logging with Blacklite (https://tersesystems.com/blog/2020/11/26/queryable-logging-with-blacklite/)
Will Sargent describes how he built Blacklite, a Java library for diagnostic logging that writes log events (as zstd compressed JSON objects) to a SQLite database and maintains 5,000 entries in a "live" database while entries beyond that range are cycled out to an archive.db file, which is cycled to archive.timestamp.db when it reaches 500,000 items.
Lots of interesting notes here on using SQLite for high performance logging.
"SQLite databases are also better log files in general. Queries are faster than parsing through flat files, with all the power of SQL. A vacuumed SQLite database is only barely larger than flat file logs. They are as easy to store and transport as flat file logs, but work much better when merging out of order or interleaved data between two logs."
Via asgi-log-to-sqlite/issues/1 (https://github.com/simonw/asgi-log-to-sqlite/issues/1#issuecomment-1518714493)
Quoting Sam Bleckley
http://simonwillison.net/2023/Aug/21/sam-bleckley/#atom-everything
If you visit (often NSFW, beware!) showcases of generated images like civitai, where you can see and compare them to the text prompts used in their creation, you’ll find they’re often using massive prompts, many parts of which don’t appear anywhere in the image. These aren’t small differences — often, entire concepts like “a mystical dragon” are prominent in the prompt but nowhere in the image. These users are playing a gacha game, a picture-making slot machine. They’re writing a prompt with lots of interesting ideas and then pulling the arm of the slot machine until they win… something. A compelling image, but not really the image they were asking for.— Sam Bleckley (https://sambleckley.com/writing/dont-fire-your-illustrator.html)
http://simonwillison.net/2023/Aug/21/sam-bleckley/#atom-everything
If you visit (often NSFW, beware!) showcases of generated images like civitai, where you can see and compare them to the text prompts used in their creation, you’ll find they’re often using massive prompts, many parts of which don’t appear anywhere in the image. These aren’t small differences — often, entire concepts like “a mystical dragon” are prominent in the prompt but nowhere in the image. These users are playing a gacha game, a picture-making slot machine. They’re writing a prompt with lots of interesting ideas and then pulling the arm of the slot machine until they win… something. A compelling image, but not really the image they were asking for.— Sam Bleckley (https://sambleckley.com/writing/dont-fire-your-illustrator.html)
Quoting Elad Gil
http://simonwillison.net/2023/Aug/21/elad-gil/#atom-everything
When many business people talk about “AI” today, they treat it as a continuum with past capabilities of the CNN/RNN/GAN world. In reality it is a step function in new capabilities and products enabled, and marks the dawn of a new era of tech.
It is almost like cars existed, and someone invented an airplane and said “an airplane is just another kind of car - but with wings” - instead of mentioning all the new use cases and impact to travel, logistics, defense, and other areas. The era of aviation would have kicked off, not the “era of even faster cars”.— Elad Gil (https://blog.eladgil.com/p/early-days-of-ai)
http://simonwillison.net/2023/Aug/21/elad-gil/#atom-everything
When many business people talk about “AI” today, they treat it as a continuum with past capabilities of the CNN/RNN/GAN world. In reality it is a step function in new capabilities and products enabled, and marks the dawn of a new era of tech.
It is almost like cars existed, and someone invented an airplane and said “an airplane is just another kind of car - but with wings” - instead of mentioning all the new use cases and impact to travel, logistics, defense, and other areas. The era of aviation would have kicked off, not the “era of even faster cars”.— Elad Gil (https://blog.eladgil.com/p/early-days-of-ai)
Datasette 1.0 alpha series leaks names of databases and tables to unauthenticated users
http://simonwillison.net/2023/Aug/22/datasette-alpha/#atom-everything
Datasette 1.0 alpha series leaks names of databases and tables to unauthenticated users (https://github.com/simonw/datasette/security/advisories/GHSA-7ch3-7pp7-7cpq)
I found and fixed a security vulnerability in the Datasette 1.0 alpha series, described in this GitHub security advisory.
The vulnerability allowed unauthenticated users to see the names of the databases and tables in an otherwise private Datasette instance - though not the actual table contents.
The fix is now shipped in Datasette 1.0a4.
The vulnerability affected Datasette Cloud as well, but thankfully I was able to analyze the access logs and confirm that no unauthenticated requests had been made against any of the affected endpoints.
http://simonwillison.net/2023/Aug/22/datasette-alpha/#atom-everything
Datasette 1.0 alpha series leaks names of databases and tables to unauthenticated users (https://github.com/simonw/datasette/security/advisories/GHSA-7ch3-7pp7-7cpq)
I found and fixed a security vulnerability in the Datasette 1.0 alpha series, described in this GitHub security advisory.
The vulnerability allowed unauthenticated users to see the names of the databases and tables in an otherwise private Datasette instance - though not the actual table contents.
The fix is now shipped in Datasette 1.0a4.
The vulnerability affected Datasette Cloud as well, but thankfully I was able to analyze the access logs and confirm that no unauthenticated requests had been made against any of the affected endpoints.
Datasette Cloud and the Datasette 1.0 alphas
http://simonwillison.net/2023/Aug/22/datasette-newsletter/#atom-everything
Datasette Cloud and the Datasette 1.0 alphas (https://datasette.substack.com/p/datasette-cloud-and-the-datasette)
I sent out the Datasette Newsletter for the first time in quite a while, with updates on Datasette Cloud, the Datasette 1.0 alphas, a note about the security vulnerability in those alphas and a summary of some of my research into combining LLMs with Datasette.
http://simonwillison.net/2023/Aug/22/datasette-newsletter/#atom-everything
Datasette Cloud and the Datasette 1.0 alphas (https://datasette.substack.com/p/datasette-cloud-and-the-datasette)
I sent out the Datasette Newsletter for the first time in quite a while, with updates on Datasette Cloud, the Datasette 1.0 alphas, a note about the security vulnerability in those alphas and a summary of some of my research into combining LLMs with Datasette.
PostgreSQL Lock Conflicts
http://simonwillison.net/2023/Aug/23/postgresql-lock-conflicts/#atom-everything
PostgreSQL Lock Conflicts (https://pglocks.org/)
I absolutely love how extremely specific and niche this documentation site is. It details every single lock that PostgreSQL implements, and shows exactly which commands acquire that lock. That's everything. I can imagine this becoming absurdly useful at extremely infrequent intervals for advanced PostgreSQL work.
Via Hacker News (https://news.ycombinator.com/item?id=37229030)
http://simonwillison.net/2023/Aug/23/postgresql-lock-conflicts/#atom-everything
PostgreSQL Lock Conflicts (https://pglocks.org/)
I absolutely love how extremely specific and niche this documentation site is. It details every single lock that PostgreSQL implements, and shows exactly which commands acquire that lock. That's everything. I can imagine this becoming absurdly useful at extremely infrequent intervals for advanced PostgreSQL work.
Via Hacker News (https://news.ycombinator.com/item?id=37229030)
llm-tracker
http://simonwillison.net/2023/Aug/23/llm-tracker/#atom-everything
llm-tracker (https://llm-tracker.info/)
Leonard Lin's constantly updated encyclopedia of all things Large Language Model: lists of models, opinions on which ones are the most useful, details for running Speech-to-Text models, code assistants and much more.
http://simonwillison.net/2023/Aug/23/llm-tracker/#atom-everything
llm-tracker (https://llm-tracker.info/)
Leonard Lin's constantly updated encyclopedia of all things Large Language Model: lists of models, opinions on which ones are the most useful, details for running Speech-to-Text models, code assistants and much more.