How I Found All The Questions I've Ever Answered With LLMs

My latest Walden's World project is the Ask A Question page.

I wanted to make the questions page for a while, but it always felt a little sad that I'd be launching it with nothing there. I suddenly wanted to make the page though when I had an idea to go through my public chat history and pull in questions from my past.

Here are my sources:

• My Walden's World Discord server: 50,000 messages
• Chat logs from my various games: 3 million messages
• Comments across my various YouTube channels: 1,300 messages

Needless to say this is not super feasible by hand, so I thought LLMs could find these questions for me.

After a week, I'm finished and super happy with my results. This is the best practical LLM application project I've done so far. Out of those sources the LLMs found around 10k question/answer conversations, which I then vibe coded a web interface to quickly scan through and then manually selected around 350 for my website. The rest of this post is a breakdown of how I did it.

1. Discord

There's an open source project (DiscordChatExporter) for exporting chat logs from Discord. You can grab your Discord user account's token from the browser and run the tool as you, but technically it's against the discord TOS. Looking online, it seems a lot of people are running it without problems on main. Thankfully though, this is my server, so I have the option to create an official Discord bot and then add it to my server. That way I can use the bot's token instead of my user token. I'm really glad I had an official option because I'm not sure I would've taken any chance to risk a ban...

After the export, I had around 100MB of discord json message output. Time to start vibe coding a script to parse through this!

I've been using VS Code with GitHub Copilot as my coding agent. Right now I'm bouncing back and forth between GPT 5 Codex and Claude Sonnet 4. For the implementation I tried out having it make separate individually verifiable Python CLI scripts for the different sections: Discord json to text then LLM prompting from that json. Finally, I generated third python script to call both for me. I thought it was a clever way to verify each part was working as I built it, but it ended up being terrible because I lost the ability to debug into CLI scripts when running it together. At least I know for next time.

The final prompt I used to find questions & answers was this:

You are analyzing a chat log to extract question-answer pairs where the user "B0sh" provides answers to questions.

Please identify:

1. Questions asked by any user in the chat
2. Answers provided by the user "B0sh" to those questions
3. The answer may be in the form of text, links, or mention of attachments

Rules:

• Only extract pairs where B0sh provides an answer
• A question-answer pair may span multiple messages, other people may talk in between the answer or questions.
• Include ALL original line text for both question and answer messages
• If B0sh's response mentions an attachment or image, include that as the answer
• Context matters - look for conversational flow to identify related Q&A
• For multi-message answers, include all B0sh messages that are part of the response
• For multi-message questions, include all messages that form the complete question

JSON Output format:

• question_lines: The exact original chat lines with usernames that form the question (may be multiple lines)
• answer_lines: The exact original chat lines with usernames where B0sh provides the answer (may be multiple lines)

In addition I used structured json outputs to enforce the json output requirements.

I let it rip on all my Discord channels which took around 15 minutes. With Gemini Flash 2.5 I felt like I was getting tons of inaccurate output. Mostly questions that I wasn't answering. I decided I'd rather just go through that garbage manually over further fine tuning the prompt. I spot checked Gemini Pro 2.5 as well, which was objectively better, but it was a huge price difference and I have a lot of tokens to burn through. Because of that, I added a validation script, which compared the AI outputs messages to the original chat log. It updated the json to flag any question that didn't have a match in the logs, so I could go through and review it if I so desired.

2. In Game Chat

Next, I have about 10 years of chat logs for my browser game. It's over 3 million messages, so I was starting to get worried about how much it would cost me. Discord was already a few dollars at 50,000 messages. That's when I had a realization.

Why analyze conversations where I'm not even participating? I'll be saving a lot of context (= cost) on messages I know won't be what I want. I'm shocked I didn't think of this the first time, but that's part of learning.

So, as part of the text extraction phase, I built conversations sections that included me. Every time I say something, it pulls the last 5 messages, then continues until I don't say anything for another 5 messages. This way conversations where I participate can have their whole context together. (Fun idea for a leetcode challenge here too) I also wanted to add batch processing this time to run multiple LLM calls in parallel.

Here's the final prompt I ended up using for the code:

Write me a simple batch LLM request python script in one file.

• It takes in an input file of a chat log.
• The chat log is <USERNAME>: <MESSAGE>
• First split the logs into sections where I'm talking. 5 messages before and after I (B0sh) talk should be in one section. This is so to not AI analyze conversations where I do not participate
• Write a prompt to extract questions and answers from the chat logs. It should only look for questions answered by "B0sh"
• Question and answer pairs should be returned in the json, with the original line of text. There may be more than one message for a single question and answer pair.
• The llm call should use the llm cli tool to keep the code simple.

Here's an example usage of the LLM command line:

llm -m openrouter/google/gemini-2.5-flash "prompt" --schema-multi 'json_prop_1,json_prop_2'

By the way, for both scripts so far I used the llm cli tool so I don't have to manage API keys in my code and lets me try different models out super easily.

My explanation of the buffer idea was insufficient, so I ended up ripping that part out and putting my own algorithm in. Somewhat poetically, this cut the message count to send to the LLM to around 50,000 as well, which is around the same message count as all of my Discord logs. However, the density of questions was extremely high as all of the messages had me involved this time. I had over 8,000 questions pulled from this part.

3. YouTube comments

This prompt worked one shot:

Write a python script in a new folder "/youtube" to pull all the comments from a youtube channel. Please note that I own the channel. Use uv over pip.

It correctly uses the YouTube API python library to loop over the videos in a channel and then loop over the comments. I happen to already have YouTube API key so configuring was fast too. This found around 1300 comments.

It was short enough that I figured I'd just go through them manually myself. Here is the prompt for the parser script:

<youtube_comments.json> write a script that takes this json and turns it into text based threads with author name and the text message. If its in a thread it should be obvious in the text only output that they are together. Don't put any other meta data other than the comment text and author

• at the end of the thread, show a sql script for the question answer like this:

INSERT INTO question_answers (question, answer, timestamp_asked, timestamp_answered, tags) VALUES ('<QUESTION>', '<ANSWER>', <QUESTION_TIME>, <ANSWER_TIME>, 'youtube');

• The timestamps need to be converted into PHP unix time()

This text output was so much clearer than the JSON which was not in thread order, so I just scrolled through it and copied over the sql queries that I liked.

4. Question Viewer

For the first two sources I still had to go through the LLM question results. So the final piece to this project is an html/javascript viewer for the LLM json output with a "SQL copy" button so I could add it into my questions database.

Here was the prompt I used:

<qa_results.json> code a simple js page to render this data so I can review it. allow the user to drag in the json

Include a "Copy SQL" button:

• It should copy an SQL insert query for this table: Table question_answers: question, answer, timestamp_asked, timestamp_answered
• The timestamps for PHP time() functions, so it should be converted into unix epoc
• Use the time of first message for asked and answered
• Remove usernames for question and answers, merge all messages into one text variable with new lines between

Notice I didn't care what the layout looked like, but I really needed my sql copy to be perfect. After a little more back and forth it was usable. It's saddening not to care about the code quality here, but I knew I would never use this again after the project concludes.

I had on the order of 10,000 questions to go through. I thought about sending it through another round of LLMs again to whittle it down further, but I just decided to parse through it all manually since I had a bunch of travelling coming up anyway. I didn't add anything to the prompt about filtering for "interesting" questions, only looking to see if a conversation was answering a question.

Talking about the final product, I know nobody cares about this silly thing as much as I do, but I love how Questions came out. I enjoyed the stroll down memory lane as much as the development.

This is personally my first project that the practical application of LLMs made possible. It's completely infeasible to review my entire public chat history for fun. It feels like a massive achievement to actually have "looked at" millions of messages and come out with some (according to me) interesting snippets to put on the site.

If you have any questions about this blog post, perhaps consider asking me a question.

Why do LLMs freak out over the seahorse emoji? →

Interesting article on why LLMs think that there's a seahorse emoji.

But unlike with 🐟, the seahorse emoji doesn't exist. The model tries to construct a "seahorse + emoji" vector just as it would for a real emoji, and on layer 72 we even get a very similar construction as with the fish emoji - " se", "horse", and the emoji prefix byte prefix:

But alas, there's no continuation to ĠðŁ corresponding to a seahorse, so the lm_head similarity score calculation maxes out with horse- or sea-animal-related emoji bytes instead, and an unintended emoji is sampled.

I hadn't realized before that the tokens generated by an LLM are fed back into itself. I guess until now I had naively thought that everything was based off the starting prompt only. When it fails to generate the seahorse emoji, it can see that the previous token it generated was not a seahorse and that instantly has an effect on the next token's output.

I tried it myself and ChatGPT completely spiraled out of control.

TIL: Cloudflare email protection →

I've been doing a bit of web scraping lately and ran across [email protected] in the HTML of the site I was downloading. Not that I had any use for their email, but I was curious what was doing the "protecting" so I dug into it. It turns out it's a program by Cloudflare to combat scrapers. If Cloudflare thinks a request is a scraping, it looks for email like strings in the response and "protects" the email by removing it from being directly in the response. I checked my domain that I have on Cloudflare and it was on for me, so I guess its a default setting.

I put the below JavaScript on my website years ago. The vast majority of visitors (99.9%+? my access log files are huge and simply there's no way that many people know me) to Walden's World and this blog is bots, and this email link has been on my contact page for 10 years so you'd think I'd be on a bunch of spam email lists by now. I get remarkably little spam which makes me think that there's a decent effectiveness to js execution.

<script type="text/javascript" >
//Is this even effective at stopping spam?
var walden = 'walden';
var world = 's.world';

document.getElementById("email").innerHTML = walden+"@"+walden+world;
document.getElementById("email").href = 'mailto:'+walden+"@"+walden+world;
</script>

Cloudflare's implementation is basically just this with extra steps. They have to ship the full email so that if it was a real user they can overwrite [email protected] with the actual email. So they encode the email and ship the decode function. What's I thought was cool about this approach is that Cloudflare can be a lot more aggressive with detecting bots, since even if a real user was accidentally caught up in this, they would still be able to view the email. However, any motivated scraper would be able to thwart this method by executing the JavaScript on the page. Or ever writing a special parser just for the Cloudflare email obfuscation format. I'll leave that as an exercise to the data harvesters.

Setting Up Personal Notifications With Pushover

I've had a pretty good personal notification system going on for like eight years now with SendGrid. I have a gmail alt account that is only for these notification emails, and I use the Sendgrid free API plan to send me notifications from my websites. Or well, I should say used to, because this month I discovered that my free plan was taken from me. How rude. I've been a (non-paying) customer for 8 years! Well no matter what, I had to find a new solution, and preferably one that didn't involve email.

The first thing I found was this blog post on selfhosting the ntfy service, which I did manage to get working with Docker in a few hours. You can download their client app, and configure it to look at your domain if you host their server. Then you get push notifications. But right at the end I got hit with the self hosting gotcha: iOS backgrounding. Ok so the docs say it won't be instant but I can wait a few minutes or so, right? But my notifications never came hours later... Something about APNS? They have a solution where you relayed the messages their server, but at this point if I had to sign up for their service anyway then what was the point of self hosting? So I reverted my branch and tried again.

Then I took another look around and found Pushover. I was only looking for self hosted at first so I passed over it initially, but it seemed to have a good reputation. It's similar in where you download their client app for push notifications, but they host the API side as well. For a $5 one time fee you get an individual license with a measly 10,000 notifications a month (I will never hit this). And I get a bit more peace of mind that with some money on the line maybe I won't have to redo this system for another 8 years.

My site is in the beloved PHP language, so the code looks like this:

$pushover_api_key = getenv('PUSHOVER_API_KEY');
$pushover_user_key = getenv('PUSHOVER_USER_KEY');

$ch = curl_init();
curl_setopt_array($ch, [
    CURLOPT_URL => "https://api.pushover.net/1/messages.json",
    CURLOPT_POST => true,
    CURLOPT_POSTFIELDS => [
        'token' => $pushover_api_key,
        'user' => $pushover_user_key,
        'title' => $subject,
        'message' => $body
    ],
    CURLOPT_RETURNTRANSFER => true,
    CURLOPT_SSL_VERIFYPEER => true,
    CURLOPT_TIMEOUT => 30
]);

$result = curl_exec($ch);
$http_code = curl_getinfo($ch, CURLINFO_HTTP_CODE);
$curl_error = curl_error($ch);
curl_close($ch);

It couldn't be more dead simple. It's just an HTTPS call, and their website walks you through making the tokens. I was done the entire change even through deploying in like 20 minutes. And something tells me long term this is going to be more stable then that ntfy Docker container self hosted solution. Let's go!

This Web Dev Guy Vibes With SwiftUI

As a web guy, my attempts to make apps have always been confined to Electron. I always looked up to native apps very highly, but had never given it a true shot. So I when had an idea for an personal audio player app thing, I decided to ditch what I know and give SwiftUI a shot. Just to be sure we're all clear: I don't know how to do this. I make websites. However, in 2025, surely coding agents have 10x productivity right? With my Github Copilot subscription I can run Claude 4 or GPT-5-Codex agents, so I've got the top of the line LLM access. Easy.

Well, here I am after a week of hacking and prompting with agents in my downtime and I do have a quite functional app now with 7 screens and almost all of my long shot wish list features in already. Considering I started from downloading Xcode, I'm confident there's no way I would have had something this feature complete this quickly without LLMs. Bototm line, this app works for me.

What I loved about this development journey though is how I got progressively exposed to SwiftUI concepts as I needed them. Or to put it another way, I did the terrible thing of ignoring all the code I didn't understand until I realized it was a problem. That was the time I dug in deeper. Obviously that's something I can get away with here because I'm the only user for this project. But at the same time I can feel the code getting further and further away from me as more and more code I don't understand is creeping in this project. And as I said it works for me. I went in from the start knowing I'd never have another user. I feel very far from a hypothetical production app. So I thought I'd do a bit of a postmorterm on what I learned and didn't learn with agent first development.

What I learned

• View: I got the best handle on creating and using Views. I really enjoyed thinking about UI problems in terms of HStack/VStack. With CSS and flexbox you'd have to set up this pattern for yourself. Just the whole concept of View was great to work with.

• Swift: While obviously I still would have so much to learn, I do feel like I picked up Swift quite quickly. Before long I was comfortable writing my own Swift pure functions without AI assistance as needed. I'm talking about all the basics here: arrays, variables, functions, typing, error handling, etc.

• LazyVGrid: This is an awesome API for doing content aware grid reflowing. I know browsers have CSS Grid and flexbox, and I know how to use them, but I really liked the API that's in SwiftUI. My app has several of these.

• #Preview: The #Preview block lets you write some code to live preview your component views while in Xcode. You can setup as many different inputs as you want to test various states and edge cases. The fact that it's just built right in was super helpful for my project. I can think of times in the past where I've gone and created test pages or design pages to test these things before, but this was a whole new level to that concept. Soon I want to investigate this for my typical React/Angular development. Storybook might be a good place to start.

• SQLite: While not specific to Mac development, this was my first project where I used SQLite to run the data layer. This ended up being a great choice. Swift has several great community libraries for SQLite integration.

What I didn't learn

• Threading and MainActor: My app has network calls and a SQLite database, and I tried to keep it simple and not care about blocking the main thread, but even I couldn't put up with that. I had 3 or 4 failed attempts where eventually the SQLite database would end up locked. Finally I prompted for "everything to be on the main thread except for network calls", which led to a successful program. In theory, it would've been nice to move that whole process to its own thread, but in practice it hasn't mattered for me.

• AVFoundation: Considering I made an app that plays audio, I very rarely had to touch the generated AVFoundation player code. Getting the player UI to look good was much harder, but the playing of an mp3 file part was not affected.

• extension: I ended up with 5 extension blocks that were LLM generated, and I never needed to learn what that is. I assumed it was like extending a prototype like in JavaScript, which makes sense just fine. But, critically, I didn't understand the LLM chose an extension and not just a regular function.

• Eventing with @State, @ObservedObject: I have a lot of state getting passed around, and every single bit of it I didn't write myself.

• Navigation: Speaking of not writing code, I also didn't write any of the navigation handling code. There's several issues with the Navigation still but it's good enough for me at this point.

Conclusion

I don't want to say I learned absolutely nothing about these "What I didn't learn" points, because I was directing the direction of the agents and making tweaks as I went. I just know I wouldn't be able to defend the logic or reasoning behind the code in those spots because I don't understand it enough. And my main focus was on the UI design, and I think that comes across in the concepts I glossed over.

The way I've always pushed my programming skill is by doing. If the doing is getting done for me, does the value of doing side projects for learning go down? I'm still thinking through this.

Bonus: Struggles

• App Icon: I got stuck for an hour setting up an app icon. I was completely positive I did it right dragging in icons into Xcode. Turns out I needed to do a Clean & Rebuild... Maybe one day computer use LLMs will be available enough to debug Xcode for me.

• iOS hallucinations: The LLMs would often try to bring in iOS only APIs that would be an instant not available in macOS build fail. In retrospect, I think an AGENTS.md file would've been helpful here to note that this was a Mac app.