Pete's Log: Coding with AI

Entry #2722, (Coding, Hacking, & CS stuff)
(posted when I was 46 years old.)

I still don't know what to make of generative AI. It's a neat technology with many not-so-neat consequences.

It feels inevitable, however, that I should mess with it. So I've been playing around with a few different things lately, both for work and personal stuff. I'm just going to collectively refer to the models and wrappers around those models that I've tried so far as "it" below.

For work, so far I've mostly been using it to write PowerShell scripts for me. It consistently writes scripts that look great. However it usually takes some prodding to turn a great-looking script into a functional script. Hallucinations remain common and it's hit or miss if it pays attention to my instructions to write the script for SharePoint on-premises or if it gives me something that only works with SharePoint Online.

For personal stuff, I've enlisted its help in my ongoing rewrite of esgeroth.org. It's been useful in getting me past my current bottleneck of having a lot of tedious forms to implement. Here too I've had trouble with hallucinations.

Despite the issues, it's been helpful. The biggest thing it helps me with is ADHD struggles. Tedious tasks have a way of becoming roadblocks. AI tools let me get past those. Or—as I write this I realize there's an important caveat—AI tools are still novel enough that in the present moment they keep me moving.

With the PowerShell scripts, I've mostly had the following results:

  • Works on the first try (maybe 10% of the time)
  • Works after a few more prompts (50+% of the time)
  • Is so unhelpful that I say "F it, I'll do it myself"

Even the last outcome is useful, because the frustration motivates me to get things done.

The esgeroth.org rewrite has been the more telling experience so far, since we're working on a larger code base. My internal model of using an AI agent to help me code is that I am working with an eager young developer who memorized all of Stack Overflow, but neglected to memorize the scores on the answers.

It seems most helpful so far in the tedious tasks that slow me down, like adding forms. It is remarkably good at remembering to validate all fields and to always encode data to prevent XSS. It's just a nuisance that it keeps making up methods that don't exist or forgetting what coding style I asked it to use.

In addition to developing prompting as a skill, I'm learning my way around context windows and other such factors. I'm getting better at getting my assistants into the "sweet spot" where they know enough context to write the code the way I want it written. The faster I can get them there, the longer the useful context window lasts before they start going off the rails again.

The biggest challenge I see facing me is remaining diligent about reviewing all the code these tools give me. It gets tempting after a while to just glance at it and say "close enough" but I don't like the path that sets me on.

I'm only just scratching the surface. There's lots more to learn. Lots more to think about. Lots more to fear for any developers using these tools without understanding fundamentals.