One of the most intriguing features of ChatGPT is its ability to take a plain-English description of desired functionality and turn that into a simple script or program in a variety of programming and scripting languages, including PowerShell.

Unfortunately, it doesn’t do this well.

Take, for example, this test that I performed:

Here, I have asked ChatGPT to write a PowerShell function to perform a rather simple task, and the resulting code LOOKS reasonable at first. However, this code does not work. At all. It contains two major bugs (putting aside for a moment that the pane containing the code mislabels it as PHP).

First of all, the use of a variable named $input does not work in PowerShell. This is reserved as an Automatic Variable, such as $_, $HOME, and $error. Big no-no. This does not result in an error being thrown, but the input string does not get successfully passed to the function, resulting in no action being taken. Of course, fixing this error is simple enough. Rename the variable.

But that isn’t enough. With the variable renamed, the function runs, but throws errors:

(Note that in this screenshot the param block has been modified to accept positional parameters.)

The problem here is that the “B” in the formatting block is not a proper format type in PowerShell. The simple fix here is to replace the problematic line with the following:

$binary = [Convert]::ToString($byte,2)

With these changes in place, the script runs just fine. But the errors are so fundamental, that they really shouldn’t have been there to begin with.

Of course, ChatGPT is only as good as its training data. I have no idea how many PowerShell scripts were in the training data set, but there were clearly not enough for the AI to catch on to some of the well-documented fundamentals of the language.

Keep in mind, this was not a one-off example. I pitched about half a dozen prompts to ChatGPT to write simple PowerShell scripts and functions. Only one of them worked. The code tends to be cleanly formatted and organized but is rather consistently undermined by simple mistakes.

I’m not worried just yet about this technology taking over my job.

Update

ChatGPT does indeed learn. I gave it the same prompt again today. It generated a slightly different script, sidestepping the $input issue by using a different variable name, but it still had the issue with the “B” format specifier, so I gave it a reply pointing out the error. And it fixed it with a new version that works.

What’s more, it even took suggestions for improvements: