Prompt Engineering Resource Guide

1. Exploration
   1. General Guides
   2. Prompt examples
2. Evaluation
   1. Datasets
   2. OpenAI evals
3. Beyond Prompt Engineering

"The hottest new programming language is English" - Andrej Karpathy, 24 Jan 2023

Prompt engineering is about skillfully creating input queries (prompts) to communicate with AI models like ChatGPT effectively. Think of it as writing instructions for a highly capable yet sometimes unpredictably dumb personal assistant.

This guide serves as a hands-on resource for developers and early adopters using large language models (LLMs). It goes beyond the usual one-off task prompts, focusing instead on processing large quantities of inputs via an API. When manual review of every output isn't feasible, it's critical to evaluate and manage the trade-offs between cost, speed, and output quality. Therefore we emphasize the 'engineering' part of prompt engineering here.

Our aim with this guide is to organize links to key external resources, and give concise commentary to help you find what's relevant for your task.

If you want to contribute to this guide, please open an issue, send a PR, or email me at prompts@matthiasberth.com.

<a id="exploration"></a>

Exploration

In the exploration phase of prompt engineering, the focus is on generating a range of candidate prompts that perform effectively on example inputs. This phase involves using a playground environment to experiment with various combinations of instructions, examples, and inputs, allowing for the identification and resolution of issues. Rapid iteration and drawing inspiration from existing prompts in the wild are key strategies during this phase.

<a id="guides"></a>

General Guides

1. Prompt engineering guide from OpenAI

   The Prompt engineering guide from OpenAI covers "Six strategies for getting better results":

   • Write clear instructions
   • Provide reference text
   • Split complex tasks into simpler subtasks
   • Give the model time to "think"
   • Use external tools
   • Test changes systematically

   Each comes with a set of tactics (like "Ask the model if it missed anything on previous passes"). The guide provides direct links to the OpenAI playground where you can try out examples.

2. Microsoft Introduction to prompt engineering,

   The intro covers common techniques and best practices. The techniques article discusses Chain of Thought prompting, and the influence of the temperature parameter, among others.

3. Principled Instructions Are All You Need for Questioning LLaMA-1/2, GPT-3.5/4

   This research paper presents 26 guiding principles and evaluates their effectiveness across several models.

4. The CO-STAR framework

   Suggests to structure the prompt as Context, Objective, Style, Tone, Audience, Response. Makes a lot of sense and helped the author win a competition. I'm still trying to track down original sources to the CO-STAR framework and that competition.

<a id="prompt-examples"></a>

Prompt examples

1. OpenAI prompt examples

   Many of these are geared to everyday use, but there are relevant prompts in the categories:

   • Extract, e.g. Classify user reviews based on a set of tags.
   • Transform, e.g. Convert ungrammatical statements into standard English.

2. Prompt collections / Libraries

   • LangChain Hub collects prompts in a variety of areas, e.g. Tagging,

   Summarization, Extraction.