openai-gpt_为什么GPT-3感觉像是编程

“What’s the next big trend in programming? Maybe it’s sloppy programming” — Guy Steele on the Future of Programming Languages

“编程的下一个大趋势是什么？ 也许是草率的编程” – Guy Steele关于编程语言的未来

I want to discuss with you today four different kinds of programming. This framework should help you better understand the uniqueness of Deep Learning as a new way of programming. I characterize Deep Learning as a technique for programming massively parallel computer systems to perform useful tasks. We will explore here why this is revolutionary and is a new kind of programming.

今天，我想与您讨论四种不同的编程。 该框架应帮助您更好地理解深度学习作为一种新的编程方式的独特性。 我将深度学习描述为一种对大型并行计算机系统进行编程以执行有用任务的技术。 我们将在这里探讨为什么这是革命性的并且是一种新型的编程。

In most introductory computer programming classes, we are introduced to the idea of imperative programming. All programming can be understood in the abstract sense as a kind of specification. Imperative programming is a specification that tells a computer the exact and detailed sequence of steps to perform. These also will include conditions to test, processes to execute and alternative paths to follow (i.e., conditions, functions, and loops). All of the more popular languages we have heard of (i.e., JavaScript, Java, Python, C etc.) are all imperative languages. When a programmer writes are program, he formulates in his mind the exact sequence of a task that needs to be composed to arrive at a solution. Von Neumann computer architectures are intrinsically capable of interpreting and executing imperative code.

在大多数入门计算机编程课程中，我们都介绍了命令式编程的思想。 所有编程都可以抽象地理解为一种规范。 命令式编程是一种规范，它告诉计算机要执行的确切步骤和详细顺序。 这些还将包括要测试的条件，要执行的过程以及要遵循的替代路径(即条件，功能和循环)。 我们听说过的所有较流行的语言(即JavaScript，Java，Python，C等)都是命令性语言。 当程序员编写程序时，他会在脑海中制定出为完成解决方案而必须组成的确切任务顺序。 冯·诺依曼计算机体系结构本质上能够解释和执行命令性代码。

Another kind of programming is known as declarative programming. This kind of programming does not burden the user with the details of the exact sequence of steps that must be performed. Rather, a user only needs to specify (or declare) the form of the final solution. The burden of figuring out the exact steps to execute to arrive at the specified solution is algorithmically discovered by the system. Spreadsheets are an example of this kind of programming. With Spreadsheets, you don’t specify how a computer should compute its results; rather you only need to specify the dependencies between the cells to compute a final conclusion. You could have a long chain of dependencies, and the spreadsheet will figure out which to calculate first. The query language SQL is also a well-known example of declarative programming. A SQL processor optimizes the kinds of retrievals it needs to execute to arrive at a table of data that satisfies the user-specified query. Other examples of declarative programming are Haskell (i.e., functional programming) and Prolog (i.e., logic programming). Code written in declarative programming languages are either interpreted by or compiled to imperative instructions. Non-constructive mathematics, of the symbolic computation kind, can also be classified as declarative programming.

另一种编程称为声明式编程。 这种编程不会使用户负担必须执行的确切步骤顺序的细节。 而是，用户仅需要指定(或声明)最终解决方案的形式。 系统通过算法发现了找出要执行的确切步骤以达到指定解决方案的负担。 电子表格就是这种编程的一个例子。 使用电子表格，您无需指定计算机应如何计算其结果。 您只需要指定单元格之间的依赖关系即可计算最终结论。 您可能有很长的依赖关系，电子表格将确定要首先计算的内容。 查询语言SQL也是声明式编程的一个著名示例。 SQL处理器可以优化其执行所需的检索类型，以得出满足用户指定查询的数据表。 声明式编程的其他示例是Haskell(即功能编程)和Prolog(即逻辑编程)。 用声明性编程语言编写的代码可以由命令性指令解释或编译为命令性指令。 符号计算类型的非构造数学也可以归类为声明性编程。

A subclass of declarative programming comes from practices that predate mechanical computers. You may have heard of the subject “linear programming,” “integer programming,” and “non-linear programming. These are optimization algorithms where the use of the word programming was synonymous to planning. Just as the word ‘computer’ used to refer to people performing calculations, similarly ‘programming’ meant something different before the introduction of mechanical computers. In general, though, constraints are specified, and a solution is discovered that satisfies the constraints. A more general form of this is known as constraint programming. Yann LeCun has proposed the term “differential programming” as a substitute for Deep Learning.

声明性编程的子类来自早于机械计算机的实践。 您可能听说过“线性编程”，“整数编程”和“非线性编程”主题。 这些是优化算法，其中编程一词的使用与计划同义。 就像“计算机”一词曾经用来指代人们进行计算一样，“编程”在引入机械计算机之前也有不同的含义。 通常，尽管指定了约束，但发现了满足约束的解决方案。 这种方法的更一般形式称为约束编程。 Yann LeCun提出了“ 差分编程 ”一词来替代深度学习。

There is a fourth kind of programming that I would christen as “recombinant programming”. Recombinant is a word that originates in genetics. The generators of organisms and the process of evolution originates from the recombination of genetic material. Thus, the Recombinant Programming is an appropriate metaphor to describe this entirely new kind of proigramming.

我将第四种编程称为“重组编程”。 重组是起源于遗传学的单词。 生物的产生者和进化过程起源于遗传物质的重组。 因此，重组编程是描述这种全新编程方式的合适隐喻。

This new kind of programming has as its origins methods in connectionist inspired artificial intelligence. In complexity sciences, dynamics is a consequence of progression. It derives from methods coming from Deep Learning, evolutionary algorithms, and reinforcement learning. Recombinant programming is best visually demonstrated by what is known as Generative Adversarial Networks. I will argue here that this should be classified differently from declarative programming or its subset constraint programming. One main distinction is the generality of the algorithm. However, it’s essential to make this distinction because it shifts the mindset from one of pursuing a reductionist approach to one that is based on ideas in the complexity sciences. The reductionist perspective is counterproductive in understanding the nature of this new kind of programming.

这种新型的编程方法源于连接主义启发的人工智能。 在复杂性科学中，动力学是进步的结果。 它源自深度学习，进化算法和强化学习的方法。 重组编程最好通过所谓的对抗生成器网络直观地展示出来。 我将在这里辩称，这应该与声明式编程或其子集约束编程不同地进行分类。 主要区别之一是算法的通用性。 但是，区分这一点非常重要，因为它将思维方式从追求还原论的方法转变为基于复杂性科学思想的方法。 还原论的观点在理解这种新型编程的本质方面适得其反。

If you think imperative programming is hard, then you would be surprised that implementing a declarative programming language is even harder. To achieve something like SQL, a programmer must know how to translate a formal specification language into a sequence of instructions that correctly satisfies the different sentences that can be expressed in the language. A computer program has to be written that understand the language and the steps to translate the language into a solution. There is a massive cognitive bar that needs to be hurdled to implement a declarative language.

如果您认为命令式编程很困难，那么实现声明式编程语言会更加困难，您会感到惊讶。 为了实现类似SQL的功能，程序员必须知道如何将形式规范语言转换为正确满足可以用该语言表达的不同语句的指令序列。 必须编写一个能够理解该语言以及将语言转换为解决方案的步骤的计算机程序。 要实施一种声明式语言，就必须克服巨大的认知障碍。

The advantage of declarative languages is that it is easier for humans to express their needs. Although many Domain Specific Languages (DSLs) have been invented, it is merely too difficult to create more of them. Even if you could implement these economically, it’s also difficult for the user to learn all the variety of DSLs.

声明性语言的优点是使人类更容易表达自己的需求。 尽管已经发明了许多领域特定语言(DSL)，但是创建它们中的更多只是太困难了。 即使您可以经济地实现这些功能，用户也很难学习所有种类的DSL。

Constraint programming, differential programming (i.e. Deep Learning) and recombinant programming share a common trait. The program or algorithm that discovers the solution is fixed. In other words, a programmer does not need to write a program that translates the specification into a solution. Unfortunately, though, the fixed program is applicable only in narrow domains. This deficiency is known as the “No Free Lunch” theorem in machine learning. You can’t use a linear programming algorithm to solve an integer programming problem. Deep Learning, however, has a unique kind of general capability that the same kind of algorithm (i.e., stochastic gradient descent) appears to apply to many problems.

约束编程，差异编程(即深度学习)和重组编程具有一个共同的特征。 发现解决方案的程序或算法是固定的。 换句话说，程序员无需编写将规范转换为解决方案的程序。 但是，不幸的是，固定程序仅适用于狭窄的领域。 这种缺陷在机器学习中被称为“无免费午餐”定理。 您不能使用线性规划算法来解决整数规划问题。 但是，深度学习具有一种独特的通用功能，即同一种算法(即，随机梯度下降)似乎适用于许多问题。

There are distinct advantages to a fixed ‘master algorithm’ that solves all kinds of declarative specifications. This may appear to be an impossibility. However, if we relax the strictness requirements and prioritize the best effort approach then perhaps we can avoid the complexity. This prioritization is in fact what evolutional biology follows. The rich diversity that we find in biology and the evolution of general intelligence in humans is a testament to the viability of this approach.

固定的“主算法”可解决各种声明性规范，具有明显的优势。 这似乎是不可能的。 但是，如果我们放松严格性要求并优先考虑尽力而为方法，那么也许我们可以避免复杂性。 实际上，这种优先次序是进化生物学遵循的。 我们在生物学中发现的丰富多样性以及人类普通情报的发展证明了这种方法的可行性。

Generative or recombinant programming is based on meta-learning algorithms. These are algorithms that are of the same fixed form, but in its construction of a solution, do not have the same level of cognitive understanding as required to construct the translations required of declarative programming language. This is what Daniel Dennett refers to as competence without comprehension. A GAN is capable of generating high fidelity realistic human faces but is unable to comprehend what it generated. It only knows if it is close to being realistic or not. Biological processes generate human beings from instructions encoded in our DNA but is unable to comprehend the process it is executing. In both cases, there is no programmer that specifies the sequence of steps required.

生成式或重组式编程基于元学习算法。 这些算法具有相同的固定形式，但是在其解决方案的构建中，它们与构建声明性编程语言所需的翻译所具有的认知理解水平不同。 这就是丹尼尔·丹内特 ( Daniel Dennett)所说的无须理解的能力。 GAN能够生成高保真逼真的人脸，但无法理解其生成的内容。 它只知道它是否接近现实。 生物过程从我们DNA编码的指令中产生人类，但无法理解其执行的过程。 在这两种情况下，都没有程序员指定所需步骤的顺序。

There exist only meta-learning algorithms that mindlessly generate solution one generative incremental action at a time.

只有元学习算法一次无意识地生成一个生成增量动作的解决方案。

Why is this classification important? It is important because it tells us about the kinds of models or reality that we can invent. One can think of the models that we create as belonging to two general classes. There are descriptive models and there are generative models. Descriptive models of reality are the mental models that we employ to describe the ‘shape’ of reality. Humans create causal models of the world to help reason about the world. All these models are descriptive models, some descriptive models with the greater predictive capability (i.e., Chemistry) than other models (i.e. Alchemy).

为什么这个分类很重要？ 这很重要，因为它告诉我们可以发明的模型或现实的种类。 可以想到我们创建的模型属于两个通用类。 有描述性模型，也有生成模型。 现实的描述模型是我们用来描述现实的“形状”的心理模型。 人类创造世界的因果模型，以帮助世界推理。 所有这些模型都是描述性模型，一些描述性模型(例如化学)比其他模型(例如炼金术)具有更大的预测能力。

Reductionists will have the delusion that there are no limits to the descriptive models that we invent. However, we are aware of these limitations in our studies in the complexity sciences. Despite our precise understanding of physics and chemistry, we remain unable to formulate strong predictive models of biology. Yet, biology does what it does without the need for descriptive models. Biology only needs generative models, but how are generative models different from descriptive models?

简化论者会幻想我们发明的描述模型没有局限性。 但是，我们在复杂性科学的研究中意识到了这些局限性。 尽管我们对物理和化学有精确的了解，但我们仍然无法建立强大的生物学预测模型。 然而，生物学无需描述性模型即可完成其工作。 生物学仅需要生成模型，但是生成模型与描述模型有何不同？

This could be a universal characteristic of generative models. This is because generative models are ‘grown’ to favor robustness.

这可能是生成模型的普遍特征。 这是因为生成模型被“成长”以支持鲁棒性。

Generative models are compressed models of how to generate solutions. These compressed models, however, are not compressed in a way to facilitate comprehension. They are compressed to facilitate robust generation. A recent discovery in genetics known as the ‘omnigenic hypothesis’ reveals that coding instructions in DNA are not clustered in chunks but rather is spread out across the entire DNA. Deep Learning has similar characteristics in that if we attempt to create networks that generalize well, we lose interpretability (see: Deep Learning Uncertainty Principle). This uncertainty could be a universal characteristic of generative models. Generative models are ‘grown’ to favor robustness. Generative models in nature have no motivation to favor interpretability. Furthermore, robustness is enhanced through the use of redundancy and randomization.

生成模型是有关如何生成解决方案的压缩模型。 但是，这些压缩模型并未以有助于理解的方式进行压缩。 它们被压缩以促进可靠的生成。 遗传学中的一项最新发现被称为“全基因假说 ”，它揭示了DNA中的编码指令不是成簇聚集，而是分布在整个DNA中。 深度学习具有类似的特征，如果我们尝试创建能够很好地泛化的网络，则会失去可解释性(请参阅： 深度学习不确定性原理 )。 这种不确定性可能是生成模型的普遍特征。 生成模型“成长”以支持健壮性。 自然界中的生成模型没有动机去支持可解释性。 此外，通过使用冗余和随机化增强了鲁棒性。

As we explore generative models in deeper detail we will discover that the forces that lead to their creation are also the same forces that encourage its obfuscation. The execution of generative models inevitably works in mysterious ways. It’s important to realize the kind of information that a generative model captures. It captures information only generation instruction, this does not mean that it includes information that facilitates its interpretability. Following instructions requires lower intelligence than understanding instructions.

当我们更深入地探索生成模型时，我们会发现导致其产生的力量也与鼓励其产生混淆的力量相同。 生成模型的执行不可避免地以神秘的方式进行。 重要的是要认识到生成模型捕获的信息的种类。 它仅捕获信息生成指令，这并不意味着它包含有助于其解释性的信息。 遵循说明比理解说明所需的智力低。

Now that we have defined ‘recombinant programming’, let’s explore if its similar to context priming in GPT-3. It’s useful to compare declarative programming languages to languages like GPT-3. In both cases, unlike imperative languages that you need to specify how to do something, declarative languages are specified by what you want to see.

现在，我们已经定义了“重组编程”，让我们探讨一下它是否类似于GPT-3中的上下文启动。 将声明性编程语言与GPT-3之类的语言进行比较很有用。 在这两种情况下，与需要指定如何操作的命令式语言不同，声明式语言由您要查看的内容指定。

GPT-3 is like that and similar to declarative programming languages, you specify constraints on what the final output will look like. These constraints are in the form of examples. One way to describe these examples are that they are contextual priming strings (see my previous post). How GPT-3 interprets this and generates its output still needs further investigation. But from the interface point of view, it’s similar.

GPT-3与声明式编程语言类似，您可以指定最终输出的外观约束。 这些约束是以示例的形式。 描述这些示例的一种方法是，它们是上下文启动字符串(请参阅我的上一篇文章)。 GPT-3如何解释这一点并产生其输出仍需进一步研究。 但是从界面的角度来看，这是相似的。

That’s why, if you want to build a language on top of stuff like GPT-3, then get inspiration from declarative languages (and not imperative ones).

这就是为什么如果您想在GPT-3之类的东西上构建一种语言，然后从声明性语言(而不是命令性语言)中获得启发。