Document (#43864)

Author

Noever, D.
Ciolino, M.

Title

¬The Turing deception

Source

https%3A%2F%2Farxiv.org%2Fabs%2F2212.06721&usg=AOvVaw3i_9pZm9y_dQWoHi6uv0EN

Year

2022

Abstract

This research revisits the classic Turing test and compares recent large language models such as ChatGPT for their abilities to reproduce human-level comprehension and compelling text generation. Two task challenges- summary and question answering- prompt ChatGPT to produce original content (98-99%) from a single text entry and sequential questions initially posed by Turing in 1950. We score the original and generated content against the OpenAI GPT-2 Output Detector from 2019, and establish multiple cases where the generated content proves original and undetectable (98%). The question of a machine fooling a human judge recedes in this work relative to the question of "how would one prove it?" The original contribution of the work presents a metric and simple grammatical set for understanding the writing mechanics of chatbots in evaluating their readability and statistical clarity, engagement, delivery, overall quality, and plagiarism risks. While Turing's original prose scores at least 14% below the machine-generated output, whether an algorithm displays hints of Turing's true initial thoughts (the "Lovelace 2.0" test) remains unanswerable.

Theme

Computerlinguistik

Object

ChatGPT
Turing-Test

Similar documents (content)

1. Aydin, Ö.; Karaarslan, E.: OpenAI ChatGPT generated literature review: : digital twin in healthcare (2022) 0.20

   0.20094076 = sum of:
     0.20094076 = product of:
       0.83725315 = sum of:
         0.027598588 = weight(abstract_txt:text in 851) [ClassicSimilarity], result of:
           0.027598588 = score(doc=851,freq=4.0), product of:
             0.07279789 = queryWeight, product of:
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.01800205 = queryNorm
             0.37911248 = fieldWeight in 851, product of:
               2.0 = tf(freq=4.0), with freq of:
                 4.0 = termFreq=4.0
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.046875 = fieldNorm(doc=851)
         0.2451935 = weight(abstract_txt:openai in 851) [ClassicSimilarity], result of:
           0.2451935 = score(doc=851,freq=10.0), product of:
             0.18261796 = queryWeight, product of:
               1.1199467 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.01800205 = queryNorm
             1.3426582 = fieldWeight in 851, product of:
               3.1622777 = tf(freq=10.0), with freq of:
                 10.0 = termFreq=10.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.046875 = fieldNorm(doc=851)
         0.08156343 = weight(abstract_txt:chatbots in 851) [ClassicSimilarity], result of:
           0.08156343 = score(doc=851,freq=1.0), product of:
             0.18888661 = queryWeight, product of:
               1.1390065 = boost
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.01800205 = queryNorm
             0.4318116 = fieldWeight in 851, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.046875 = fieldNorm(doc=851)
         0.052798003 = weight(abstract_txt:human in 851) [ClassicSimilarity], result of:
           0.052798003 = score(doc=851,freq=6.0), product of:
             0.09800361 = queryWeight, product of:
               1.1602769 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.01800205 = queryNorm
             0.5387353 = fieldWeight in 851, product of:
               2.4494898 = tf(freq=6.0), with freq of:
                 6.0 = termFreq=6.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.046875 = fieldNorm(doc=851)
         0.032327604 = weight(abstract_txt:content in 851) [ClassicSimilarity], result of:
           0.032327604 = score(doc=851,freq=2.0), product of:
             0.116667606 = queryWeight, product of:
               1.5504628 = boost
               4.17991 = idf(docFreq=1838, maxDocs=44218)
               0.01800205 = queryNorm
             0.2770915 = fieldWeight in 851, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.17991 = idf(docFreq=1838, maxDocs=44218)
               0.046875 = fieldNorm(doc=851)
         0.39777204 = weight(abstract_txt:chatgpt in 851) [ClassicSimilarity], result of:
           0.39777204 = score(doc=851,freq=13.0), product of:
             0.2910645 = queryWeight, product of:
               1.9995637 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.01800205 = queryNorm
             1.3666114 = fieldWeight in 851, product of:
               3.6055512 = tf(freq=13.0), with freq of:
                 13.0 = termFreq=13.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.046875 = fieldNorm(doc=851)
       0.24 = coord(6/25)
   

2. Räwel, J.: Automatisierte Kommunikation (2023) 0.12

   0.12280676 = sum of:
     0.12280676 = product of:
       1.5350845 = sum of:
         0.6525074 = weight(abstract_txt:chatbots in 909) [ClassicSimilarity], result of:
           0.6525074 = score(doc=909,freq=1.0), product of:
             0.18888661 = queryWeight, product of:
               1.1390065 = boost
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.01800205 = queryNorm
             3.4544928 = fieldWeight in 909, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.375 = fieldNorm(doc=909)
         0.882577 = weight(abstract_txt:chatgpt in 909) [ClassicSimilarity], result of:
           0.882577 = score(doc=909,freq=1.0), product of:
             0.2910645 = queryWeight, product of:
               1.9995637 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.01800205 = queryNorm
             3.0322385 = fieldWeight in 909, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.375 = fieldNorm(doc=909)
       0.08 = coord(2/25)
   

3. Jha, A.: Why GPT-4 isn't all it's cracked up to be (2023) 0.11

   0.11122242 = sum of:
     0.11122242 = product of:
       0.46342677 = sum of:
         0.06461416 = weight(abstract_txt:openai in 923) [ClassicSimilarity], result of:
           0.06461416 = score(doc=923,freq=1.0), product of:
             0.18261796 = queryWeight, product of:
               1.1199467 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.01800205 = queryNorm
             0.3538215 = fieldWeight in 923, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.0390625 = fieldNorm(doc=923)
         0.02540245 = weight(abstract_txt:human in 923) [ClassicSimilarity], result of:
           0.02540245 = score(doc=923,freq=2.0), product of:
             0.09800361 = queryWeight, product of:
               1.1602769 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.01800205 = queryNorm
             0.2591991 = fieldWeight in 923, product of:
               1.4142135 = tf(freq=2.0), with freq of:
                 2.0 = termFreq=2.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.0390625 = fieldNorm(doc=923)
         0.022350324 = weight(abstract_txt:test in 923) [ClassicSimilarity], result of:
           0.022350324 = score(doc=923,freq=1.0), product of:
             0.11337681 = queryWeight, product of:
               1.2479659 = boost
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.01800205 = queryNorm
             0.19713312 = fieldWeight in 923, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.046608 = idf(docFreq=772, maxDocs=44218)
               0.0390625 = fieldNorm(doc=923)
         0.02557539 = weight(abstract_txt:machine in 923) [ClassicSimilarity], result of:
           0.02557539 = score(doc=923,freq=1.0), product of:
             0.1240366 = queryWeight, product of:
               1.3053156 = boost
               5.2785225 = idf(docFreq=612, maxDocs=44218)
               0.01800205 = queryNorm
             0.20619228 = fieldWeight in 923, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               5.2785225 = idf(docFreq=612, maxDocs=44218)
               0.0390625 = fieldNorm(doc=923)
         0.15923625 = weight(abstract_txt:chatgpt in 923) [ClassicSimilarity], result of:
           0.15923625 = score(doc=923,freq=3.0), product of:
             0.2910645 = queryWeight, product of:
               1.9995637 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.01800205 = queryNorm
             0.54708236 = fieldWeight in 923, product of:
               1.7320508 = tf(freq=3.0), with freq of:
                 3.0 = termFreq=3.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.0390625 = fieldNorm(doc=923)
         0.16624819 = weight(abstract_txt:turing in 923) [ClassicSimilarity], result of:
           0.16624819 = score(doc=923,freq=1.0), product of:
             0.49454227 = queryWeight, product of:
               3.1921842 = boost
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.01800205 = queryNorm
             0.3361658 = fieldWeight in 923, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.6058445 = idf(docFreq=21, maxDocs=44218)
               0.0390625 = fieldNorm(doc=923)
       0.24 = coord(6/25)
   

4. Lund, B.D.: ¬A brief review of ChatGPT : its value and the underlying GPT technology (2023) 0.10

   0.09588766 = sum of:
     0.09588766 = product of:
       0.5992979 = sum of:
         0.022998825 = weight(abstract_txt:text in 873) [ClassicSimilarity], result of:
           0.022998825 = score(doc=873,freq=1.0), product of:
             0.07279789 = queryWeight, product of:
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.01800205 = queryNorm
             0.3159271 = fieldWeight in 873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.078125 = fieldNorm(doc=873)
         0.12922832 = weight(abstract_txt:openai in 873) [ClassicSimilarity], result of:
           0.12922832 = score(doc=873,freq=1.0), product of:
             0.18261796 = queryWeight, product of:
               1.1199467 = boost
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.01800205 = queryNorm
             0.707643 = fieldWeight in 873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.05783 = idf(docFreq=13, maxDocs=44218)
               0.078125 = fieldNorm(doc=873)
         0.035924487 = weight(abstract_txt:human in 873) [ClassicSimilarity], result of:
           0.035924487 = score(doc=873,freq=1.0), product of:
             0.09800361 = queryWeight, product of:
               1.1602769 = boost
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.01800205 = queryNorm
             0.3665629 = fieldWeight in 873, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.692005 = idf(docFreq=1101, maxDocs=44218)
               0.078125 = fieldNorm(doc=873)
         0.41114628 = weight(abstract_txt:chatgpt in 873) [ClassicSimilarity], result of:
           0.41114628 = score(doc=873,freq=5.0), product of:
             0.2910645 = queryWeight, product of:
               1.9995637 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.01800205 = queryNorm
             1.4125607 = fieldWeight in 873, product of:
               2.236068 = tf(freq=5.0), with freq of:
                 5.0 = termFreq=5.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.078125 = fieldNorm(doc=873)
       0.16 = coord(4/25)
   

5. Dampz, N.: ChatGPT interpretiert jetzt auch Bilder : Neue Version (2023) 0.08

   0.08227394 = sum of:
     0.08227394 = product of:
       0.68561614 = sum of:
         0.04599765 = weight(abstract_txt:text in 874) [ClassicSimilarity], result of:
           0.04599765 = score(doc=874,freq=1.0), product of:
             0.07279789 = queryWeight, product of:
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.01800205 = queryNorm
             0.6318542 = fieldWeight in 874, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               4.0438666 = idf(docFreq=2106, maxDocs=44218)
               0.15625 = fieldNorm(doc=874)
         0.27187812 = weight(abstract_txt:chatbots in 874) [ClassicSimilarity], result of:
           0.27187812 = score(doc=874,freq=1.0), product of:
             0.18888661 = queryWeight, product of:
               1.1390065 = boost
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.01800205 = queryNorm
             1.4393721 = fieldWeight in 874, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               9.211981 = idf(docFreq=11, maxDocs=44218)
               0.15625 = fieldNorm(doc=874)
         0.3677404 = weight(abstract_txt:chatgpt in 874) [ClassicSimilarity], result of:
           0.3677404 = score(doc=874,freq=1.0), product of:
             0.2910645 = queryWeight, product of:
               1.9995637 = boost
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.01800205 = queryNorm
             1.2634326 = fieldWeight in 874, product of:
               1.0 = tf(freq=1.0), with freq of:
                 1.0 = termFreq=1.0
               8.085969 = idf(docFreq=36, maxDocs=44218)
               0.15625 = fieldNorm(doc=874)
       0.12 = coord(3/25)