{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Example Usage of TopicGPT: 20 Newsgroups Dataset"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "In this notebook, we will use the 20 Newsgroups dataset to demonstrate the use of the topicgpt package"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 138,
   "metadata": {},
   "outputs": [],
   "source": [
    "from topicgpt.TopicGPT import TopicGPT\n",
    "from sklearn.datasets import fetch_20newsgroups "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 139,
   "metadata": {},
   "outputs": [],
   "source": [
    "# select your own API key here. (Note: This specific code will not work for you unless you specified an environment variable for OPENAI_API_KEY)\n",
    "import os\n",
    "api_key_openai = os.environ.get('OPENAI_API_KEY') "
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Load Data"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 154,
   "metadata": {},
   "outputs": [],
   "source": [
    "data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')) #download the 20 Newsgroups dataset\n",
    "corpus = data['data']\n",
    "corpus = [doc for doc in corpus if doc != \"\"]\n"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Initialize and fit the model "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tm = TopicGPT(\n",
    "    openai_api_key = api_key_openai,\n",
    "    n_topics = 20 # select 20 topics since the true number of topics is 20 \n",
    ")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Computing vocabulary...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Processing corpus: 100%|██████████| 980/980 [00:04<00:00, 208.30it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Most common words:\n",
      "n't: 813\n",
      "would: 557\n",
      "one: 481\n",
      "people: 353\n",
      "know: 312\n",
      "like: 307\n",
      "think: 291\n",
      "get: 259\n",
      "use: 258\n",
      "also: 254\n",
      "Computing embeddings...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 980/980 [00:00<00:00, 1301.57it/s]\n",
      "100%|██████████| 980/980 [09:41<00:00,  1.69it/s]\n",
      "100%|██████████| 2609/2609 [00:00<00:00, 4873.34it/s]\n",
      "100%|██████████| 2609/2609 [24:19<00:00,  1.79it/s] \n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Extracting topics...\n",
      "UMAP(angular_rp_forest=True, metric='cosine', min_dist=0, n_components=5, random_state=42, verbose=True)\n",
      "Mon Sep  4 19:29:23 2023 Construct fuzzy simplicial set\n",
      "Mon Sep  4 19:29:38 2023 Finding Nearest Neighbors\n",
      "Mon Sep  4 19:29:49 2023 Finished Nearest Neighbor Search\n",
      "Mon Sep  4 19:29:58 2023 Construct embedding\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 500/500 [00:08]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Mon Sep  4 19:30:07 2023 Finished embedding\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 100/100 [00:02]\n",
      "Computing word-topic matrix: 100%|██████████| 12/12 [00:04<00:00,  2.48it/s]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of tfidf:  (2609, 11)\n",
      "shape fo word_topic_mat:  (2609, 11)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 100/100 [00:03]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Describing topics...\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "100%|██████████| 11/11 [01:00<00:00,  5.53s/it]\n"
     ]
    }
   ],
   "source": [
    "tm.fit(corpus)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "# in case the respective pickle files for this example exist, you can also load the model directly\n",
    "\n",
    "with open(\"../Data/SavedTopicRepresentations/TopicGPT_20ng.pkl\", \"rb\") as f:\n",
    "    tm = pickle.load(f)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "## Get an overview over the identified topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [
    {
     "ename": "NameError",
     "evalue": "name 'tm' is not defined",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mNameError\u001b[0m                                 Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[1], line 1\u001b[0m\n\u001b[1;32m----> 1\u001b[0m tm\u001b[39m.\u001b[39mtopic_lis\n",
      "\u001b[1;31mNameError\u001b[0m: name 'tm' is not defined"
     ]
    }
   ],
   "source": [
    "tm.topic_lis"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 144,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Topic 0: Electronics Equipment Sales\n",
      "\n",
      "Topic_description: The common topic of the given words appears to be \"electronics and technology\". \n",
      "\n",
      "Various aspects and sub-topics of this topic include:\n",
      "1. Buying and selling: \"offer\", \"sale\", \"sell\", \"price\", \"buy\"\n",
      "2. Device usage and features: \"use\", \"get\", \"new\", \"used\", \"condition\"\n",
      "3. Technical specifications: \"wire\", \"ground\", \"power\", \"circuit\", \"voltage\"\n",
      "4. Communication and connectivity: \"phone\", \"email\", \"modem\", \"wireless\", \"connection\"\n",
      "5. Accessories and peripherals: \"battery\", \"cable\", \"manuals\", \"disk\", \"monitor\"\n",
      "Top words: [\"n't\", 'one', 'would', 'use', 'like', 'get', 'new', 'used', 'offer', 'sale']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 1: Image Processing\n",
      "\n",
      "Topic_description: The common topic of the given words is \"Image Processing and Graphics\". \n",
      "\n",
      "Aspects and sub-topics of this topic include:\n",
      "1. Image Manipulation: \"file\", \"image\", \"format\", \"data\", \"color\"\n",
      "2. Software and Tools: \"program\", \"software\", \"package\", \"tools\", \"library\"\n",
      "3. Display and Visualization: \"window\", \"display\", \"widget\", \"graphics\", \"screen\"\n",
      "4. Application and Usage: \"use\", \"application\", \"program\", \"work\", \"run\"\n",
      "5. Conversion and Compatibility: \"convert\", \"version\", \"formats\", \"compatible\", \"support\"\n",
      "Top words: [\"n't\", 'file', 'image', 'use', 'available', 'get', 'also', 'program', 'files', 'one']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 2: Gun control\n",
      "\n",
      "Topic_description: The common topic of the provided words is \"gun control\". \n",
      "\n",
      "Aspects and sub-topics of the topic include:\n",
      "1. Public opinion and government: \"people\", \"government\", \"think\", \"believe\", \"public\"\n",
      "2. Laws and regulations: \"law\", \"control\", \"laws\", \"crime\", \"illegal\"\n",
      "3. Safety and crime prevention: \"safety\", \"crime\", \"violence\", \"criminals\", \"protection\"\n",
      "4. Rights and individual freedom: \"rights\", \"freedom\", \"individual\", \"rights\", \"personal\"\n",
      "5. Policy and legislation: \"government\", \"policy\", \"legislation\", \"enforcement\", \"regulations\"\n",
      "Top words: [\"n't\", 'would', 'people', 'think', 'one', 'know', 'gun', 'get', 'like', 'government']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 3: Online Privacy and Anonymity\n",
      "\n",
      "Topic_description: The common topic of the given words is \"Internet Privacy\". \n",
      "\n",
      "Aspects of the topic include:\n",
      "1. Anonymity: Protecting identity online.\n",
      "2. Security: Ensuring the safety of information.\n",
      "3. Privacy: Maintaining confidentiality of personal data.\n",
      "4. Encryption: Securing data through coding.\n",
      "5. Anonymity tools: Services to hide online activities.\n",
      "Top words: ['anonymous', 'email', 'internet', 'address', 'information', \"n't\", 'privacy', 'mail', 'one', 'use']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 4: Conflict and Violence.\n",
      "\n",
      "Topic_description: The common topic of the provided words appears to be \"conflict and violence\". \n",
      "\n",
      "Aspects and sub-topics related to this topic include:\n",
      "1. War and military actions: soldiers, troops, genocide, massacre, attack.\n",
      "2. Government and politics: government, political, state, policy, authorities.\n",
      "3. Human rights and justice: rights, civilians, ethnic, leaders, atrocities.\n",
      "4. Victims and suffering: killed, dead, bodies, wounded, victims.\n",
      "5. International involvement: international, countries, support, forces, nations.\n",
      "Top words: [\"n't\", 'people', 'would', 'one', 'said', 'could', 'know', 'like', 'time', 'also']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 5: Computer Hardware\n",
      "\n",
      "Topic_description: The common topic of the provided words is computer hardware. The various aspects and sub-topics of this topic include: \n",
      "- Storage: disk, drive, hard, floppy, drives, data\n",
      "- Components: card, controller, board, chip, motherboard, power, connector\n",
      "- Peripherals: monitor, video, printer, modem, cable, port\n",
      "- Performance: speed, memory, clock, processor, cache\n",
      "- Software: system, software, program, interface, drivers\n",
      "- User experience: problem, work, support, problem, run, help\n",
      "Top words: [\"n't\", 'drive', 'card', 'one', 'would', 'use', 'know', 'get', 'like', 'disk']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 6: Belief and Atheism\n",
      "\n",
      "Topic_description: The common topic of the provided words is \"Belief and Religion\". \n",
      "\n",
      "Aspects and sub-topics of this topic include:\n",
      "1. Atheism: \"atheists\", \"atheism\", \"atheist\"\n",
      "2. God and Religion: \"god\", \"religion\", \"religious\", \"religions\"\n",
      "3. Faith and Beliefs: \"faith\", \"belief\", \"beliefs\"\n",
      "4. Morality and Ethics: \"moral\", \"morality\", \"morals\"\n",
      "5. Science and Truth: \"science\", \"truth\", \"evidence\"\n",
      "\n",
      "Please note that these descriptions are based solely on the provided words and may not capture the full complexity of the topic.\n",
      "Top words: [\"n't\", 'one', 'would', 'people', 'think', 'believe', 'say', 'know', 'like', 'evidence']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 7: Online Discussions\n",
      "\n",
      "Topic_description: The common topic of the given words appears to be online discussions or forums. \n",
      "\n",
      "Various aspects and sub-topics of this topic include:\n",
      "1. Communication: Words like \"say,\" \"post,\" \"letter,\" and \"quote\" indicate the act of expressing thoughts and opinions.\n",
      "2. Interaction: Words like \"reply,\" \"discuss,\" and \"answer\" suggest engagement and exchange of ideas.\n",
      "3. Opinion sharing: Words like \"think,\" \"believe,\" and \"opinions\" indicate the expression of personal viewpoints.\n",
      "4. Group dynamics: Words like \"group,\" \"members,\" and \"names\" suggest the presence of a community or collective.\n",
      "5. Internet culture: Words like \"internet,\" \"newsgroup,\" and \"usenet\" imply the online nature of the discussions.\n",
      "Top words: [\"n't\", 'people', 'say', 'like', 'group', 'would', 'post', 'know', 'one', 'think']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 8: Computer Software\n",
      "\n",
      "Topic_description: The common topic of the given words is **computer software and troubleshooting**.\n",
      "\n",
      "Aspects and sub-topics of the topic include:\n",
      "1. **Operating system**: Windows, DOS, system, memory, disk, drive.\n",
      "2. **Software usage**: Program, use, run, files, software, applications.\n",
      "3. **Problem-solving**: Help, support, fix, error, crashes.\n",
      "4. **Hardware**: Keyboard, mouse, graphics, hardware, device.\n",
      "5. **User experience**: Interface, user, easy, features, user-friendly.\n",
      "Top words: [\"n't\", 'file', 'use', 'program', 'windows', 'would', 'one', 'disk', 'get', 'like']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 9: Car Features and Performance\n",
      "\n",
      "Topic_description: The common topic among the provided words is \"cars\" or \"automobiles\". \n",
      "\n",
      "Aspects and sub-topics of the topic include:\n",
      "- Car features: engine, radar, clutch, transmission, gear, tires, brakes, suspension, steering, throttle, gas, body, top, convertible, interior, seat, foot, door, shifter, gears, radio, road, wheels, oil, air, gauge, dash, windows, fuel, speedo, battery, factory, automatics, manuals, tire, recall, rust, water, damage, paint, reading, stereo, sunroof, safety, lights, coupe, sport, mechanical, handling.\n",
      "- Car performance: speed, power, torque, performance, highway, acceleration, handling.\n",
      "- Car condition: used, problem, condition, mileage, parts, odometer, maintenance, service, repair, rust, damage.\n",
      "- Car buying/selling: price, dealer, used, buying, sell, market, value, dealership, trade-in, sales, invoice, purchase, insurance.\n",
      "- Car models and brands: new, model, make, year, models, brand, manufacturer.\n",
      "- Car driving experience: drive, automatic, manual, clutch, transmission, gear, shift, driver, driving, pedal, road, traffic, highway, speed, control, handling.\n",
      "- Car opinions and reviews: opinion, review, experience, question, opinion, mine, works, questions, opinions.\n",
      "\n",
      "Please note that these sub-topics are inferred based on the provided words and may not cover all possible aspects and sub-topics related to cars.\n",
      "Top words: ['car', \"n't\", 'would', 'like', 'one', 'new', 'engine', 'cars', 'get', 'know']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 10: Encryption and Government\n",
      "\n",
      "Topic_description: The common topic of the given words is \"encryption and government access\". \n",
      "\n",
      "Aspects and sub-topics:\n",
      "1. Encryption methods: algorithms, keys, public-key, ciphers.\n",
      "2. Government access: law, enforcement, wiretap, wiretaps, agencies.\n",
      "3. Privacy and security: secure, secure communication, secure system.\n",
      "4. Technology and devices: chip, phone, device, hardware, computer.\n",
      "5. Public opinion: debate, controversy, opinions, concerns.\n",
      "Top words: [\"n't\", 'key', 'would', 'encryption', 'government', 'use', 'one', 'chip', 'keys', 'people']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 11: Technology and Computing.\n",
      "\n",
      "Topic_description: The common topic of the given words is \"Plumbing\". The various aspects and sub-topics of this topic include: \n",
      "1. \"Pleasure\" and \"Pleasing\" - Satisfaction and enjoyment in plumbing work.\n",
      "2. \"Plenty\" and \"Plentiful\" - Abundance of plumbing supplies.\n",
      "3. \"Plugs\" and \"Plugging\" - Sealing and fitting pipes.\n",
      "4. \"Plunger\" and \"Plunged\" - Unclogging drains and toilets.\n",
      "5. \"Plumbing Fixtures\" - Various components used in plumbing systems.\n",
      "Top words: ['a.a', 'plesetsk', 'plenty', 'plentiful', 'pledged', 'pledge', 'pleasures', 'pleasure', 'pleasing', 'pleases']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 12: Technology and Computing\n",
      "\n",
      "Topic_description: The common topic of the given words is \"Plants\". \n",
      "\n",
      "Aspects and sub-topics of the topic \"Plants\":\n",
      "1. Pleasure and enjoyment: pleasures, pleasure, pleasing, pleases, pleased.\n",
      "2. Pledges and commitments: pledged, pledge.\n",
      "3. Variety and abundance: plenty, plentiful, plethora.\n",
      "4. Play and entertainment: plays, playmates, playmation, playground.\n",
      "5. Plumbing and construction: plumbing, plugs, plugging, pliers, plunger.\n",
      "Top words: ['a.a', 'plesetsk', 'plenty', 'plentiful', 'pledged', 'pledge', 'pleasures', 'pleasure', 'pleasing', 'pleases']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 13: Space Exploration\n",
      "\n",
      "Topic_description: The common topic of the given words is \"space exploration\". \n",
      "\n",
      "Aspects and sub-topics of the topic include:\n",
      "1. Spacecraft and Missions: Launch, orbit, satellite, mission, space probes.\n",
      "2. Technology and Systems: Systems, data, technology, software, hardware.\n",
      "3. Astronomical Objects: Planets, moon, stars, universe, galaxies.\n",
      "4. Challenges and Problems: Cost, funding, safety, technical difficulties.\n",
      "5. Scientific Research: Astronomy, physics, scientific discoveries, research projects.\n",
      "Top words: [\"n't\", 'would', 'space', 'one', 'like', 'could', 'launch', 'also', 'orbit', 'time']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 14: Motorcycle Riding Techniques\n",
      "\n",
      "Topic_description: The common topic of the provided words is \"Motorcycling\". \n",
      "\n",
      "Aspects and sub-topics of this topic include:\n",
      "1. Riding experience: \"ride\", \"riding\", \"bikes\", \"road\", \"miles\"\n",
      "2. Motorcycle maintenance: \"oil\", \"engine\", \"gas\", \"exhaust\", \"brake\"\n",
      "3. Safety and gear: \"helmet\", \"gear\", \"traffic\", \"lights\", \"helmets\"\n",
      "4. Road conditions and hazards: \"lane\", \"traffic\", \"curve\", \"speed\", \"pothole\"\n",
      "5. Tips and advice: \"advice\", \"tips\", \"technique\", \"recommend\", \"learn\"\n",
      "Top words: [\"n't\", 'bike', 'one', 'would', 'get', 'like', 'time', 'car', 'much', 'dog']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 15: Technology\n",
      "\n",
      "Topic_description: The common topic of the provided words is \"planning and activities\". \n",
      "\n",
      "Sub-topics and aspects of this topic include:\n",
      "1. Pledges and pleas: pledge, pledge, please, pleased, pledging\n",
      "2. Pleasure and enjoyment: pleasure, pleasing, pleasant, pleasures\n",
      "3. Play and entertainment: plays, playmates, playoffs, playmation\n",
      "4. Planting and growth: plants, planting, planters, planted\n",
      "5. Policing and policies: police, policy, policing, policies\n",
      "Top words: ['a.a', 'plesetsk', 'plenty', 'plentiful', 'pledged', 'pledge', 'pleasures', 'pleasure', 'pleasing', 'pleases']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 16: Hockey Games\n",
      "\n",
      "Topic_description: The common topic of the given words is \"hockey\". \n",
      "\n",
      "Aspects of the topic include: \n",
      "1. Game: \"play\", \"games\", \"played\", \"scoring\", \"shot\"\n",
      "2. Teams: \"team\", \"teams\", \"players\", \"player\", \"defense\"\n",
      "3. Seasons: \"season\", \"year\", \"years\", \"playoffs\", \"regular\"\n",
      "4. Goals: \"goal\", \"goals\", \"scored\", \"net\", \"scoring\"\n",
      "5. Fans: \"fans\", \"watch\", \"watching\", \"crowd\", \"cheer\"\n",
      "Top words: [\"n't\", 'game', 'team', 'would', 'games', 'play', 'hockey', 'period', 'one', 'season']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 17: Health and Medicine.\n",
      "\n",
      "Topic_description: The common topic of the given words appears to be \"health and medicine\". \n",
      "\n",
      "Aspects and sub-topics related to this topic include:\n",
      "1. Diseases and conditions: \"patients\", \"disease\", \"infection\", \"cancer\", \"symptoms\"\n",
      "2. Medical treatment: \"treatment\", \"medicine\", \"therapy\", \"drug\", \"clinical\"\n",
      "3. Health information and research: \"health\", \"research\", \"medical\", \"evidence\", \"studies\"\n",
      "4. Prevention and risk factors: \"prevention\", \"risk\", \"causes\", \"risk\", \"factors\"\n",
      "5. Healthcare professionals: \"doctor\", \"physician\", \"doctors\", \"physicians\", \"hospital\"\n",
      "\n",
      "Please note that these are general sub-topics inferred from the provided words and may not cover all possible aspects within the topic.\n",
      "Top words: [\"n't\", 'one', 'would', 'patients', 'disease', 'people', 'use', 'know', 'also', 'like']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 18: Baseball games and teams.\n",
      "\n",
      "Topic_description: The common topic of the given words is \"baseball\". \n",
      "\n",
      "Aspects and sub-topics of the topic \"baseball\" include: \n",
      "1. Teams and players: \"team\", \"players\", \"pitcher\", \"hitter\", \"fielder\"\n",
      "2. Game elements: \"game\", \"hit\", \"runs\", \"ball\", \"run\"\n",
      "3. Performance and skills: \"good\", \"better\", \"average\", \"pitching\", \"batting\"\n",
      "4. Seasons and years: \"year\", \"last\", \"season\", \"years\", \"career\"\n",
      "5. Strategy and analysis: \"think\", \"strategy\", \"stats\", \"record\", \"analysis\"\n",
      "Top words: [\"n't\", 'would', 'year', 'game', 'team', 'think', 'one', 'good', 'last', 'games']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "Topic 19: Beliefs about Homosexuality.\n",
      "\n",
      "Topic_description: The common topic of the given words appears to be \"religion and morality\". \n",
      "\n",
      "Aspects and sub-topics of this topic include:\n",
      "1. Beliefs and faith: \"sin\", \"faith\", \"belief\", \"eternal\", \"heaven\"\n",
      "2. Morality and ethics: \"homosexuality\", \"marriage\", \"sinful\", \"sexual\", \"wrong\"\n",
      "3. Biblical interpretation: \"passage\", \"context\", \"interpretation\", \"translation\", \"verses\"\n",
      "4. Religious practices: \"baptism\", \"worship\", \"ceremony\", \"churches\", \"prayer\"\n",
      "5. Salvation and redemption: \"salvation\", \"repent\", \"saved\", \"forgiveness\", \"sins\"\n",
      "Top words: [\"n't\", 'would', 'one', 'people', 'sin', 'think', 'say', 'know', 'believe', 'homosexuality']\n",
      "\n",
      "------------------------------------------------------------------------------------------------------------------------------------------------------\n",
      "\n"
     ]
    }
   ],
   "source": [
    "tm.print_topics()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [],
   "source": [
    "tm.visualize_clusters()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Obtain more detailed information about the topics"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 145,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT wants to the call the function:  {\n",
      "  \"name\": \"knn_search\",\n",
      "  \"arguments\": \"{\\n  \\\"topic_index\\\": 13,\\n  \\\"query\\\": \\\"moon landing\\\"\\n}\"\n",
      "}\n",
      "Topic 13, which is related to the keyword \"moon landing,\" contains information about various aspects of space exploration and missions to the Moon. Here are some key points:\n",
      "\n",
      "1. The United States has sent automated spacecraft and human-crewed expeditions to explore the Moon. These missions have provided significant knowledge and understanding of the lunar surface.\n",
      "   - Document index: 258\n",
      "\n",
      "2. NASA's automated spacecraft for solar system exploration come in various shapes and sizes. Each spacecraft consists of scientific instruments selected for specific missions and is supported by basic subsystems for electrical power, trajectory control, and data communication with Earth.\n",
      "   - Document index: 535\n",
      "\n",
      "3. The Mariner missions, conducted between 1962 and 1975, played a crucial role in the early planetary reconnaissance of the Moon and other terrestrial planets like Venus and Mars.\n",
      "   - Document index: 357, 321\n",
      "\n",
      "4. There was a proposal for a Back to the Moon bill that aimed to incentivize private companies to develop lunar orbiters and explore the Moon. The program had a budget cap of $65 million.\n",
      "   - Document index: 102\n",
      "\n",
      "5. There have been discussions and speculations about the possibility of alternative approaches to lunar missions, such as launching from altitude instead of ground pads and using bio-engineered CO2 absorbing plants for life support.\n",
      "   - Document index: 165, 458\n",
      "\n",
      "6. There have been ideas and discussions about creating a lunar habitat, developing cost-effective moon missions, and setting up a moon base.\n",
      "   - Document index: 323, 50, 344, 545\n",
      "\n",
      "7. There have been debates regarding the authenticity of the first spacewalk and speculation about the possibility of staging a fake moon landing.\n",
      "   - Document index: 571\n",
      "\n",
      "Please note that the information is a summary of the key points found in the documents related to topic 13. For more detailed information, please refer to the corresponding document indices mentioned.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "(['This file and other text and image files from JPL missions are available from the JPL Info public access computer site, reachable by Internet via anonymous ftp to pubinfo.jpl.nasa.gov (128.149.6.2); or by dialup modem to +1 (818) 354-1333, up to 9600 bits per second, parameters N-8-1. -----------------------------------------------------------------  Our Solar System at a Glance  Information Summary  PMS 010-A (JPL) June 1991  JPL 410-34-1  6/91  NASA National Aeronautics and Space Administration  Jet Propulsion Laboratory California Institue of Technology Pasadena, California   For a printed copy of this publication contact the public mail office at the NASA center in your geographic region.    INTRODUCTION       From our small world we have gazed upon the cosmic ocean for untold thousands of years. Ancient astronomers observed points of light that appeared to move among the stars. They called these objects planets, meaning wanderers, and named them after Roman deities -- Jupiter, king of the gods; Mars, the god of war; Mercury, messenger of the gods; Venus, the god of love and beauty, and Saturn, father of Jupiter and god of agriculture. The stargazers also observed comets with sparkling tails, and meteors or shooting stars apparently falling from the sky.       Science flourished during the European Renaissance. Fundamental physical laws governing planetary motion were discovered, and the orbits of the planets around the Sun were calculated. In the 17th century, astronomers pointed a new device called the telescope at the heavens and made startling discoveries.       But the years since 1959 have amounted to a golden age of solar system exploration. Advancements in rocketry after World War II enabled our machines to break the grip of Earth\\'s gravity and travel to the Moon and to other planets.       The United States has sent automated spacecraft, then human-crewed expeditions, to explore the Moon. Our automated machines have orbited and landed on Venus and Mars; explored the Sun\\'s environment; observed comets, and made close-range surveys while flying past Mercury, Jupiter, Saturn, Uranus and Neptune.       These travelers brought a quantum leap in our knowledge and understanding of the solar system. Through the electronic sight and other \"senses\" of our automated spacecraft, color and complexion have been given to worlds that for centuries appeared to Earth-bound eyes as fuzzy disks or indistinct points of light. And dozens of previously unknown objects have been discovered.       Future historians will likely view these pioneering flights through the solar system as some of the most remarkable achievements of the 20th century.        AUTOMATED SPACECRAFT       The National Aeronautics and Space Administration\\'s (NASA\\'s) automated spacecraft for solar system exploration come in many shapes and sizes. While they are designed to fulfill separate and specific mission objectives, the craft share much in common.       Each spacecraft consists of various scientific instruments selected for a particular mission, supported by basic subsystems for electrical power, trajectory and orientation control, as well as for processing data and communicating with Earth.       Electrical power is required to operate the spacecraft instruments and systems. NASA uses both solar energy from arrays of photovoltaic cells and small nuclear generators to power its solar system missions. Rechargeable batteries are employed for backup and supplemental power.       Imagine that a spacecraft has successfully journeyed millions of miles through space to fly but one time near a planet, only to have its cameras and other sensing instruments pointed the wrong way as it speeds past the target! To help prevent such a mishap, a subsystem of small thrusters is used to control spacecraft.       The thrusters are linked with devices that maintain a constant gaze at selected stars. Just as Earth\\'s early seafarers used the stars to navigate the oceans, spacecraft use stars to maintain their bearings in space. With the subsystem locked onto fixed points of reference, flight controllers can keep a spacecraft\\'s scientific instruments pointed at the target body and the craft\\'s communications antennas pointed toward Earth. The thrusters can also be used to fine-tune the flight path and speed of the spacecraft to ensure that a target body is encountered at the planned distance and on the proper trajectory.       Between 1959 and 1971, NASA spacecraft were dispatched to study the Moon and the solar environment; they also scanned the inner planets other than Earth -- Mercury, Venus and Mars. These three worlds, and our own, are known as the terrestrial planets because they share a solid-rock composition.       For the early planetary reconnaissance missions, NASA employed a highly successful series of spacecraft called the Mariners. Their flights helped shape the planning of later missions. Between 1962 and 1975, seven Mariner missions conducted the first surveys of our planetary neighbors in space.       All of the Mariners used solar panels as their primary power source. The first and the',\n",
       "  'Although the $1 billion scheme is a fantasy (it\\'s an old canard in the space business called \"trolling for billionaires\"), there is a good chance that a much smaller program ($65 million) will pass the 103rd Congress. This is the Back to the Moon bill, put together by the people who passed the Launch Services Purchase Act. The bill would incent private companies to develop lunar orbiters, with vendors selected on the basis of competitive bidding. There is an aggregate cap on the bids of $65 million.   Having a single rich individual paying billions for lunar missions is probably worse than having the government bankroll a $65 million program, as the Delta Clipper program has shown (DC-X was funded by SDIO at $59 million). We have a clear chance of making a lunar mission happen in this decade - as opposed to simply wishing for our dreams to come true. Please support the Back to the Moon bill.   For more information, please send E-mail with your U.S. postal service address.',\n",
       "  \"Afraid I can't give any more info on this.. and hoping someone in greter NETLAND has some details.  A short story in the newspaper a few days ago made some sort of mention about how the Japanese, using what sounded like a gravity assist, had just managed to crash (or crash-land) a package on the moon.  the article was very vague and unclear.  and, to make matters worse, I didn't clip it.  does this jog anyone's memory? \",\n",
       "  \"Suppose the Soviets had managed to get their moon rocket working and had made it first.  They could have beaten us if either:  * Their rocket hadn't blown up on the pad thus setting them back,  and/or  * A Saturn V went boom.  If they had beaten us, I speculate that the US would have gone head and done some landings, but we also would have been more determined to set up a base (both in Earth Orbit and on the Moon).  Whether or not we would be on Mars by now would depend upon whether the Soviets tried to go.  Setting up a lunar base would have stretched the budgets of both nations and I think that the military value of a lunar base would outweigh the value of going to Mars (at least in the short run).  Thus we would have concentrated on the moon. \",\n",
       "  ' Their Hiten engineering-test mission spent a while in a highly eccentric Earth orbit doing lunar flybys, and then was inserted into lunar orbit using some very tricky gravity-assist-like maneuvering.  This meant that it would crash on the Moon eventually, since there is no such thing as a stable lunar orbit (as far as anyone knows), and I believe I recall hearing recently that it was about to happen.',\n",
       "  'COMMERCIAL SPACE NEWS/SPACE TECHNOLOGY INVESTOR NUMBER 22     This is number twenty-two in an irregular series on commercial  space activities.  The commentaries included are my thoughts on  these developments.       Sigh... as usual, I\\'ve gotten behind in getting this column  written.  I can only plead the exigency of the current dynamics in  the space biz.  This column is put together at lunch hour and after  the house quiets down at night, so data can quickly build up if  there\\'s a lot of other stuff going on.  I\\'ve complied a lot of  information and happenings since the last column, so I\\'m going to  have to work to keep this one down to a readable length.  Have fun!   CONTENTS: 1- US COMMERCIAL SPACE SALES FLATTEN IN 1993 2- DELTA WINS TWO KEY LAUNCH CONTRACTS 3- COMMERCIAL REMOTE SENSING VENTURE GETS DOC \"GO-AHEAD\" 4- INVESTMENT FIRM CALLS GD\\'S SPACE BIZ \"STILL A GOOD INVESTMENT\"  5- ARIANE PREDICTS DIP IN LAUNCH DEMAND 6- NTSB INVESTIGATES PEGASUS LAUNCH OVER ABORTED ABORT 7- ANOTHER PEGASUS COMPETITOR IS ANNOUNCED 8- GEORGIA LAUNCH SITE DROPPED FROM PLANNING 9- SPAIN\\'S CAPRICORNIA LAUNCHER STILL PROCEEDING 10- PACASTRO SIGNS LAUNCH RESERVATION WITH SWEDISH SPACE CORP 11- CHINA AND TAIWAN JOINT SATELLITE VENTURE REPORTED 12- SOUTH KOREA ANNOUNCES NATIONAL MOVE INTO SPACE TECHNOLOGIES 13- SPACE TECHNOLOGY INDEXES THROUGH MARCH FINAL NOTES    ARTICLES -------------------------------------------------------------------- 1- US COMMERCIAL SPACE SALES FLATTEN IN 1993    The US Department of Commerce projects US commercial space sales  will remain flat in 1993, with current data showing only a 2 percent  growth over 1992.  As published in \"US Industrial Outlook 1993\"  (which was released in January), revenues from the 1993 US space  business are currently projected to be about $4,890 M.     In contrast to previous years when US commercial space sales had  shown double digits growth rates, this year\\'s projected results are  driven by the US satellite manufacturing industry, where sales are  projected to drop from 12 satellites worth $1,300 M in 1992 to 7  satellites worth $ 670 M in 1993.  The US Industrial Outlook also  projects U.S. commercial launchers faces flat demand in coming year,  and while predicting that 1993 revenues will increase 10 percent to  $450 M, future sales will be \"adversely affected by the downward  revision in Department of Defense launch plans.\"                               Offsetting flat launch revenues and satellite deliveries,  revenues for fixed and mobile satellite services are projected to  increase to $1,900 M, primarily driven by increased revenues from  broadcast and cable TV networks.  Similarly, remote sensing products  and sales are projected to increase to $250 M in 1993 (up 15%).    US COMMERCIAL SPACE REVENUES     1989  1990   1991 1992(r) 1993(e)  Commercial satellites             900 1,000  1,100  1,300     670  Satellite services                750   800  1,200  1,500   1,900    Fixed                          (700) (735)(1,115)(1,275) (1,520)    Mobile                          (50) ( 65)(   85)(  225) (  380)  Satellite ground equip            790   860  1,350  1,400   1,560    Mobile equipment                (40)  (85)  (280)  (352)    ???  Commercial launches               150   570    380    450     450  Remote sensing data and services  125   155    190    215     250  Private microgravity research lab  --    --     --     --      60                                  ===== =====  ======  =====   =====    TOTAL ANNUAL REVENUES         2,715 3,385   4,220  4,815   4,890         (r) = revised data for 1992   (e) = estimated data for 1993    [Commentary: This is the first look at',\n",
       "  \"Original to: keithley@apple.com G'day keithley@apple.com  21 Apr 93 22:25, keithley@apple.com wrote to All:   kc> keithley@apple.com (Craig Keithley), via Kralizec 3:713/602    kc> But back to the contest goals, there was a recent article in AW&ST about a  kc> low cost (it's all relative...) manned return to the moon.  A General  kc> Dynamics scheme involving a Titan IV & Shuttle to lift a Centaur upper  kc> stage, LEV, and crew capsule.  The mission consists of delivering two  kc> unmanned payloads to the lunar surface, followed by a manned mission.  kc> Total cost:  US was $10-$13 billion.  Joint ESA(?)/NASA project was $6-$9  kc> billion for the US share.   kc> moon for a year.   Hmmm.  Not really practical.  Anyone got a  kc> cheaper/better way of delivering 15-20 tonnes to the lunar surface within  kc> the decade?  Anyone have a more precise guess about how much a year's  kc> supply of consumables and equipment would weigh?  Why not modify the GD plan into Zurbrin's Compact Moon Direct scheme? let one of those early flight carry an O2 plant and make your own.  ta  Ralph\",\n",
       "  \"Why use a ground launch pad. It is entirely posible to launch from altitude. This was what the Shuttle was originally intended to do! It might be seriously cheaper.   Also, what about bio-engineered CO2 absorbing plants instead of many LOX bottles? Stick 'em in a lunar cave and put an airlock on the door. \",\n",
       "  '  The gravity maneuvering that was used was to exploit \\'fuzzy regions\\'.  These are described by the inventor as exploiting the second-order perturbations in a three body system.  The probe was launched into this region for the earth-moon-sun system, where the perturbations affected it in such a way as to allow it to go into lunar orbit without large expenditures of fuel to slow down. The idea is that \\'natural objects sometimes get captured without expending fuel, we\\'ll just find the trajectory that makes it possible\". The originator of the technique said that NASA wasn\\'t interested, but that Japan was because their probe was small and couldn\\'t hold a lot of fuel for deceleration.   This from an issue of \\'Science News\\' or \\'The Planetary Report\\' I believe, about 2 months ago(?). ',\n",
       "  'Archive-name: space/probe Last-modified: $Date: 93/04/01 14:39:19 $  PLANETARY PROBES - HISTORICAL MISSIONS      This section was lightly adapted from an original posting by Larry Klaes     (klaes@verga.enet.dec.com), mostly minor formatting changes. Matthew     Wiener (weemba@libra.wistar.upenn.edu) contributed the section on     Voyager, and the section on Sakigake was obtained from ISAS material     posted by Yoshiro Yamada (yamada@yscvax.ysc.go.jp).  US PLANETARY MISSIONS       MARINER (VENUS, MARS, & MERCURY FLYBYS AND ORBITERS)      MARINER 1, the first U.S. attempt to send a spacecraft to Venus, failed     minutes after launch in 1962. The guidance instructions from the ground     stopped reaching the rocket due to a problem with its antenna, so the     onboard computer took control. However, there turned out to be a bug in     the guidance software, and the rocket promptly went off course, so the     Range Safety Officer destroyed it. Although the bug is sometimes claimed     to have been an incorrect FORTRAN DO statement, it was actually a     transcription error in which the bar (indicating smoothing) was omitted     from the expression \"R-dot-bar sub n\" (nth smoothed value of derivative     of radius). This error led the software to treat normal minor variations     of velocity as if they were serious, leading to incorrect compensation.      MARINER 2 became the first successful probe to flyby Venus in December     of 1962, and it returned information which confirmed that Venus is a     very hot (800 degrees Fahrenheit, now revised to 900 degrees F.) world     with a cloud-covered atmosphere composed primarily of carbon dioxide     (sulfuric acid was later confirmed in 1978).      MARINER 3, launched on November 5, 1964, was lost when its protective     shroud failed to eject as the craft was placed into interplanetary     space. Unable to collect the Sun\\'s energy for power from its solar     panels, the probe soon died when its batteries ran out and is now in     solar orbit. It was intended for a Mars flyby with MARINER 4.      MARINER 4, the sister probe to MARINER 3, did reach Mars in 1965 and     took the first close-up images of the Martian surface (22 in all) as it     flew by the planet. The probe found a cratered world with an atmosphere     much thinner than previously thought. Many scientists concluded from     this preliminary scan that Mars was a \"dead\" world in both the     geological and biological sense.      MARINER 5 was sent to Venus in 1967. It reconfirmed the data on that     planet collected five years earlier by MARINER 2, plus the information     that Venus\\' atmospheric pressure at its surface is at least 90 times     that of Earth\\'s, or the equivalent of being 3,300 feet under the surface     of an ocean.      MARINER 6 and 7 were sent to Mars in 1969 and expanded upon the work     done by MARINER 4 four years earlier. However, they failed to take away     the concept of Mars as a \"dead\" planet, first made from the basic     measurements of MARINER 4.      MARINER 8 ended up in the Atlantic Ocean in 1971 when the rocket     launcher autopilot failed.      MARINER 9, the sister probe to MARINER 8, became the first craft to     orbit Mars in 1971. It returned information on the Red Planet that no     other probe had done before, revealing huge volcanoes on the Martian     surface, as well as giant canyon systems, and evidence that water once     flowed across the planet. The probe also took the first detailed closeup     images of Mars\\' two small moons, Phobos and Deimos.      MARINER 10 used Venus as a gravity assist to Mercury in 1974. The probe     did return the first close-up images of the Venusian atmosphere in     ultraviolet, revealing previously unseen details in the cloud cover,     plus the fact that the entire cloud system circles the planet in four     Earth days. MARINER 10 eventually made three flybys of Mercury from 1974     to 1975 before running out of attitude control gas. The probe revealed     Mercury as a heavily cratered world with a mass much greater than     thought. This would seem to indicate that Mercury has an iron core which     makes up 75 percent of the entire planet.       PIONEER (',\n",
       "  'That is an idea.. The most efficient moon habitat..   also the idea of how to get the people off the moon once the prize was won..  Also the idea of how to rescue someone who is \"dying\" on the moon.  Maybe have a area where they can all \"see\" each other, and can help each other if something happens..   I liek the idea of one prize for the first moon landing and return, by a non-governmental body..  Also the idea of then having a moon habitat race..   I know we need to do somthing to get people involved..  Eccentric millionaire/billionaire would be nice.. We see how old Ross feels about it.. After all it would be a great promotional thing and a way to show he does care about commericalization and the people.. Will try to broach the subject to him..   Moonbase on the cheap is a good idea.. NASA and friends seem to take to much time and give us to expensive stuff that of late does not work (hubble and such). Basically what is the difference between a $1mil peice of junk and a multi $1mil piece of junk.. I know junk..',\n",
       "  'With the continuin talk about the \"End of the Space Age\" and complaints  by government over the large cost, why not try something I read about  that might just work.  Announce that a reward of $1 billion would go to the first corporation  who successfully keeps at least 1 person alive on the moon for a year.  Then you\\'d see some of the inexpensive but not popular technologies begin  to be developed. THere\\'d be a different kind of space race then! ',\n",
       "  \"an image of the moon has been caught in a weather satellite images of the earth. it appears in both the 0430-1500UT ir and visual images of the earth. the GIF images can be down loaded from vmd.cso.uiuc.edu and are named CI043015.GIF and CV043015.GIF for the IR and visual images respectively.  pretty cool pictures;  in the ir it's saturated but in the visual image details on the moon are viewable.  the moon is not in the 1400UT images. \",\n",
       "  \"    It may be that they just didn't mention it, or that they actually haven't  thought about it.  I got the vague impression from their mission proposal that they weren't taking a very holistic aproach to the whole thing.  They seemed to want to land people on the Moon by the end of the decade without  explaining why, or what they would do once they got there.  The only application I remember from the Av Week article was placing a telescope on the Moon.  That's great, but they don't explain why it can't be done robotically.     But I'm a _member_.  Besides Bill, I hang out with you :)  \",\n",
       "  'From the article \"What\\'s New\" Apr-16-93 in sci.physics.research:  ........ WHAT\\'S NEW (in my opinion), Friday, 16 April 1993  Washington, DC  1. SPACE BILLBOARDS! IS THIS ONE THE \"SPINOFFS\" WE WERE PROMISED? In 1950, science fiction writer Robert Heinlein published \"The Man Who Sold the Moon,\" which involved a dispute over the sale of rights to the Moon for use as billboard. NASA has taken the firsteps toward this  hideous vision of the future.  Observers were startled this spring when a NASA launch vehicle arrived at the pad with \"SCHWARZENEGGER\" painted in huge block letters on the side of the booster rockets.  Space Marketing Inc. had arranged for the ad to promote Arnold\\'s latest movie. Now, Space Marketing is working with University of Colorado and Livermore engineers on a plan to place a mile-long inflatable billboard in low-earth orbit.  NASA would provide contractual launch services. However, since NASA bases its charge on seriously flawed cost estimates (WN 26 Mar 93) the taxpayers would bear most of the expense. This may look like environmental vandalism, but Mike Lawson, CEO of Space Marketing, told us yesterday that the real purpose of the project is to help the environment! The platform will carry ozone monitors he explained--advertising is just to help defray costs. ..........  What do you think of this revolting and hideous attempt to vandalize the night sky? It is not even April 1 anymore. What about light pollution in observations? (I read somewhere else that it might even be visible during the day, leave alone at night). Is NASA really supporting this junk? Are protesting groups being organized in the States? Really, really depressed.               Enzo',\n",
       "  'I have often thought about, if its possible to have a powerfull laser on earth, to light at the Moon, and show lasergraphics at the surface so clearly that you can see it with your eyes when there is a new moon.  How about a Coca Cola logo at the moon, easy way to target billions of people.  Do you know if its possible?  ',\n",
       "  'At one time there was speculation that the first spacewalk  (Alexei Leonov ?) was a staged fake.  Has any evidence to support or contradict this claim emerged ?  Was this claim perhaps another fevered Cold War hallucination ? ',\n",
       "  'From the \"JPL Universe\" April 23, 1993  VLBI project meets with international space agencies',\n",
       "  ' Think for a moment about the technology required to do that.  By  the time they could make the Earth\\'s sky look like Las Vegas,  the people could afford to go backpacking on the Moon. Round trip costs for 500 kg to the Moon would be about the same as 5000 kg in a Low Earth \"advertising\" orbit: Very roughly the same cost as a smallish billboard, therefore. If such ads were to become common place, that would have to be a very low price...  The night sky on a Lunar backpacking trip would still be very  pristine...   There\\'s always been a problem of having to get  away from civilization before you can really find \"natural\" scenery. 100 years ago, this usually didn\\'t take a trip of over 5 miles. Today, most people would have to go 100 miles or more. If we ever get to the point where we have billboards on orbit, that essentially means that no place on Earth is still \"wild.\" While that may or may not be a good thing, the orbital billboards aren\\'t the problem: They are just a symptom of  growing, densely-populated civilization. Banning such ads will not save your view of the night sky, because by the time  such ads could become widespread you will probably have trouble finding a place without street lights, where you can _see_ the stars...    An ad on a moon of Jupiter would be rather pointless, since you need a telescope to see them. However, I\\'d love to see them get all the publicity they could from underwritting the \"Coca Cola Io Orbital Mapping Probe.\"     They already can, to some extent: The IAU allows names derived from sponsors or patrons of scientific research. If Microscum donates money to a university astronomy program, one of the galactic  astronomers could easily get a newly discovered galaxy named after them.',\n",
       "  ';From the article \"What\\'s New\" Apr-16-93 in sci.physics.research: ; ;........ ;WHAT\\'S NEW (in my opinion), Friday, 16 April 1993  Washington, DC ; ;1. SPACE BILLBOARDS! IS THIS ONE THE \"SPINOFFS\" WE WERE PROMISED? ;What about light pollution in observations? (I read somewhere else that ;it might even be visible during the day, leave alone at night). ;Is NASA really supporting this junk? ;Are protesting groups being organized in the States? ;Really, really depressed. ; ;             Enzo  I wouldn\\'t worry about it.  There\\'s enough space debris up there that a mile-long inflatable would probably deflate in some very short period of time (less than a year) while cleaning up LEO somewhat. Sort of a giant fly-paper in orbit.  Hmm, that could actually be useful.  As for advertising -- sure, why not?  A NASA friend and I spent one drunken night figuring out just exactly how much gold mylar we\\'d need to put the golden arches of a certain American fast food organization on the face of the Moon.  Fortunately, we sobered up in the morning.'],\n",
       " [258,\n",
       "  535,\n",
       "  357,\n",
       "  321,\n",
       "  102,\n",
       "  165,\n",
       "  458,\n",
       "  323,\n",
       "  50,\n",
       "  344,\n",
       "  545,\n",
       "  571,\n",
       "  640,\n",
       "  532,\n",
       "  143,\n",
       "  171,\n",
       "  312,\n",
       "  427,\n",
       "  134,\n",
       "  141])"
      ]
     },
     "execution_count": 145,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.pprompt(\"Which information on the keyword 'moon landing' does topic 13 have?\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 146,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      " Their Hiten engineering-test mission spent a while in a highly eccentric Earth orbit doing lunar flybys, and then was inserted into lunar orbit using some very tricky gravity-assist-like maneuvering.  This meant that it would crash on the Moon eventually, since there is no such thing as a stable lunar orbit (as far as anyone knows), and I believe I recall hearing recently that it was about to happen.\n"
     ]
    }
   ],
   "source": [
    "print(tm.topic_lis[13].documents[102])"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 147,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT wants to the call the function:  {\n",
      "  \"name\": \"split_topic_kmeans\",\n",
      "  \"arguments\": \"{\\n  \\\"topic_idx\\\": 6,\\n  \\\"n_clusters\\\": 5\\n}\"\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:01<00:00,  1.30s/it]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:03]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "100%|██████████| 1/1 [00:04<00:00,  4.34s/it]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:00<00:00,  2.93it/s]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:05]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "100%|██████████| 1/1 [00:05<00:00,  5.68s/it]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:00]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:00<00:00,  4.42it/s]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "100%|██████████| 1/1 [00:03<00:00,  3.94s/it]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:00]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:00<00:00,  8.14it/s]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "100%|██████████| 1/1 [00:04<00:00,  4.51s/it]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:00<00:00, 14.76it/s]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:00]\n",
      "100%|██████████| 1/1 [00:04<00:00,  4.09s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Here are five potential subtopics of topic 6:\n",
      "\n",
      "1. Existence of God: This subtopic involves questioning the existence of God and examining the evidence for and against it.\n",
      "\n",
      "2. Sexual Orientation: This subtopic relates to homosexuality and encompasses aspects such as sexual orientation, rights and discrimination, social attitudes, relationships and partners, and public perception.\n",
      "\n",
      "3. Ethics and Morality: This subtopic focuses on moral and ethical principles, including moral philosophy, moral reasoning, moral standards, moral dilemmas, and moral relativism.\n",
      "\n",
      "4. Religion and Law: This subtopic explores the intersection of religion and law, including beliefs, practices, interpretation, controversies, and the role of religion in society and politics.\n",
      "\n",
      "5. Argumentation and Atheism: This subtopic revolves around debates and arguments, involving communication, logical reasoning, disagreements, the intersection of religion and atheism, and the criticism and analysis of arguments.\n",
      "\n",
      "Please note that these subtopics are inferred from the information retrieved from the topic modelling results.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Topic 0: Electronics Equipment Sales,\n",
       " Topic 1: Image Processing,\n",
       " Topic 2: Gun control,\n",
       " Topic 3: Online Privacy and Anonymity,\n",
       " Topic 4: Conflict and Violence.,\n",
       " Topic 5: Computer Hardware,\n",
       " Topic 6: Online Discussions,\n",
       " Topic 7: Computer Software,\n",
       " Topic 8: Car Features and Performance,\n",
       " Topic 9: Encryption and Government,\n",
       " Topic 10: Technology and Computing.,\n",
       " Topic 11: Technology and Computing,\n",
       " Topic 12: Space Exploration,\n",
       " Topic 13: Motorcycle Riding Techniques,\n",
       " Topic 14: Technology,\n",
       " Topic 15: Hockey Games,\n",
       " Topic 16: Health and Medicine.,\n",
       " Topic 17: Baseball games and teams.,\n",
       " Topic 18: Beliefs about Homosexuality.,\n",
       " Topic 19: Existence of God,\n",
       " Topic 20: Sexual Orientation,\n",
       " Topic 21: Ethics and Morality,\n",
       " Topic 22: Religion and Law,\n",
       " Topic 23: Argumentation and Atheism.]"
      ]
     },
     "execution_count": 147,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.pprompt(\"What are 5 potential subtopics of topic 6\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Topic splitting"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Based on the previously identified topics, we decide to split topic 6 not into 5 but into three subtopics based on the keywords 'religious faith', 'atheism' and 'ethics and philosophy'."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 148,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT wants to the call the function:  {\n",
      "  \"name\": \"split_topic_keywords\",\n",
      "  \"arguments\": \"{\\n  \\\"topic_idx\\\": 6,\\n  \\\"keywords\\\": [\\\"religious faith\\\", \\\"atheism\\\", \\\"ethics and philosophy\\\"],\\n  \\\"inplace\\\": true\\n}\"\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 100/100 [00:00]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:00<00:00, 12.43it/s]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "100%|██████████| 1/1 [00:04<00:00,  4.88s/it]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:00]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:00<00:00,  1.42it/s]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:03]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "100%|██████████| 1/1 [00:06<00:00,  6.45s/it]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:00]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:00<00:00,  2.04it/s]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:02]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "100%|██████████| 1/1 [00:04<00:00,  4.19s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "[Topic 0: Electronics Equipment Sales\n",
      ", Topic 1: Image Processing\n",
      ", Topic 2: Gun control\n",
      ", Topic 3: Online Privacy and Anonymity\n",
      ", Topic 4: Conflict and Violence.\n",
      ", Topic 5: Computer Hardware\n",
      ", Topic 6: Online Discussions\n",
      ", Topic 7: Computer Software\n",
      ", Topic 8: Car Features and Performance\n",
      ", Topic 9: Encryption and Government\n",
      ", Topic 10: Technology and Computing.\n",
      ", Topic 11: Technology and Computing\n",
      ", Topic 12: Space Exploration\n",
      ", Topic 13: Motorcycle Riding Techniques\n",
      ", Topic 14: Technology\n",
      ", Topic 15: Hockey Games\n",
      ", Topic 16: Health and Medicine.\n",
      ", Topic 17: Baseball games and teams.\n",
      ", Topic 18: Beliefs about Homosexuality.\n",
      ", Topic 19: Religious Beliefs\n",
      ", Topic 20: Existence of God\n",
      ", Topic 21: Ethics and Morality\n",
      "]\n",
      "Topic 6 has been split into the following subtopics based on the keywords 'religious faith', 'atheism', and 'ethics and philosophy':\n",
      "\n",
      "1. Subtopic: Religious Beliefs\n",
      "   - Description: The common topic of these words is \"Religion and Beliefs\". Aspects and sub-topics of this topic include faith and belief, religious practices, interpretation and understanding, controversies and disagreements, and the impact on society.\n",
      "   - Topic index: 19\n",
      "\n",
      "2. Subtopic: Existence of God\n",
      "   - Description: The common topic of the provided words is \"Atheism and Belief\". Aspects and sub-topics of this topic include atheist beliefs, atheist arguments, atheist perspective, atheist skepticism, and atheist criticism.\n",
      "   - Topic index: 20\n",
      "\n",
      "3. Subtopic: Ethics and Morality\n",
      "   - Description: The common topic of the given words appears to be \"Morality and Ethics\". Various aspects and sub-topics related to this topic include moral reasoning and arguments, ethical dilemmas and moral judgments, objective vs subjective morality, homosexuality and sexual ethics, and science and morality.\n",
      "   - Topic index: 21\n",
      "\n",
      "Please note that these subtopics have been created based on the provided keywords and may not capture all aspects related to the keywords.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Topic 0: Electronics Equipment Sales,\n",
       " Topic 1: Image Processing,\n",
       " Topic 2: Gun control,\n",
       " Topic 3: Online Privacy and Anonymity,\n",
       " Topic 4: Conflict and Violence.,\n",
       " Topic 5: Computer Hardware,\n",
       " Topic 6: Online Discussions,\n",
       " Topic 7: Computer Software,\n",
       " Topic 8: Car Features and Performance,\n",
       " Topic 9: Encryption and Government,\n",
       " Topic 10: Technology and Computing.,\n",
       " Topic 11: Technology and Computing,\n",
       " Topic 12: Space Exploration,\n",
       " Topic 13: Motorcycle Riding Techniques,\n",
       " Topic 14: Technology,\n",
       " Topic 15: Hockey Games,\n",
       " Topic 16: Health and Medicine.,\n",
       " Topic 17: Baseball games and teams.,\n",
       " Topic 18: Beliefs about Homosexuality.,\n",
       " Topic 19: Religious Beliefs,\n",
       " Topic 20: Existence of God,\n",
       " Topic 21: Ethics and Morality]"
      ]
     },
     "execution_count": 148,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.pprompt(\"Please split topic 6 into subtopics based on the keywords 'religious faith', 'atheism' and 'ethics and philosophy'. Do this inplace.\")"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 150,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[Topic 0: Electronics Equipment Sales,\n",
       " Topic 1: Image Processing,\n",
       " Topic 2: Gun control,\n",
       " Topic 3: Online Privacy and Anonymity,\n",
       " Topic 4: Conflict and Violence.,\n",
       " Topic 5: Computer Hardware,\n",
       " Topic 6: Online Discussions,\n",
       " Topic 7: Computer Software,\n",
       " Topic 8: Car Features and Performance,\n",
       " Topic 9: Encryption and Government,\n",
       " Topic 10: Technology and Computing.,\n",
       " Topic 11: Technology and Computing,\n",
       " Topic 12: Space Exploration,\n",
       " Topic 13: Motorcycle Riding Techniques,\n",
       " Topic 14: Technology,\n",
       " Topic 15: Hockey Games,\n",
       " Topic 16: Health and Medicine.,\n",
       " Topic 17: Baseball games and teams.,\n",
       " Topic 18: Beliefs about Homosexuality.,\n",
       " Topic 19: Religious Beliefs,\n",
       " Topic 20: Existence of God,\n",
       " Topic 21: Ethics and Morality]"
      ]
     },
     "execution_count": 150,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.topic_lis"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Combine Topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Topics 15 and 17 both seem to be about sports, so let's merge them into one topic."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 153,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT wants to the call the function:  {\n",
      "  \"name\": \"combine_topics\",\n",
      "  \"arguments\": \"{\\n  \\\"topic_idx_lis\\\": [15, 17],\\n  \\\"inplace\\\": true\\n}\"\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Computing word-topic matrix: 100%|██████████| 1/1 [00:07<00:00,  7.16s/it]\n",
      "Epochs completed: 100%| ██████████ 30/30 [00:09]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:02]\n",
      "100%|██████████| 1/1 [00:06<00:00,  6.62s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The topics 15 and 17 have been combined into a new topic called \"Sports\". This topic includes aspects and sub-topics related to sports such as team and players, games and seasons, performance and skills, fans and audience, and statistics and records. Some of the common words found in this topic include \"team,\" \"players,\" \"hockey,\" \"baseball,\" \"game,\" \"games,\" \"season,\" \"playoffs,\" \"good,\" \"better,\" \"win,\" \"hit,\" \"score,\" \"fans,\" \"series,\" \"watch,\" \"fan,\" \"stats,\" \"record,\" \"pts,\" and \"career\".\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Topic 0: Electronics Equipment Sales,\n",
       " Topic 1: Image Processing,\n",
       " Topic 2: Gun control,\n",
       " Topic 3: Online Privacy and Anonymity,\n",
       " Topic 4: Conflict and Violence.,\n",
       " Topic 5: Computer Hardware,\n",
       " Topic 6: Online Discussions,\n",
       " Topic 7: Computer Software,\n",
       " Topic 8: Car Features and Performance,\n",
       " Topic 9: Encryption and Government,\n",
       " Topic 10: Technology and Computing.,\n",
       " Topic 11: Technology and Computing,\n",
       " Topic 12: Space Exploration,\n",
       " Topic 13: Motorcycle Riding Techniques,\n",
       " Topic 14: Technology,\n",
       " Topic 15: Health and Medicine.,\n",
       " Topic 16: Beliefs about Homosexuality.,\n",
       " Topic 17: Religious Beliefs,\n",
       " Topic 18: Existence of God,\n",
       " Topic 19: Ethics and Morality,\n",
       " Topic 20: Sports]"
      ]
     },
     "execution_count": 153,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.pprompt(\"Please combine topics 15 and 17. Do this inplace.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Delete Topics"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Since Topic 10 and 11 have the same title, we can combine them into one topic. Note that this doesn't delete the documents from the delted topic, but rather distributes them over the other topics."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 159,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT wants to the call the function:  {\n",
      "  \"name\": \"delete_topic\",\n",
      "  \"arguments\": \"{\\n  \\\"topic_idx\\\": 10,\\n  \\\"inplace\\\": true\\n}\"\n",
      "}\n",
      "Tue Sep  5 13:50:48 2023 Building and compiling search function\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 100/100 [00:02]\n",
      "Computing word-topic matrix: 100%|██████████| 20/20 [02:01<00:00,  6.06s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Shape of tfidf:  (31365, 20)\n",
      "shape fo word_topic_mat:  (31365, 20)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 30/30 [00:05]\n",
      "100%|██████████| 20/20 [01:43<00:00,  5.17s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "The topic with index 10 has been successfully deleted. \n",
      "\n",
      "After removing topic 10, the new topic we have is topic with index 19, which is related to \"Sports\". The various aspects and sub-topics of this topic include:\n",
      "\n",
      "1. Games: \"game\", \"games\", \"play\", \"team\", \"players\"\n",
      "2. Seasons: \"year\", \"season\", \"last\", \"years\", \"playoffs\"\n",
      "3. Performance: \"good\", \"better\", \"well\", \"great\", \"average\"\n",
      "4. Strategies: \"think\", \"strategy\", \"tactics\", \"coach\", \"plan\"\n",
      "5. Results: \"win\", \"score\", \"goal\", \"points\", \"victory\"\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Topic 0: Electronics equipment sales,\n",
       " Topic 1: Image Processing,\n",
       " Topic 2: Gun control,\n",
       " Topic 3: Online Privacy,\n",
       " Topic 4: Conflict and violence.,\n",
       " Topic 5: Computer Hardware,\n",
       " Topic 6: Anonymity in online discussions.,\n",
       " Topic 7: Computer Software,\n",
       " Topic 8: Car Features and Performance,\n",
       " Topic 9: Encryption,\n",
       " Topic 10: Technology and Computing,\n",
       " Topic 11: Space Exploration,\n",
       " Topic 12: Motorcycle Riding Tips,\n",
       " Topic 13: Technology and Computing,\n",
       " Topic 14: Healthcare and Medicine.,\n",
       " Topic 15: Biblical interpretation,\n",
       " Topic 16: Religious Beliefs,\n",
       " Topic 17: Existence of God,\n",
       " Topic 18: Morality in Health Insurance,\n",
       " Topic 19: Sports]"
      ]
     },
     "execution_count": 159,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.pprompt(\"Please delete topic 10. Do this inplace.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Compare Topics "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 160,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[nltk_data] Downloading package stopwords to\n",
      "[nltk_data]     C:\\Users\\arik_\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package stopwords is already up-to-date!\n",
      "[nltk_data] Downloading package punkt to\n",
      "[nltk_data]     C:\\Users\\arik_\\AppData\\Roaming\\nltk_data...\n",
      "[nltk_data]   Package punkt is already up-to-date!\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT wants to the call the function:  {\n",
      "  \"name\": \"get_topic_information\",\n",
      "  \"arguments\": \"{\\n  \\\"topic_idx_lis\\\": [5, 7]\\n}\"\n",
      "}\n",
      "Topic 5 is about computer hardware, while topic 7 is about computer software.\n",
      "\n",
      "In topic 5, the common topic is \"computer hardware,\" and it covers various aspects and sub-topics such as storage, components, performance, connectivity, and display. Some of the top words in this topic include \"drive,\" \"card,\" \"disk,\" \"memory,\" and \"video.\" It seems to be discussing various hardware-related issues, including problems, compatibility, performance, and configuration. It also mentions specific components like hard drives, cards, motherboards, and monitors. Overall, this topic focuses on the physical components of a computer system.\n",
      "\n",
      "On the other hand, topic 7 is about computer software and usage. It covers aspects such as operating systems, software programs, computer hardware, user interface, and troubleshooting. Some of the top words in this topic include \"file,\" \"program,\" \"windows,\" \"software,\" and \"run.\" It seems to discuss different software-related topics, including file management, application installation, error messages, and system performance. It also mentions specific software programs like file utilities and font management. This topic focuses on the software aspect of using a computer system.\n",
      "\n",
      "In summary, topic 5 focuses on computer hardware, while topic 7 focuses on computer software. They cover different aspects but are both related to the functioning and usage of a computer system.\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "{5: '\\n            Topic index: 5\\n            Topic name: Computer Hardware\\n            Topic description: The common topic of the given words is \"computer hardware\". \\n\\nThe various aspects and sub-topics of this topic include:\\n1. Storage: disk, hard drive, floppy, drives, disks.\\n2. Components: card, controller, board, chip, motherboard.\\n3. Performance: speed, memory, clock, mhz, faster.\\n4. Connectivity: bus, port, connector, cable, serial.\\n5. Display: monitor, video, screen, color, graphics.\\n            Topic topwords: [\"n\\'t\", \\'drive\\', \\'card\\', \\'one\\', \\'would\\', \\'use\\', \\'know\\', \\'get\\', \\'like\\', \\'disk\\', \\'system\\', \\'problem\\', \\'drives\\', \\'work\\', \\'also\\', \\'controller\\', \\'hard\\', \\'anyone\\', \\'using\\', \\'drivers\\', \\'need\\', \\'two\\', \\'monitor\\', \\'bus\\', \\'new\\', \\'used\\', \\'software\\', \\'speed\\', \\'data\\', \\'could\\', \\'think\\', \\'driver\\', \\'memory\\', \\'board\\', \\'time\\', \\'problems\\', \\'mode\\', \\'video\\', \\'port\\', \\'good\\', \\'much\\', \\'cards\\', \\'computer\\', \\'machine\\', \\'run\\', \\'modem\\', \\'want\\', \\'may\\', \\'chip\\', \\'got\\', \\'hardware\\', \\'floppy\\', \\'help\\', \\'support\\', \\'really\\', \\'even\\', \\'set\\', \\'motherboard\\', \\'power\\', \\'well\\', \\'price\\', \\'make\\', \\'cable\\', \\'works\\', \\'way\\', \\'tape\\', \\'better\\', \\'program\\', \\'faster\\', \\'back\\', \\'etc\\', \\'people\\', \\'interface\\', \\'standard\\', \\'see\\', \\'still\\', \\'since\\', \\'version\\', \\'available\\', \\'performance\\', \\'running\\', \\'buy\\', \\'printer\\', \\'read\\', \\'sure\\', \\'bit\\', \\'serial\\', \\'able\\', \\'internal\\', \\'machines\\', \\'different\\', \\'fast\\', \\'information\\', \\'something\\', \\'might\\', \\'right\\', \\'info\\', \\'question\\', \\'first\\', \\'screen\\', \\'fine\\', \\'another\\', \\'color\\', \\'many\\', \\'number\\', \\'please\\', \\'advance\\', \\'find\\', \\'getting\\', \\'though\\', \\'thing\\', \\'clock\\', \\'seems\\', \\'going\\', \\'seen\\', \\'looking\\', \\'mhz\\', \\'tried\\', \\'someone\\', \\'connector\\', \\'possible\\', \\'tell\\', \\'without\\', \\'anything\\', \\'uses\\', \\'transfer\\', \\'old\\', \\'bought\\', \\'must\\', \\'heard\\', \\'external\\', \\'either\\', \\'devices\\', \\'appreciated\\', \\'try\\', \\'pin\\', \\'sound\\', \\'second\\', \\'around\\', \\'come\\', \\'cache\\', \\'put\\', \\'least\\', \\'chips\\', \\'upgrade\\', \\'probably\\', \\'disks\\', \\'following\\', \\'high\\', \\'send\\', \\'rate\\', \\'local\\', \\'supports\\', \\'done\\', \\'called\\', \\'say\\', \\'installed\\', \\'monitors\\', \\'access\\', \\'bad\\', \\'type\\', \\'ports\\', \\'line\\', \\'windows\\', \\'graphics\\', \\'cost\\', \\'end\\', \\'last\\', \\'email\\', \\'switch\\', \\'lot\\', \\'little\\', \\'anybody\\', \\'adapter\\', \\'mail\\', \\'found\\', \\'give\\', \\'systems\\', \\'take\\', \\'boot\\', \\'device\\', \\'quality\\', \\'kind\\', \\'things\\', \\'meg\\', \\'file\\', \\'trying\\', \\'never\\', \\'boards\\', \\'order\\', \\'case\\', \\'look\\', \\'original\\', \\'comes\\', \\'display\\', \\'keyboard\\', \\'said\\', \\'mouse\\', \\'address\\', \\'best\\', \\'jumper\\', \\'recently\\', \\'via\\', \\'far\\', \\'post\\', \\'long\\', \\'seem\\', \\'believe\\', \\'pins\\', \\'change\\', \\'experience\\', \\'difference\\', \\'mac\\', \\'allow\\', \\'based\\', \\'add\\', \\'interested\\', \\'processor\\', \\'printers\\', \\'connect\\', \\'modes\\', \\'says\\', \\'times\\', \\'else\\', \\'control\\', \\'less\\', \\'pretty\\', \\'several\\', \\'working\\', \\'e-mail\\', \\'true\\', \\'extra\\', \\'thanks\\', \\'write\\', \\'jumpers\\', \\'ago\\', \\'full\\', \\'real\\', \\'quite\\', \\'low\\', \\'actually\\', \\'results\\', \\'let\\', \\'phone\\', \\'list\\', \\'check\\', \\'every\\', \\'means\\', \\'supply\\', \\'programs\\', \\'thought\\', \\'lines\\', \\'colors\\', \\'speeds\\', \\'large\\', \\'slow\\', \\'others\\', \\'higher\\', \\'backup\\', \\'heads\\', \\'told\\', \\'directly\\', \\'due\\', \\'three\\', \\'stuff\\', \\'print\\', \\'place\\', \\'per\\', \\'needs\\', \\'slave\\', \\'great\\', \\'unless\\', \\'buying\\', \\'already\\', \\'format\\', \\'prices\\', \\'slot\\', \\'built-in\\', \\'interrupt\\', \\'model\\', \\'part\\', \\'nice\\', \\'compatible\\', \\'everything\\', \\'yet\\', \\'answer\\', \\'size\\', \\'made\\', \\'point\\', \\'enough\\', \\'months\\', \\'except\\', \\'hear\\', \\'configuration\\', \\'note\\', \\'laser\\', \\'reason\\', \\'horizontal\\', \\'came\\', \\'numbers\\', \\'side\\', \\'makes\\', \\'setup\\', \\'however\\', \\'resolution\\', \\'error\\', \\'similar\\', \\'correct\\', \\'course\\', \\'couple\\', \\'single\\', \\'questions\\', \\'article\\', \\'guess\\', \\'types\\', \\'applications\\', \\'cables\\', \\'slower\\', \\'goes\\', \\'expensive\\', \\'ink\\', \\'usually\\', \\'start\\', \\'idea\\', \\'paper\\', \\'output\\', \\'solution\\', \\'wrong\\', \\'needed\\', \\'computers\\', \\'socket\\', \\'present\\', \\'allows\\', \\'plug\\', \\'runs\\', \\'maybe\\', \\'deal\\', \\'money\\', \\'settings\\', \\'open\\', \\'install\\', \\'year\\', \\'reply\\', \\'slots\\', \\'mean\\', \\'setting\\', \\'normal\\', \\'instead\\', \\'manual\\', \\'accelerator\\', \\'handle\\', \\'left\\', \\'nothing\\', \\'sell\\', \\'current\\', \\'files\\', \\'cheap\\', \\'service\\', \\'thinking\\', \\'greatly\\', \\'always\\', \\'market\\', \\'mine\\', \\'level\\', \\'errors\\', \\'ftp\\', \\'code\\', \\'purchased\\', \\'box\\', \\'user\\', \\'cheaper\\', \\'often\\', \\'requires\\', \\'include\\', \\'vertical\\', \\'optional\\', \\'burst\\', \\'wondering\\', \\'comments\\', \\'master\\', \\'worth\\', \\'main\\', \\'purchase\\', \\'feature\\', \\'years\\', \\'asked\\', \\'currently\\', \\'whether\\', \\'test\\', \\'wide\\', \\'day\\', \\'call\\', \\'example\\', \\'friend\\', \\'ones\\', \\'soon\\', \\'plus\\', \\'copy\\', \\'clear\\', \\'keep\\', \\'formatting\\', \\'free\\', \\'image\\', \\'cause\\', \\'dealer\\', \\'big\\', \\'gives\\', \\'small\\', \\'includes\\', \\'although\\', \\'maximum\\', \\'controllers\\', \\'products\\', \\'reading\\', \\'option\\', \\'provide\\', \\'inches\\', \\'almost\\', \\'rates\\', \\'built\\', \\'company\\', \\'ever\\', \\'common\\', \\'together\\', \\'name\\', \\'properly\\', \\'week\\', \\'cylinders\\', \\'talking\\', \\'section\\', \\'asynchronous\\', \\'latest\\', \\'advice\\', \\'product\\', \\'connected\\', \\'compared\\', \\'mentioned\\', \\'wait\\', \\'message\\', \\'weeks\\', \\'trouble\\', \\'newer\\', \\'third\\', \\'special\\', \\'older\\', \\'looks\\', \\'included\\', \\'hope\\', \\'signal\\', \\'features\\', \\'next\\', \\'ask\\', \\'appreciate\\', \\'transfers\\', \\'details\\', \\'bits\\', \\'bytes\\', \\'rather\\', \\'including\\', \\'fact\\', \\'parity\\', \\'hook\\', \\'general\\', \\'expansion\\', \\'require\\', \\'especially\\', \\'gets\\', \\'coprocessor\\', \\'home\\', \\'net\\', \\'operational\\', \\'sometimes\\', \\'future\\', \\'parallel\\', \\'sent\\', \\'figure\\', \\'buffer\\', \\'versions\\', \\'lower\\', \\'ideas\\', \\'top\\', \\'wires\\', \\'range\\', \\'later\\', \\'package\\', \\'went\\', \\'ram\\', \\'exactly\\', \\'share\\', \\'issue\\', \\'four\\', \\'chipset\\', \\'printing\\', \\'group\\', \\'result\\', \\'supported\\', \\'choice\\', \\'tower\\', \\'happens\\', \\'written\\']',\n",
       " 7: '\\n            Topic index: 7\\n            Topic name: Computer Software\\n            Topic description: The common topic of the given words is \"computer software and usage\". \\n\\nVarious aspects and sub-topics of this topic include:\\n1. Operating systems: Windows, DOS, virtual machines.\\n2. Software programs: file management, disk utilities, font management, application installation.\\n3. Computer hardware: memory allocation, graphics cards, printers.\\n4. User interface: command line, window display, mouse and keyboard.\\n5. Troubleshooting: error messages, software crashes, system performance.\\n            Topic topwords: [\"n\\'t\", \\'file\\', \\'use\\', \\'program\\', \\'windows\\', \\'would\\', \\'one\\', \\'get\\', \\'like\\', \\'disk\\', \\'files\\', \\'know\\', \\'run\\', \\'using\\', \\'fonts\\', \\'time\\', \\'software\\', \\'think\\', \\'system\\', \\'problem\\', \\'also\\', \\'people\\', \\'copy\\', \\'computer\\', \\'help\\', \\'version\\', \\'good\\', \\'font\\', \\'running\\', \\'way\\', \\'make\\', \\'anyone\\', \\'work\\', \\'memory\\', \\'need\\', \\'much\\', \\'support\\', \\'new\\', \\'programs\\', \\'users\\', \\'even\\', \\'available\\', \\'drive\\', \\'set\\', \\'user\\', \\'want\\', \\'could\\', \\'used\\', \\'something\\', \\'etc\\', \\'see\\', \\'information\\', \\'better\\', \\'driver\\', \\'really\\', \\'higher\\', \\'may\\', \\'many\\', \\'hard\\', \\'find\\', \\'without\\', \\'machine\\', \\'apps\\', \\'well\\', \\'applications\\', \\'still\\', \\'two\\', \\'since\\', \\'protection\\', \\'data\\', \\'info\\', \\'got\\', \\'directory\\', \\'application\\', \\'keyboard\\', \\'swap\\', \\'thing\\', \\'allocation\\', \\'unit\\', \\'problems\\', \\'first\\', \\'change\\', \\'printer\\', \\'read\\', \\'say\\', \\'access\\', \\'mode\\', \\'screen\\', \\'number\\', \\'able\\', \\'linked\\', \\'mouse\\', \\'installed\\', \\'cross\\', \\'product\\', \\'things\\', \\'sure\\', \\'code\\', \\'long\\', \\'lot\\', \\'space\\', \\'hacker\\', \\'ftp\\', \\'going\\', \\'around\\', \\'take\\', \\'try\\', \\'install\\', \\'either\\', \\'point\\', \\'seen\\', \\'different\\', \\'least\\', \\'course\\', \\'though\\', \\'comes\\', \\'right\\', \\'else\\', \\'never\\', \\'full\\', \\'character\\', \\'tell\\', \\'probably\\', \\'seems\\', \\'command\\', \\'window\\', \\'look\\', \\'anything\\', \\'card\\', \\'found\\', \\'copies\\', \\'text\\', \\'compatible\\', \\'manager\\', \\'last\\', \\'every\\', \\'server\\', \\'runs\\', \\'part\\', \\'stuff\\', \\'box\\', \\'free\\', \\'upgrade\\', \\'create\\', \\'type\\', \\'question\\', \\'graphics\\', \\'products\\', \\'enough\\', \\'size\\', \\'original\\', \\'mail\\', \\'error\\', \\'floppy\\', \\'drivers\\', \\'bit\\', \\'another\\', \\'disks\\', \\'play\\', \\'give\\', \\'systems\\', \\'name\\', \\'yet\\', \\'easy\\', \\'mean\\', \\'recommended\\', \\'line\\', \\'little\\', \\'characters\\', \\'load\\', \\'someone\\', \\'message\\', \\'utility\\', \\'database\\', \\'large\\', \\'back\\', \\'sound\\', \\'standard\\', \\'done\\', \\'must\\', \\'working\\', \\'actually\\', \\'icon\\', \\'believe\\', \\'hardware\\', \\'came\\', \\'fine\\', \\'virtual\\', \\'machines\\', \\'possible\\', \\'key\\', \\'network\\', \\'might\\', \\'real\\', \\'dos\\', \\'best\\', \\'write\\', \\'typing\\', \\'come\\', \\'works\\', \\'called\\', \\'quite\\', \\'bad\\', \\'true\\', \\'environment\\', \\'pretty\\', \\'setup\\', \\'color\\', \\'wrong\\', \\'needed\\', \\'gets\\', \\'keep\\', \\'please\\', \\'later\\', \\'compressed\\', \\'major\\', \\'far\\', \\'print\\', \\'versions\\', \\'getting\\', \\'looking\\', \\'included\\', \\'email\\', \\'means\\', \\'package\\', \\'less\\', \\'always\\', \\'uses\\', \\'idea\\', \\'installation\\', \\'display\\', \\'next\\', \\'based\\', \\'said\\', \\'reading\\', \\'compiler\\', \\'post\\', \\'makes\\', \\'including\\', \\'article\\', \\'temp\\', \\'marketing\\', \\'trying\\', \\'old\\', \\'features\\', \\'world\\', \\'address\\', \\'made\\', \\'seem\\', \\'year\\', \\'example\\', \\'already\\', \\'faster\\', \\'start\\', \\'fact\\', \\'sort\\', \\'group\\', \\'times\\', \\'heard\\', \\'break\\', \\'provides\\', \\'says\\', \\'company\\', \\'often\\', \\'couple\\', \\'amount\\', \\'utilities\\', \\'everything\\', \\'person\\', \\'button\\', \\'device\\', \\'within\\', \\'remember\\', \\'money\\', \\'includes\\', \\'however\\', \\'put\\', \\'manual\\', \\'send\\', \\'great\\', \\'tried\\', \\'stop\\', \\'similar\\', \\'cview\\', \\'port\\', \\'crash\\', \\'maybe\\', \\'second\\', \\'via\\', \\'shell\\', \\'sales\\', \\'operating\\', \\'rather\\', \\'exercises\\', \\'several\\', \\'megs\\', \\'general\\', \\'normal\\', \\'years\\', \\'word\\', \\'writing\\', \\'results\\', \\'let\\', \\'hackers\\', \\'programming\\', \\'written\\', \\'allow\\', \\'game\\', \\'open\\', \\'goes\\', \\'bytes\\', \\'call\\', \\'various\\', \\'others\\', \\'tools\\', \\'small\\', \\'check\\', \\'guess\\', \\'supported\\', \\'clients\\', \\'takes\\', \\'difference\\', \\'performance\\', \\'wait\\', \\'correct\\', \\'posted\\', \\'library\\', \\'registration\\', \\'schemes\\', \\'order\\', \\'case\\', \\'interface\\', \\'anyway\\', \\'names\\', \\'legitimate\\', \\'following\\', \\'supposed\\', \\'inferior\\', \\'instance\\', \\'backup\\', \\'whatever\\', \\'document\\', \\'told\\', \\'simply\\', \\'local\\', \\'compression\\', \\'batch\\', \\'matter\\', \\'price\\', \\'instead\\', \\'pirates\\', \\'control\\', \\'resolution\\', \\'hand\\', \\'top\\', \\'certain\\', \\'wondering\\', \\'config.sys\\', \\'computers\\', \\'started\\', \\'unless\\', \\'value\\', \\'day\\', \\'useful\\', \\'choose\\', \\'companies\\', \\'section\\', \\'big\\', \\'ever\\', \\'ones\\', \\'worth\\', \\'slow\\', \\'expect\\', \\'required\\', \\'end\\', \\'enhanced\\', \\'answer\\', \\'thought\\', \\'client\\', \\'format\\', \\'kind\\', \\'high\\', \\'ago\\', \\'needs\\', \\'certainly\\', \\'almost\\', \\'opinions\\', \\'reason\\', \\'worked\\', \\'language\\', \\'e-mail\\', \\'programmers\\', \\'appreciated\\', \\'archive\\', \\'completely\\', \\'interesting\\', \\'cluster\\', \\'project\\', \\'option\\', \\'future\\', \\'nice\\', \\'usually\\', \\'important\\', \\'groups\\', \\'patch\\', \\'recently\\', \\'special\\', \\'multiple\\', \\'keys\\', \\'provide\\', \\'stable\\', \\'delete\\', \\'single\\', \\'list\\', \\'fixed\\', \\'commercial\\', \\'consider\\', \\'learn\\', \\'entry\\', \\'experience\\', \\'multitasking\\', \\'programmer\\', \\'laser\\', \\'computing\\', \\'workstation\\', \\'databases\\', \\'libraries\\', \\'reboot\\', \\'anybody\\', \\'board\\', \\'desktop\\', \\'midi\\', \\'left\\', \\'click\\', \\'larger\\', \\'add\\', \\'exactly\\', \\'packet\\', \\'path\\', \\'advance\\', \\'fix\\', \\'cache\\', \\'numbers\\', \\'site\\', \\'place\\', \\'actual\\', \\'main\\', \\'nothing\\', \\'buy\\', \\'development\\', \\'gives\\', \\'reply\\', \\'printing\\', \\'questions\\', \\'looks\\', \\'easier\\', \\'registered\\', \\'include\\', \\'starting\\', \\'agree\\', \\'current\\', \\'prevent\\', \\'properly\\', \\'decided\\', \\'job\\', \\'business\\', \\'opinion\\', \\'startup\\', \\'servers\\', \\'injuries\\', \\'i.e\\', \\'offer\\', \\'mentioned\\', \\'whether\\', \\'discussion\\', \\'ask\\', \\'service\\', \\'newsgroup\\', \\'soda.berkeley.edu\\', \\'knows\\', \\'release\\', \\'floppies\\', \\'monochrome\\', \\'slower\\', \\'industry\\', \\'update\\', \\'lots\\', \\'due\\', \\'cause\\', \\'documentation\\', \\'app\\', \\'view\\', \\'appears\\', \\'rest\\', \\'power\\', \\'changed\\', \\'interested\\', \\'dont\\', \\'requires\\', \\'breaks\\']'}"
      ]
     },
     "execution_count": 160,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.pprompt(\"Please compare topics 5 and 7.\")"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Add a completely new topic"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can also add a completely new topic based on the keyword \"Politics and government\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 168,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "GPT wants to the call the function:  {\n",
      "  \"name\": \"add_new_topic_keyword\",\n",
      "  \"arguments\": \"{\\n  \\\"keyword\\\": \\\"Politics and the government\\\"\\n}\"\n",
      "}\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Epochs completed: 100%| ██████████ 100/100 [00:01]\n",
      "Epochs completed: 100%| ██████████ 100/100 [00:03]\n",
      "Computing word-topic matrix: 100%|██████████| 21/21 [01:18<00:00,  3.72s/it]\n",
      "Epochs completed: 100%| ██████████ 30/30 [00:04]\n",
      "100%|██████████| 21/21 [01:58<00:00,  5.64s/it]\n"
     ]
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Sure! I have added a new topic based on the keyword \"Politics and the government\". The new topic is called \"Government and Policy\".\n",
      "\n",
      "The common theme of the given words is \"government and public policy\". This topic encompasses various aspects and sub-topics, including:\n",
      "\n",
      "1. Government actions and decisions: This sub-topic focuses on the actions and decisions made by the government. It includes words like \"make\", \"said\", \"believe\", \"question\", and \"decisions\".\n",
      "\n",
      "2. Healthcare system: This sub-topic is centered around the healthcare system, including topics such as health, care, insurance, private healthcare, and drugs.\n",
      "\n",
      "3. Economy and jobs: This sub-topic explores the economy and job-related issues. It includes words like jobs, work, economic, business, and spending.\n",
      "\n",
      "4. Education and schools: This sub-topic pertains to education and schools. It includes words like education, school, students, learning, and teachers.\n",
      "\n",
      "5. Law and order: This sub-topic focuses on the legal system and maintaining law and order. It includes words like law, crime, police, legal, and punishment.\n",
      "\n",
      "These are just a few examples of the aspects and sub-topics that fall under the broader topic of \"Government and Policy\".\n"
     ]
    },
    {
     "data": {
      "text/plain": [
       "[Topic 0: Electronics equipment sales,\n",
       " Topic 1: Image Processing,\n",
       " Topic 2: Gun control,\n",
       " Topic 3: Internet Privacy,\n",
       " Topic 4: Conflict and Violence,\n",
       " Topic 5: Computer Hardware,\n",
       " Topic 6: Anonymous Posting,\n",
       " Topic 7: Computer Software,\n",
       " Topic 8: Car features and performance.,\n",
       " Topic 9: Encryption,\n",
       " Topic 10: Technology,\n",
       " Topic 11: Space Exploration,\n",
       " Topic 12: Motorcycle Riding Tips,\n",
       " Topic 13: Technology and Computing,\n",
       " Topic 14: Healthcare and Medicine,\n",
       " Topic 15: Biblical interpretation,\n",
       " Topic 16: Beliefs and Religion,\n",
       " Topic 17: Existence of God,\n",
       " Topic 18: Sexual Morality,\n",
       " Topic 19: Sports,\n",
       " Topic 20: Government and Policy]"
      ]
     },
     "execution_count": 168,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "tm.pprompt(\"Please add a completely new topic based on the keyword 'Politics and the government'.\")"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "test_llm_sem1",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.11.4"
  },
  "orig_nbformat": 4
 },
 "nbformat": 4,
 "nbformat_minor": 2
}