The LLM-based topic recognition model is complete and adapted to quickly updating Weibo topics.

@@ -568,4 +568,26 @@ Apache License
    distributed under the License is distributed on an "AS IS" BASIS,
    distributed under the License is distributed on an "AS IS" BASIS,
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
    See the License for the specific language governing permissions and
    See the License for the specific language governing permissions and
-   limitations under the License.
+   limitations under the License.
+
+   MIT License
+
+Copyright (c) 2023 Arik Reuter
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

+version: 2
+
+sphinx:
+  configuration: docs/source/conf.py
+
+python:
+  version: 3.8
+  install:
+    - requirements: docs/requirements.txt

+# MANIFEST.in
+
+include README.md
+recursive-include quicktests *

+# TopicGPT
+TopicGPT integrates the remarkable capabilities of current LLMs such as GPT-3.5 and GPT-4 into topic modelling. 
+
+While traditional topic models extract topics as simple lists of top-words, such as ["Lion", "Leopard", "Rhino", "Elephant", "Buffalo"], TopicGPT offers rich and dynamic topic representations that can be intuitively understood, extensively investigated and modified in various ways via a simple text commands in natural language. 
+
+More specifically, it provides the following core functionalities: 
+- Identification of clusters within document-embeddings and top-word extraction
+- Generation of informative topic descriptions 
+- Extraction of detailed information about topics via Retrieval-Augmented-Generation (RAG)
+- Comparison of topics
+- Splitting and combining of identified topics
+- Addition of new topics based on keywords
+- Deletion of topics
+  
+When directly interacting with TopicGPT via prompting and without explicitly calling  functions, an LLM autonomously decides which functionality to use.
+
+## Paper
+
+To read more about the model, checkout the corresponding [paper](https://arxiv.org/abs/2403.03628): https://arxiv.org/abs/2403.03628
+
+## Installation
+
+You can install TopicGPT via [PyPI](https://pypi.org/project/topicgpt/)
+
+```
+pip install topicgpt
+```
+
+## Further Documentation
+
+You can find detailed documentation of the available classes and functions [here](https://lmu-seminar-llms.github.io/TopicGPT/).
+
+
+## Example 
+
+The following short example demonstrates how TopicGPT could be used on a real-world dataset. The Twenty Newsgroups corpus (https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html) is used for this purpose. 
+
+Further example-notebooks can be found under examples/ in the repository.
+
+### Load the data
+
+```python
+from sklearn.datasets import fetch_20newsgroups
+
+data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')) #download the 20 Newsgroups dataset
+corpus = data['data'] 
+
+corpus = [doc for doc in corpus if doc != ""] #remove empty documents
+```
+### Initialize the model 
+
+Note that an OpenAi API-Key is needed to compute the embeddings and execute the prompts. See https://platform.openai.com/account/api-keys for more details. We select 20 topics in this case since the Twenty Newsgroups corpus comprises documents from 20 different newsgroups. It is also possible to let Hdbscan determine the number of topics automatically. 
+
+```python 
+from topicgpt.TopicGPT import TopicGPT
+
+tm = TopicGPT(
+    api_key = <your-openai-api-key>,
+    n_topics = 20 # select 20 topics since the true number of topics is 20 
+)
+
+# Or, to use with Azure
+tm = TopicGPT(
+    api_key = <your-azure-openai-api-key>,
+    azure_endpoint = {
+         "endpoint": <your-azure-openai-endpoint-url>,
+         "api_version": <api-version>
+     },
+    n_topics = 20
+)
+```
+
+### Fit the model 
+
+The fit-method fits the model. This can take, depending on the size of the dataset and wether embeddings have been provided, from a few minutes to several hours. Especially the computation of the embeddings can take some time. 
+
+```python 
+tm.fit(corpus) # the corpus argument should be of type list[str] where each string represents one document
+```
+
+### Inspect the found topics
+
+Obtain an overview over the indentified topics
+```python
+print(tm.topic_lis)
+```
+_Output_
+```
+[Topic 0: Electronics Equipment Sales,
+ Topic 1: Image Processing,
+ Topic 2: Gun control,
+ Topic 3: Online Privacy and Anonymity,
+ Topic 4: Conflict and Violence.,
+ Topic 5: Computer Hardware,
+ Topic 6: Belief and Atheism,
+ Topic 7: Online Discussions,
+ Topic 8: Computer Software,
+ Topic 9: Car Features and Performance,
+ Topic 10: Encryption and Government,
+ Topic 11: Technology and Computing.,
+ Topic 12: Technology and Computing,
+ Topic 13: Space Exploration,
+ Topic 14: Motorcycle Riding Techniques,
+ Topic 15: Technology,
+ Topic 16: Hockey Games,
+ Topic 17: Health and Medicine.,
+ Topic 18: Baseball games and teams.,
+ Topic 19: Beliefs about Homosexuality.]
+```
+To obtain more detailed information on each topic, we can call the "print_topics" method: 
+
+```python
+tm.print_topics()
+```
+_Output_
+```
+Topic 0: Electronics Equipment Sales
+
+Topic_description: The common topic of the given words appears to be "electronics and technology". 
+
+Various aspects and sub-topics of this topic include:
+1. Buying and selling: "offer", "sale", "sell", "price", "buy"
+2. Device usage and features: "use", "get", "new", "used", "condition"
+3. Technical specifications: "wire", "ground", "power", "circuit", "voltage"
+4. Communication and connectivity: "phone", "email", "modem", "wireless", "connection"
+5. Accessories and peripherals: "battery", "cable", "manuals", "disk", "monitor"
+Top words: ["n't", 'one', 'would', 'use', 'like', 'get', 'new', 'used', 'offer', 'sale']
+
+[...]
+```
+We can also visualize the resulting clusters to get an overview of the shape and size of the clusters
+```
+tm.visualize_clusters()
+```
+
+### Find out more detailed information about the identified topics
+
+First, we might be interested in knowing what information the space topic (topic 13) contains on the moon landing. 
+
+```python 
+tm.pprompt("Which information related to the keyword 'moon landing' does topic 13 have?")
+```
+
+_Output_
+```
+GPT wants to the call the function:  {
+  "name": "knn_search",
+  "arguments": "{\n  \"topic_index\": 13,\n  \"query\": \"moon landing\",\n  \"k\": 5\n}"
+}
+Topic 13, which is related to the keyword "moon landing," has the following information:
+
+1. Document index 258: This document provides an introduction to the solar system and mentions that advancements in rocketry after World War II enabled machines to travel to the Moon and other planets. It highlights that the United States has sent both automated spacecraft and human-crewed expeditions to explore the Moon.
+
+2. Document index 535: This document discusses a $65 million program called the Back to the Moon bill, which aims to encourage private companies to develop lunar orbiters. It mentions that there is a chance of making a lunar mission happen in this decade through this program.
+
+3. Document index 357: This document is a request for more information on a recent newspaper article about the Japanese crashing or crash-landing a package on the Moon. It indicates that the article was vague and unclear.
+
+4. Document index 321: This document speculates about what would have happened if the Soviets had beaten the United States in the Moon race. It suggests that the US would have still performed Moon landings and potentially set up a lunar base. The focus on Mars exploration would have depended on the Soviets' actions.
+
+5. Document index 102: This document mentions the Hiten engineering-test mission, which spent time in a highly eccentric Earth orbit and performed lunar flybys before being inserted into lunar orbit using gravity-assist-like maneuvers. It states that the mission was expected to crash on the Moon eventually.
+
+Please note that the above summaries are based on the content of the documents and may not capture all the information contained within them.
+```
+
+From this output we see that an instance of a GPT decided to call the function "knn_search" from the class "TopicPrompting". Indeed some documents on the topic "moon landing" have been found and the model summarizes the relevant information accordingly. 
+
+If we want to check, for instance the document with index 102 in topic 13 to learn more about the Hiten engineering-test mission, we can simply do the following:
+
+```python
+print(tm.topic_lis[13].documents[535])
+```
+_Output_
+```
+Their Hiten engineering-test mission spent a while in a highly eccentric Earth orbit doing lunar flybys, and then was inserted into lunar orbit using some very tricky gravity-assist-like maneuvering.  This meant that it would crash on the Moon eventually, since there is no such thing as a stable lunar orbit (as far as anyone knows), and I believe I recall hearing recently that it was about to happen.
+```
+
+### Split Topics
+We find that topic 6 about Belief and Atheism is a bit general and would thus like to split see what 5 potential subtopics it contains: 
+
+```python
+tm.pprompt("What are 5 potential subtopics of topic 6")
+```
+
+_Output_
+
+```
+GPT wants to the call the function:  {
+  "name": "split_topic_kmeans",
+  "arguments": "{\n  \"topic_idx\": 6,\n  \"n_clusters\": 5\n}"
+}
+
+Here are five potential subtopics of topic 6:
+
+1. Existence of God: This subtopic explores arguments, evidence, proofs, and the existence of God.
+
+2. Atheism vs Theism: This subtopic delves into the concepts of atheism, theism, atheists, and theists, and the debates surrounding these belief systems.
+
+3. Belief and Faith: This subtopic focuses on beliefs, faith, believers, and the roles they play in religious and philosophical contexts.
+
+4. Rationality and Reasoning: This subtopic involves discussions on reason, rationality, logic, and argumentation as they relate to beliefs and debates about religion.
+
+5. Morality and Ethics: This subtopic explores moral values, ethics, morals, and how they influence and are influenced by religious beliefs and debates.
+
+These subtopics were inferred from the words provided and the descriptions of the topics in the corpus used for topic modeling.
+
+```
+Based on those results we decide to not actually split the topic into 5 subtopics but rather just 3 - One on "religious faith", one on "atheism" and one on "ethics and philosophy".
+We also want to actually make those changes to the existing topics, so we tell TopicGPT to do the changes inplace. 
+
+```python
+tm.pprompt("Please split topic 6 into subtopics based on the keywords 'religious faith', 'atheism' and 'ethics and philosophy'. Do this inplace.")
+```
+_Output_
+
+```
+I have split topic 6 into three subtopics based on the provided keywords: "religious faith", "atheism", and "ethics and philosophy". Here are the new subtopics:
+
+1. Topic 21: Religious Beliefs
+   - This topic encompasses the general theme of "Religion and Belief".
+   - Subtopics and aspects of this topic include:
+     - Faith and Beliefs: Believing, belief, true, evidence, truth.
+     - Religious Practices: Praying, heretic, bible, church.
+     - Religious Communities: Christian, Muslim, atheist.
+     - Interpretation and Understanding: Interpretations, interpretation, understanding, context.
+     - Morality and Ethics: Good, wrong, punishment, violence.
+
+2. Topic 22: Existence of God
+   - This topic focuses on "Atheism and Belief".
+   - Subtopics and aspects of this topic include:
+     - Arguments and Evidence: Atheists provide evidence for their beliefs.
+     - Existence of God: Atheists question the existence of God.
+     - Religious Beliefs: Atheists challenge religious beliefs.
+     - Science and Reason: Atheists rely on science and reason.
+     - Morality and Ethics: Atheists discuss morality and ethics.
+
+3. Topic 23: Ethics and Morality
+   - This topic revolves around "Morality and Ethics".
+   - Subtopics and aspects of this topic include:
+     - Moral reasoning: "Argument," "reasoning," "logic," "fallacy," "premises."
+     - Ethical principles: "Right," "wrong," "immoral," "morals," "ethical."
+     - Moral judgments: "Moral," "ethical," "morality," "judgments," "values."
+     - Human behavior: "Homosexual," "sexual," "promiscuous," "behavior," "sex."
+     - Scientific perspective: "Science," "scientific," "evidence," "empirical," "studies."
+Please note that these subtopics are based on the keywords provided and may not cover all aspects of the original topic.
+```
+
+We see that TopicGPT performed the splitting as intended. However, the names and descriptions of the topics got changed slightly in order to optimally fit to the documents that actually constitute the topics based on the keywords. (If you like you can rename them at any time). 
+
+### Combining topics
+
+Since topics 15 ("Hockey Games") and 17 ("Baseball games and teams") are both about sports, we want to combine them into a single topic.
+
+```python
+tm.pprompt("Please combine topics 15 and 17. Do this inplace.")
+```
+_Output_
+
+```
+GPT wants to the call the function:  {
+  "name": "combine_topics",
+  "arguments": "{\n  \"topic_idx_lis\": [15, 17],\n  \"inplace\": true\n}"
+}
+The topics 15 and 17 have been combined into a new topic called "Sports". This topic includes aspects and sub-topics related to sports such as team and players, games and seasons, performance and skills, fans and audience, and statistics and records. Some of the common words found in this topic include "team," "players," "hockey," "baseball," "game," "games," "season," "playoffs," "good," "better," "win," "hit," "score," "fans," "series," "watch," "fan," "stats," "record," "pts," and "career".
+```
+
+### Saving and Reusing Embeddings
+
+After generating embeddings with `tm.fit(corpus)`, save them with `tm.save_embeddings()`. By default, they are stored in `SavedEmbeddings/embeddings.pkl`. Enable reuse by setting `use_saved_embeddings=True` in `TopicGPT` initialization.
+
+```python
+tm.fit(corpus)
+tm.save_embeddings()  # Default path
+
+# Reuse saved embeddings
+tm2 = TopicGPT(use_saved_embeddings=True)
+
+# For a custom path:
+tm.save_embeddings(path='your/custom/path.pkl')
+tm3 = TopicGPT(use_saved_embeddings=True, path_saved_embeddings='your/custom/path.pkl')
+```
+
+This approach saves time by avoiding re-calculation of embeddings for large datasets.
+
+
+## Limitations and Caveats
+
+It is important to note that, as a model built on top of inherently stochastic LLMs and all their shortcomings, TopicGPT has several limitations and shortcomings as well. LLMs are Machine Learning models and as such, they are not perfect at solving the intended tasks; They may be useful because they are correct reasonably often, but they can always fail. The following list is not complete, but may provide useful information on what may go wrong when using TopicGPT:
+
+- **Hallucination**: LLMs are well known for yielding incorrect but coherent and plausible answers that seem convincing but are actually just made up. Although we tried to minimize this undesired behavior through carefully designing the used prompts, we found that TopicGPT may hallucinate (especially) with respect to the following aspects:
+  - Making up, distorting or misinterpreting content of documents retrieved via knn-search. 
+  - Incorrectly naming and describing topics based on top-words. Specifically, the model can identify topics that seem coherent and reasonable although the corresponding documents are not actually related.
+
+- **Unsdesired Behaviour**: When using the "prompt" or "pprompt" function, TopicGPT may not call the function you intended it to call. This can be alleviated by explicitly telling the model which function to use or directly calling the function yourself. It sometimes also tires to call invalid functions or functions with invalid arguments.
+
+- **Stoachasticity**: The behavior of TopicGPT is not deterministic and exhibits some randomness. There is always some probability that certain actions do not work as intended at the first try because some components of the LLM do not function as desired. Simply trying again should mostly help with those issues. 
+  - On the other hand, TopicGPT may also be overly cautious and report that no relevant information has been found or no topic exists that matches a certain keyword even though it does. This could be caused by designing prompts to prevent massive occurrence of falsely positive results. 
+  Note that using GPT-4 in TopicGPT can help to significantly alleviate issues with hallucination.
+
+- **Erroneous embeddings**: The document- and word-embeddings used in TopicGPT may not always reflect the actual semantics of the texts correctly. More specifically, the embeddings sometimes reflect, for instance, grammatical or orthographical aspects such that clusters based on those aspects may be identified.
+
+- **Size of the dataset**: TopicGPT might fail when the dataset is too small (less than 1000 documents). This is because then the identified topics might become very small and noisy. The RAG aspect will also likely not work as intended. Datasets of more than 10,000 documents are recommended. Note that the processing of very large datasets might not fit into the main memory of your computer.
+
+
+## Tips and tricks for prompting TopicGPT
+When using the "pprompt" or "prompt" function, TopicGPT can behave differently than intended. To alleviate those issues some simple tricks can help: 
+
+- Explicitly tell the model which function it should use and which parameters to select. (Sometimes the model simply cannot know what you expect it to do.) For example, instead of using ```tm.pprompt("What are the subtopic of topic 13?")```, use something like ```tm.pprompt("What are the subtopic of topic 13? Please use the function that uses the k-means algorithm to split the topic. Use a parameter of k = 5 and do this inplace")```
+
+- Just ask the same prompt again. Since TopicGPT is a stochastic system, calling the same function with the same argument again might yield a different functionality to be used or a different result. 
+
+- If this doesn't help, you can also directly call the function you want to use from the TopicPrompting class. In the example above you could do ```tm.topic_prompting.split_topic_kmeans(topic_idx = 13, n_clusters = 5, inplace = True)```. Note that all functions the model can call can also be called directly.
+
+-  In case of hallucination of facts it may help to use GPT-4 for TopicGPT
+
+## How TopicGPT works
+
+TopicGPT is centrally built on top of text embeddings and the prompting mechanisms obtained via LLMs and provided by the OpenAI API. Please also see the section [References](#references) for more details on the models and ideas used in TopicGPT.
+
+### Embeddings
+When no embeddings are provided, TopicGPT automatically computes the embeddings of the documents of the provided corpus and also of the vocabulary that is extracted from the corpus. This happens after the fit-method is called. 
+
+The class ```GetEmbeddingsOpenAI``` is used for this purpose.
+
+### Clustering
+In order to identify topics among the documents, TopicGPT reduces the dimensionality of the document embeddings via UMAP and then uses Hdbscan to identify the clusters. Dimensionality reduction is necessary since the document embeddings are of very high dimensionality  and thus the curse of dimensionality would make it very difficult, if not impossible, to identify the clusters.
+
+When not specifying the number of topics in the ```Topic GPT``` class, Hdbscan is used to automatically determine the number of topics. If the number of topics is specified, agglomerative clustering is used on top of the clusters identified by HDBSCAN. 
+
+The class ```Clustering``` is used for this purpose.
+
+### Extraction of Top-Words
+
+After the clusters have been identified, TopicGPT extracts the top-words of each topic. This is done via two different methods:
+- **Tf-idf**: The tf-idf method is based on the idea that words that occur frequently in a topic but rarely in other topics are good indicators for the topic. The top-words are thus the words with the highest tf-idf scores. 
+- **Centroid similarity**: The centroid similarity method is based on the idea that the words that are closest to the centroid of a topic are good indicators for the topic. The top-words are thus the words that are closest to the centroid of the topic.
+
+Note that the Tf-idf heuristic was introduced for the BerTopic Model (Grootendorst, Maarten. "BERTopic: Neural topic modeling with a class-based TF-IDF procedure." arXiv preprint arXiv:2203.05794 (2022)) and a similar idea to the centroid similarity method is used in Top2Vec (Angelov, Dimo. "Top2vec: Distributed representations of topics." arXiv preprint arXiv:2008.09470 (2020)).
+
+Topword extraction is performed with help of the class ```ExtractTopWords```.
+
+### Describing and naming topics
+
+In the next step, all topics are provided with a short name and a description. This is done via prompting an LLM provided by OpenAI with around 500 top-words of each topic. The LLM then generates a short name and a description for each topic.
+
+The class ```TopwordEnhancement``` is used for this purpose.
+
+
+Note that computation of Embeddings, Extraction of Top-Words and Describing and Naming Topics are all performed when calling the ```fit``` method of the ```TopicGPT``` class.	
+
+### Prompting of TopicGPT
+
+When formalizing a prompt via the ```pprompt``` or ```prompt``` function, TopicGPT uses the following steps:  
+
+1. The prompt, together with basic model- and corpus-information, is sent to an LLM provided by OpenAI. The LLM then decides which function of the ```TopicPrompting``` class to call. The LLM also decides which arguments to use for the function.
+2. The function is called according to the information by the LLM. The full result of the function will be returned to the user.
+3. Parts of the results of the function are returned to the LLM. The LLM then generates a short answer of the original prompt with help of the function result and returns it to the user.
+
+
+## References
+
+The following models, software packages and ideas are central for TopicGPT: 
+
+- **UMAP**: The Uniform Manifold Approximation and Projection for Dimension Reduction algorithm is used for reducing the dimensionality of document- and word embeddings (McInnes, Leland, John Healy, and James Melville. "Umap: Uniform manifold approximation and projection for dimension reduction." arXiv preprint arXiv:1802.03426 (2018).)
+
+- **HDBSCAN**: Hierarchical density based clustering is used to identify the clusters among the dimensionality reduced topics (McInnes, Leland, John Healy, and Steve Astels. "hdbscan: Hierarchical density based clustering." J. Open Source Softw. 2.11 (2017): 205.)
+
+- **Agglomerative Clustering**: The agglomerative clustering functionality from sklearn is used to combine topics in case the identified number of clusters exeeds the number of topics specified by the user (Pedregosa, Fabian, et al. "Scikit-learn: Machine learning in Python." the Journal of machine Learning research 12 (2011): 2825-2830., https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html)
+
+- **Topword extraction**: Even though the corresponding packages are not directly used, the topword extraction methods used for this package are based on very similar ideas as found in the BerTopic Model (Grootendorst, Maarten. "BERTopic: Neural topic modeling with a class-based TF-IDF procedure." arXiv preprint arXiv:2203.05794 (2022)) in the case of the tf-idf method and in Top2Vec for the centroid-similarity method (Angelov, Dimo. "Top2vec: Distributed representations of topics." arXiv preprint arXiv:2008.09470 (2020)). 
+
+- **LLMs from the GPT family**: Some references for the models for computing embeddings and answering the prompts include:
+  - Brown, Tom B., et al. “Language Models are Few-Shot Learners.” Advances in Neural Information Processing Systems 33 (2020).
+  - Radford, Alec, et al. “GPT-4: Generative Pre-training of Transformers with Discrete Latent Variables.” arXiv preprint arXiv:2302.07413 (2023).
+  - Radford, Alec, et al. “Improving Language Understanding by Generative Pre-Training.” URL: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf. [6]
+  - Radford, Alec, et al. “Language Models are Unsupervised Multitask Learners.” OpenAI Blog 1.8 (2019): 9. [7]

+# Minimal makefile for Sphinx documentation
+#
+
+# You can set these variables from the command line, and also
+# from the environment for the first two.
+SPHINXOPTS    ?=
+SPHINXBUILD   ?= sphinx-build
+SOURCEDIR     = source
+BUILDDIR      = build
+
+# Put it first so that "make" without argument is like "make help".
+help:
+	@$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)
+
+.PHONY: help Makefile
+
+# Catch-all target: route all unknown targets to Sphinx using the new
+# "make mode" option.  $(O) is meant as a shortcut for $(SPHINXOPTS).
+%: Makefile
+	@$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O)

+==============
+TopicGPT
+==============
+
+TopicGPT integrates the remarkable capabilities of current LLMs such as GPT-3.5 and GPT-4 into topic modeling.
+
+While traditional topic models extract topics as simple lists of top-words, such as ["Lion", "Leopard", "Rhino", "Elephant", "Buffalo"], TopicGPT offers rich and dynamic topic representations that can be intuitively understood, extensively investigated and modified in various ways via simple text commands.
+
+More specifically, it provides the following core functionalities:
+
+- Identification of clusters within document-embeddings and top-word extraction
+- Generation of informative topic descriptions
+- Extraction of detailed information about topics via Retrieval-Augmented-Generation (RAG)
+- Comparison of topics
+- Splitting and combining of identified topics
+- Addition of new topics based on keywords
+- Deletion of topics
+
+It is further possible to directly interact with TopicGPT via prompting and without explicitly calling functions - an LLM autonomously decides which functionality to use.
+
+Installation Guide
+------------------
+
+To install TopicGPT, simply use PyPI:
+
+.. code-block:: bash
+
+    pip install topicgpt
+
+GitHub Repository
+-----------------
+
+For more details, usage examples, source code, and testing procedures, please visit the TopicGPT GitHub repository: https://github.com/LMU-Seminar-LLMs/TopicGPT
+

+TopicGPT
+========
+
+TopicGPT integrates the remarkable capabilities of current LLMs such as GPT-3.5 and GPT-4 into topic modeling.
+
+While traditional topic models extract topics as simple lists of top-words, such as ["Lion", "Leopard", "Rhino", "Elephant", "Buffalo"], TopicGPT offers rich and dynamic topic representations that can be intuitively understood, extensively investigated and modified in various ways via simple text commands.
+
+More specifically, it provides the following core functionalities:
+
+- Identification of clusters within document-embeddings and top-word extraction
+- Generation of informative topic descriptions
+- Extraction of detailed information about topics via Retrieval-Augmented-Generation (RAG)
+- Comparison of topics
+- Splitting and combining of identified topics
+- Addition of new topics based on keywords
+- Deletion of topics
+
+It is further possible to directly interact with TopicGPT via prompting and without explicitly calling functions - an LLM autonomously decides which functionality to use.
+
+
+GitHub Repository
+----------------
+
+You can find the source code and related materials for this project in the GitHub repository:
+
+- [TopicGPT](https://github.com/LMU-Seminar-LLMs/TopicGPT/tree/dev)
+
+
+
+
+
+Installation
+------------
+
+You can install topicgpt via `PyPI <https://pypi.org/project/topicgpt/>`
+
+::
+
+    pip install topicgpt
+
+
+Example
+=======
+
+The following example demonstrates how TopicGPT can be used on a real-world dataset. The Twenty Newsgroups corpus (`Twenty Newsgroups Corpus Documentation <https://scikit-learn.org/0.19/datasets/twenty_newsgroups.html>`_) is used for this purpose.
+
+Load the data
+-------------
+
+.. code-block:: python
+
+    from sklearn.datasets import fetch_20newsgroups
+
+    data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')) #download the 20 Newsgroups dataset
+    corpus = data['data']
+
+    corpus = [doc for doc in corpus if doc != ""] #remove empty documents
+
+Initialize the model
+--------------------
+
+Note that an OpenAi API-Key is needed to compute the embeddings and execute the prompts. See `OpenAI API Keys Documentation <https://platform.openai.com/account/api-keys>`_ for more details. We select 20 topics in this case since the Twenty Newsgroups corpus comprises documents from 20 different newsgroups. It is also possible to let Hdbscan determine the number of topics automatically.
+
+.. code-block:: python
+
+    from topicgpt.TopicGPT import TopicGPT
+
+    tm = TopicGPT(
+        openai_api_key = <your-openai-api-key>,
+        n_topics = 20 # select 20 topics since the true number of topics is 20
+    )
+
+
+Fit the model
+------------
+
+The fit-method fits the model. This can take, depending on the size of the dataset and whether embeddings have been provided, from a few minutes to several hours. Especially the computation of the embeddings can take some time.
+
+.. code-block:: python
+
+    tm.fit(corpus) # the corpus argument should be of type list[str] where each string represents one document
+
+Inspect the found topics
+------------------------
+
+Obtain an overview of the identified topics.
+
+.. code-block:: python
+
+    print(tm.topic_lis)
+
+    Output:
+
+    .. code-block:: plaintext
+
+        [Topic 0: Electronics Equipment Sales,
+         Topic 1: Image Processing,
+         Topic 2: Gun control,
+         Topic 3: Online Privacy and Anonymity,
+         Topic 4: Conflict and Violence.,
+         Topic 5: Computer Hardware,
+         Topic 6: Belief and Atheism,
+         Topic 7: Online Discussions,
+         Topic 8: Computer Software,
+         Topic 9: Car Features and Performance,
+         Topic 10: Encryption and Government,
+         Topic 11: Technology and Computing.,
+         Topic 12: Technology and Computing,
+         Topic 13: Space Exploration,
+         Topic 14: Motorcycle Riding Techniques,
+         Topic 15: Technology,
+         Topic 16: Hockey Games,
+         Topic 17: Health and Medicine.,
+         Topic 18: Baseball games and teams.,
+         Topic 19: Beliefs about Homosexuality.]
+
+To obtain more detailed information on each topic, we can call the "print_topics" method:
+
+.. code-block:: python
+
+    tm.print_topics()
+
+    Output:
+
+    .. code-block:: plaintext
+
+        Topic 0: Electronics Equipment Sales
+
+        Topic_description: The common topic of the given words appears to be "electronics and technology".
+
+        Various aspects and sub-topics of this topic include:
+        1. Buying and selling: "offer", "sale", "sell", "price", "buy"
+        2. Device usage and features: "use", "get", "new", "used", "condition"
+        3. Technical specifications: "wire", "ground", "power", "circuit", "voltage"
+        4. Communication and connectivity: "phone", "email", "modem", "wireless", "connection"
+        5. Accessories and peripherals: "battery", "cable", "manuals", "disk", "monitor"
+        Top words: ["n't", 'one', 'would', 'use', 'like', 'get', 'new', 'used', 'offer', 'sale']
+
+        [...]
+
+We can also visualize the resulting clusters to get an overview of the shape and size of the clusters.
+
+.. code-block:: plaintext
+
+    tm.visualize_clusters()
+
+Find out more detailed information about the identified topics
+------------------------------------------------------------
+
+First, we might be interested in knowing what information the space topic (topic 13) contains on the moon landing.
+
+.. code-block:: python
+
+    tm.pprompt("Which information related to the keyword 'moon landing' does topic 13 have?")
+
+    Output:
+
+    .. code-block:: plaintext
+
+        GPT wants to call the function: {
+          "name": "knn_search",
+          "arguments": "{\n  \"topic_index\": 13,\n  \"query\": \"moon landing\",\n  \"k\": 5\n}"
+        }
+        Topic 13, which is related to the keyword "moon landing," has the following information:
+
+        1. Document index 258: This document provides an introduction to the solar system and mentions that advancements in rocketry after World War II enabled machines to travel to the Moon and other planets. It highlights that the United States has sent both automated spacecraft and human-crewed expeditions to explore the Moon.
+
+        2. Document index 535: This document discusses a $65 million program called the Back to the Moon bill, which aims to encourage private companies to develop lunar orbiters. It mentions that there is a chance of making a lunar mission happen in this decade through this program.
+
+        3. Document index 357: This document is a request for more information on a recent newspaper article about the Japanese crashing or crash-landing a package on the Moon. It indicates that the article was vague and unclear.
+
+        4. Document index 321: This document speculates about what would have happened if the Soviets had beaten the United States in the Moon race. It suggests that the US would have still performed Moon landings and potentially set up a lunar base. The focus on Mars exploration would have depended on the Soviets' actions.
+
+        5. Document index 102: This document mentions the Hiten engineering-test mission, which spent time in a highly eccentric Earth orbit and performed lunar flybys before being inserted into lunar orbit using gravity-assist-like maneuvers. It states that the mission was expected to crash on the Moon eventually.
+
+        Please note that the above summaries are based on the content of the documents and may not capture all the information contained within them.
+
+From this output, we see that an instance of a GPT decided to call the function "knn_search" from the class "TopicPrompting." Indeed, some documents on the topic "moon landing" have been found, and the model summarizes the relevant information accordingly.
+
+If we want to check, for instance, the document with index 102 in topic 13 to learn more about the Hiten engineering-test mission, we can simply do the following:
+
+.. code-block:: python
+
+    print(tm.topic_lis[13].documents[535])
+
+    Output:
+
+    .. code-block:: plaintext
+
+        Their Hiten engineering-test mission spent a while in a highly eccentric Earth orbit doing lunar flybys, and then was inserted into lunar orbit using some very tricky gravity-assist-like maneuvering. This meant that it would crash on the Moon eventually, since there is no such thing as a stable lunar orbit (as far as anyone knows), and I believe I recall hearing recently that it was about to happen.
+
+
+Split Topics
+------------
+
+We find that topic 6 about Belief and Atheism is a bit general and would thus like to split it into subtopics. Let's see what 5 potential subtopics it contains:
+
+.. code-block:: python
+
+    tm.pprompt("What are 5 potential subtopics of topic 6")
+
+    Output:
+
+    .. code-block:: plaintext
+
+        GPT wants to call the function:  {
+          "name": "split_topic_kmeans",
+          "arguments": "{\n  \"topic_idx\": 6,\n  \"n_clusters\": 5\n}"
+        }
+
+        Here are five potential subtopics of topic 6:
+
+        1. Existence of God: This subtopic explores arguments, evidence, proofs, and the existence of God.
+
+        2. Atheism vs Theism: This subtopic delves into the concepts of atheism, theism, atheists, and theists, and the debates surrounding these belief systems.
+
+        3. Belief and Faith: This subtopic focuses on beliefs, faith, believers, and the roles they play in religious and philosophical contexts.
+
+        4. Rationality and Reasoning: This subtopic involves discussions on reason, rationality, logic, and argumentation as they relate to beliefs and debates about religion.
+
+        5. Morality and Ethics: This subtopic explores moral values, ethics, morals, and how they influence and are influenced by religious beliefs and debates.
+
+        These subtopics were inferred from the words provided and the descriptions of the topics in the corpus used for topic modeling.
+
+Based on those results, we decide to not actually split the topic into 5 subtopics but rather just 3 - One on "religious faith," one on "atheism," and one on "ethics and philosophy." We also want to actually make those changes to the existing topics, so we tell TopicGPT to do the changes inplace.
+
+.. code-block:: python
+
+    tm.pprompt("Please split topic 6 into subtopics based on the keywords 'religious faith', 'atheism' and 'ethics and philosophy'. Do this inplace.")
+
+```
+
+Output:
+
+    .. code-block:: plaintext
+
+
+        I have split topic 6 into three subtopics based on the provided keywords: "religious faith", "atheism", and "ethics and philosophy". Here are the new subtopics:
+
+        1. Topic 21: Religious Beliefs
+          - This topic encompasses the general theme of "Religion and Belief".
+          - Subtopics and aspects of this topic include:
+            - Faith and Beliefs: Believing, belief, true, evidence, truth.
+            - Religious Practices: Praying, heretic, bible, church.
+            - Religious Communities: Christian, Muslim, atheist.
+            - Interpretation and Understanding: Interpretations, interpretation, understanding, context.
+            - Morality and Ethics: Good, wrong, punishment, violence.
+
+        2. Topic 22: Existence of God
+          - This topic focuses on "Atheism and Belief".
+          - Subtopics and aspects of this topic include:
+            - Arguments and Evidence: Atheists provide evidence for their beliefs.
+            - Existence of God: Atheists question the existence of God.
+            - Religious Beliefs: Atheists challenge religious beliefs.
+            - Science and Reason: Atheists rely on science and reason.
+            - Morality and Ethics: Atheists discuss morality and ethics.
+
+        3. Topic 23: Ethics and Morality
+          - This topic revolves around "Morality and Ethics".
+          - Subtopics and aspects of this topic include:
+            - Moral reasoning: "Argument," "reasoning," "logic," "fallacy," "premises."
+            - Ethical principles: "Right," "wrong," "immoral," "morals," "ethical."
+            - Moral judgments: "Moral," "ethical," "morality," "judgments," "values."
+            - Human behavior: "Homosexual," "sexual," "promiscuous," "behavior," "sex."
+            - Scientific perspective: "Science," "scientific," "evidence," "empirical," "studies."
+        Please note that these subtopics are based on the keywords provided and may not cover all aspects of the original topic.
+
+
+We see that TopicGPT performed the splitting as intended. However, the names and descriptions of the topics got changed slightly in order to optimally fit to the documents that actually constitute the topics based on the keywords. (If you like you can rename them at any time). 
+
+Combining topics
+===============
+
+Since topics 15 ("Hockey Games") and 17 ("Baseball games and teams") are both about sports, we want to combine them into a single topic.
+
+.. code-block:: python
+
+    tm.pprompt("Please combine topics 15 and 17. Do this inplace.")
+
+Output
+------
+
+GPT wants to the call the function:
+
+.. code-block:: json
+
+    {
+      "name": "combine_topics",
+      "arguments": "{\n  \"topic_idx_lis\": [15, 17],\n  \"inplace\": true\n}"
+    }
+
+The topics 15 and 17 have been combined into a new topic called "Sports". This topic includes aspects and sub-topics related to sports such as team and players, games and seasons, performance and skills, fans and audience, and statistics and records. Some of the common words found in this topic include "team," "players," "hockey," "baseball," "game," "games," "season," "playoffs," "good," "better," "win," "hit," "score," "fans," "series," "watch," "fan," "stats," "record," "pts," and "career".
+
+Tips and tricks for prompting TopicGPT
+---------------------------------------
+
+When using the "pprompt" or "prompt" function, TopicGPT can behave differently than intended. To alleviate those issues some simple tricks can help:
+
+- Explicitly tell the model which function it should use and which parameters to select. (Sometimes the model simply cannot know what you except it to do.) For example, instead of using ``tm.pprompt("What are the subtopic of topic 13?")``, use something like ``tm.pprompt("What are the subtopic of topic 13? Please use the function that uses the k-means algorithm to split the topic. Use a parameter of k = 5 and do this inplace")``.
+
+- Just ask the same prompt again. Since TopicGPT is a stochastic system, calling the same function with the same argument again might yield a different functionality to be used or a different result.
+
+- If this doesn't help, you can also directly call the function you want to use from the TopicPrompting class. In the example above you could do ``tm.topic_prompting.split_topic_kmeans(topic_idx=13, n_clusters=5, inplace=True)``. Note that all functions the model can call can also be called directly.
+
+- In case of hallucination of facts it may help to use GPT-4 for TopicGPT
+
+
+
+How TopicGPT works
+==================
+
+TopicGPT is centrally built on top of text embeddings and the prompting mechanisms obtained via LLMs and provided by the OpenAI API. Please also see the section `References <#references_>`_ for more details on the models and ideas used in TopicGPT.
+
+Embeddings
+----------
+
+When no embeddings are provided, TopicGPT automatically computes the embeddings of the documents of the provided corpus and also of the vocabulary that is extracted from the corpus. This happens after the fit-method is called.
+
+The class ``GetEmbeddingsOpenAI`` is used for this purpose.
+
+Clustering
+----------
+
+In order to identify topics among the documents, TopicGPT reduces the dimensionality of the document embeddings via UMAP and then uses Hdbscan to identify the clusters. Dimensionality reduction is necessary since the document embeddings are of very high dimensionality, and thus the curse of dimensionality would make it very difficult, if not impossible, to identify the clusters.
+
+When not specifying the number of topics in the ``Topic GPT`` class, Hdbscan is used to automatically determine the number of topics. If the number of topics is specified, agglomerative clustering is used on top of the clusters identified by HDBSCAN.
+
+The class ``Clustering`` is used for this purpose.
+
+Extraction of Top-Words
+------------------------
+
+After the clusters have been identified, TopicGPT extracts the top-words of each topic. This is done via two different methods:
+
+- **Tf-idf**: The tf-idf method is based on the idea that words that occur frequently in a topic but rarely in other topics are good indicators for the topic. The top-words are thus the words with the highest tf-idf scores.
+
+- **Centroid similarity**: The centroid similarity method is based on the idea that the words that are closest to the centroid of a topic are good indicators for the topic. The top-words are thus the words that are closest to the centroid of the topic.
+
+Note that the Tf-idf heuristic was introduced for the BerTopic Model (Grootendorst, Maarten. "BERTopic: Neural topic modeling with a class-based TF-IDF procedure." arXiv preprint arXiv:2203.05794 (2022)) and a similar idea to the centroid similarity method is used in Top2Vec (Angelov, Dimo. "Top2vec: Distributed representations of topics." arXiv preprint arXiv:2008.09470 (2020)).
+
+Topword extraction is performed with help of the class ``ExtractTopWords``.
+
+Describing and naming topics
+------------------------------
+
+In the next step, all topics are provided with a short name and a description. This is done via prompting an LLM provided by OpenAI with around 500 top-words of each topic. The LLM then generates a short name and a description for each topic.
+
+The class ``TopwordEnhancement`` is used for this purpose.
+
+Note that computation of Embeddings, Extraction of Top-Words, and Describing and Naming Topics are all performed when calling the ``fit`` method of the ``TopicGPT`` class.
+
+#### Describing and naming topics
+
+In the next step, all topics are provided with a short name and a description. This is done via prompting an LLM provided by OpenAI with around 500 top-words of each topic. The LLM then generates a short name and a description for each topic.
+
+The class ```TopwordEnhancement``` is used for this purpose.
+
+
+Note that computation of Embeddings, Extraction of Top-Words and Describing and Naming Topics are all performed when calling the ```fit``` method of the ```TopicGPT``` class.	
+
+Prompting
+---------
+
+The main way to interact with TopicGPT is via direct textual prompts. Those prompts are augmented with basic information about desired behavior and potentially useful information. Additionally, information on available functions and their parameters is provided. Then this information is used to prompt an LLM via the OpenAI API. The LLM then decides if it should call a function of the ones provided and if so, which parameters to use. The respective function call is executed, and part of the result is returned to the LLM, which uses the original prompt together with the function call and the result to generate a response.
+
+Functions available for prompting
+---------------------------------
+
+The following functions are available for the LLM to use:
+
+- ``knn_search``: This function is used to find documents that are related to a certain keyword. The LLM can specify the number of documents to be found and the number of keywords to be used. The result is retrieved by performing retrieval-augmented-generation (RAG) where the query is embedded, and the most similar documents are retrieved.
+
+- ``identify_topic_idx``: This function is used to identify the topic that is most related to a certain keyword. This is simply done by providing all topic descriptions to the LLM and then asking for the index of the topic that is most related to the keyword.
+
+- ``get_topic_information``: This function is used to obtain information on certain topics. This can be useful to compare similar topics.
+
+- ``split_topic_kmeans``: This function is used to split a topic into subtopics. The LLM can specify the number of subtopics to be created. The result is retrieved by performing k-means clustering on the document embeddings of the documents in the topic. Note that when splitting a topic, the top-words are not completely recomputed, but rather the top-words of the "super"-topic are distributed among the subtopics.
+
+- ``split_topic_hdbscan``: Works analogously to ``split_topic_kmeans`` but uses Hdbscan instead of k-means clustering. This means that the number of subtopics is not specified by the user but rather automatically determined by Hdbscan.
+
+- ``split_topic_keywords``: This function is used to split a topic into subtopics based on provided keywords. Each keyword is embedded, and the topic is split according to cosine similarity of the document embeddings within the "super"-topic. This means that documents among the "super"-topic that are most similar to a certain keyword are assigned to the corresponding subtopic.
+
+- ``add_new_topic_keyword``: This function is used to add a new topic based on a keyword. The documents belonging to this new topic are computed as the documents from all other topics that are more similar to the embedding of the new keyword than the centroid of the original topic. Then all topwords and the topic description are recomputed.
+
+- ``delete_topic``: This function is used to delete a topic. The LLM can specify the topic to be deleted. The result is retrieved by simply removing the topic from the list of topics and assigning the documents of the deleted topic to the topic with the most similar centroid. Then all topwords and the topic description are recomputed.
+
+- ``combine_topics``: This function is used to combine two topics into a single topic. The LLM can specify the two topics to be combined. The result is retrieved by simply combining the documents of the two topics and re-computing the embeddings and top-words of the new topic.
+
+
+
+Limitations and Caveats
+------------------------
+
+It is important to note that, as a model built on top of inherently stochastic LLMs and all their shortcomings, TopicGPT has several limitations and shortcomings as well. The following list is not aimed at being complete but could provide useful information on what may go wrong when using TopicGPT:
+
+- **Hallucination**: LLMs are well known for yielding incorrect but coherent and plausible answers that seem convincing but are actually just made up. Although we tried to minimize this undesired behavior through carefully designing the used prompts, we found that TopicGPT may hallucinate (especially) with respect to the following aspects:
+
+  - Making up, distorting, or misinterpreting content of documents retrieved via knn-search.
+  - Incorrectly naming and describing topics based on top-words. Specifically, the model can identify topics that seem coherent and reasonable, although the corresponding documents are not actually related.
+
+- **Undesired Behavior**: When using the "prompt" or "pprompt" function, TopicGPT may not call the function you intended it to call. This can be alleviated by explicitly telling the model which function to use or directly calling the function yourself.
+
+- **Stochasticity**: The behavior of TopicGPT is not deterministic and exhibits some randomness. There is always some probability that certain actions do not work as intended at the first try because some components of the LLM do not function as desired. Simply trying again should mostly help with those issues.
+
+  - On the other hand, TopicGPT may also be overly cautious and report that no relevant information has been found or no topic exists that matches a certain keyword, even though it does. This could be caused by designing prompts to prevent the massive occurrence of falsely positive results.
+
+  Note that using GPT-4 in TopicGPT can help to significantly alleviate issues with hallucination.
+
+- **Erroneous Embeddings**: The document- and word-embeddings used in TopicGPT may not always reflect the actual semantics of the texts correctly. More specifically, the embeddings sometimes reflect, for instance, grammatical or orthographical aspects such that clusters based on those aspects may be identified.
+
+References
+----------
+
+The following models, software packages, and ideas are central for TopicGPT:
+
+- **UMAP**: The Uniform Manifold Approximation and Projection for Dimension Reduction algorithm is used for reducing the dimensionality of document- and word embeddings (McInnes, Leland, John Healy, and James Melville. "Umap: Uniform manifold approximation and projection for dimension reduction." arXiv preprint arXiv:1802.03426 (2018)).
+
+- **HDBSCAN**: Hierarchical density-based clustering is used to identify the clusters among the dimensionality reduced topics (McInnes, Leland, John Healy, and Steve Astels. "hdbscan: Hierarchical density-based clustering." J. Open Source Softw. 2.11 (2017): 205).
+
+- **Agglomerative Clustering**: The agglomerative clustering functionality from sklearn is used to combine topics in case the identified number of clusters exceeds the number of topics specified by the user (Pedregosa, Fabian, et al. "Scikit-learn: Machine learning in Python." the Journal of machine Learning research 12 (2011): 2825-2830., https://scikit-learn.org/stable/modules/generated/sklearn.cluster.AgglomerativeClustering.html).
+
+- **Topword extraction**: Even though the corresponding packages are not directly used, the topword extraction methods used for this package are based on very similar ideas as found in the BerTopic Model (Grootendorst, Maarten. "BERTopic: Neural topic modeling with a class-based TF-IDF procedure." arXiv preprint arXiv:2203.05794 (2022)) in the case of the tf-idf method and in Top2Vec for the centroid-similarity method (Angelov, Dimo. "Top2vec: Distributed representations of topics." arXiv preprint arXiv:2008.09470 (2020)).
+
+- **LLMs from the GPT family**: Some references for the models for computing embeddings and answering the prompts include:
+
+  - Brown, Tom B., et al. “Language Models are Few-Shot Learners.” Advances in Neural Information Processing Systems 33 (2020).
+
+  - Radford, Alec, et al. “GPT-4: Generative Pre-training of Transformers with Discrete Latent Variables.” arXiv preprint arXiv:2302.07413 (2023).
+
+  - Radford, Alec, et al. “Improving Language Understanding by Generative Pre-Training.” URL: https://s3-us-west-2.amazonaws.com/openai-assets/research-covers/language-unsupervised/language_understanding_paper.pdf. [6]
+
+  - Radford, Alec, et al. “Language Models are Unsupervised Multitask Learners.” OpenAI Blog 1.8 (2019): 9. [7]
+

+@ECHO OFF
+
+pushd %~dp0
+
+REM Command file for Sphinx documentation
+
+if "%SPHINXBUILD%" == "" (
+	set SPHINXBUILD=sphinx-build
+)
+set SOURCEDIR=source
+set BUILDDIR=build
+
+%SPHINXBUILD% >NUL 2>NUL
+if errorlevel 9009 (
+	echo.
+	echo.The 'sphinx-build' command was not found. Make sure you have Sphinx
+	echo.installed, then set the SPHINXBUILD environment variable to point
+	echo.to the full path of the 'sphinx-build' executable. Alternatively you
+	echo.may add the Sphinx directory to PATH.
+	echo.
+	echo.If you don't have Sphinx installed, grab it from
+	echo.https://www.sphinx-doc.org/
+	exit /b 1
+)
+
+if "%1" == "" goto help
+
+%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+goto end
+
+:help
+%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O%
+
+:end
+popd

+gensim
+hdbscan
+nltk
+numpy
+openai
+pandas
+plotly
+regex
+scikit-learn
+seaborn
+sentence-transformers
+tiktoken
+tokenizers
+tqdm
+umap-learn
+umap-learn[plot]
+sphinx
+sphinx_rtd_theme

+# Configuration file for the Sphinx documentation builder.
+#
+# For the full list of built-in configuration values, see the documentation:
+# https://www.sphinx-doc.org/en/master/usage/configuration.html
+
+# -- Project information -----------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#project-information
+
+master_doc = 'index'
+project = 'topicgpt'
+copyright = '2023, ArikReuter'
+author = 'ArikReuter'
+release = '0.0.4'
+
+# -- General configuration ---------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#general-configuration
+
+extensions = ['sphinx.ext.autodoc', 'sphinx.ext.napoleon']
+
+templates_path = ['_templates']
+exclude_patterns = []
+
+
+# -- Options for HTML output -------------------------------------------------
+# https://www.sphinx-doc.org/en/master/usage/configuration.html#options-for-html-output
+
+html_theme = 'sphinx_rtd_theme'
+html_static_path = ['_static']
+
+
+import os
+import sys
+sys.path.insert(0, os.path.abspath('../../src'))
+sys.path.insert(0, os.path.abspath('../src'))

+.. topicgpt documentation master file, created by
+   sphinx-quickstart on Wed Sep  6 20:34:08 2023.
+   You can adapt this file completely to your liking, but it should at least
+   contain the root `toctree` directive.
+
+Welcome to topicgpt's documentation!
+====================================
+
+.. include:: ../README.rst
+
+.. toctree::
+   :maxdepth: 2
+   :caption: Contents:
+
+   topicgpt  
+
+Indices and tables
+==================
+
+* :ref:`genindex`
+* :ref:`modindex`
+* :ref:`search`

+topicgpt
+========
+
+.. toctree::
+   :maxdepth: 4
+
+   topicgpt

+topicgpt package
+================
+
+Submodules
+----------
+
+topicgpt.Clustering module
+--------------------------
+
+.. automodule:: topicgpt.Clustering
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+topicgpt.ExtractTopWords module
+-------------------------------
+
+.. automodule:: topicgpt.ExtractTopWords
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+topicgpt.GetEmbeddingsOpenAI module
+-----------------------------------
+
+.. automodule:: topicgpt.GetEmbeddingsOpenAI
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+topicgpt.TopicGPT module
+------------------------
+
+.. automodule:: topicgpt.TopicGPT
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+topicgpt.TopicPrompting module
+------------------------------
+
+.. automodule:: topicgpt.TopicPrompting
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+topicgpt.TopicRepresentation module
+-----------------------------------
+
+.. automodule:: topicgpt.TopicRepresentation
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+topicgpt.TopwordEnhancement module
+----------------------------------
+
+.. automodule:: topicgpt.TopwordEnhancement
+   :members:
+   :undoc-members:
+   :show-inheritance:
+
+Module contents
+---------------
+
+.. automodule:: topicgpt
+   :members:
+   :undoc-members:
+   :show-inheritance:

+gensim
+hdbscan
+nltk
+numpy
+openai >= 1.0.0
+pandas
+plotly
+regex
+scikit-learn
+seaborn
+sentence-transformers
+tiktoken
+tokenizers
+tqdm
+umap-learn
+umap-learn[plot]

+from setuptools import setup, find_packages
+
+
+with open("README.md", 'r', encoding='utf') as f:
+    long_description = f.read()
+
+setup(
+    name='topicgpt',
+    version='0.0.5',
+    packages=find_packages(where='src'),
+    package_dir={'': 'src'},
+    install_requires=[
+        'gensim',
+        'hdbscan',
+        'nltk',
+        'numpy', 
+        'openai>=1.0.0',
+        'pandas',
+        'plotly',
+        'regex',
+        'scikit-learn',
+        'seaborn',
+        'sentence-transformers',
+        'tiktoken',
+        'tokenizers',
+        'tqdm',
+        'umap-learn',
+        'umap-learn[plot]'
+        ],
+    include_package_data=True,
+    # Additional metadata
+    author='Arik Reuter',
+    author_email='arik_reuter@gmx.de',
+    description='A package for integrating LLMs like GPT-3.5 and GPT-4 into topic modelling',
+    long_description=long_description,
+    long_description_content_type="text/markdown",
+    license="MIT",
+    keywords=['Topic Modelling', 'GPT', 'LLM', 'OpenAI', 'Retrieval Augmented Generation', 'Chat-GPT', 'GPT-3', 'GPT-4'],
+    classifiers=[
+        "Development Status :: 3 - Alpha",
+        'Intended Audience :: Science/Research',
+        "Intended Audience :: Developers",
+        "Programming Language :: Python :: 3.11",
+        "Operating System :: Unix",
+        "Operating System :: MacOS :: MacOS X",
+        "Operating System :: Microsoft :: Windows",
+    ]
+)
+

+class Client:
+    def __init__(self, api_key: str, azure_endpoint: dict = None) -> None:
+        if azure_endpoint:
+            from openai import AzureOpenAI
+            self.client = AzureOpenAI(api_key=api_key, api_version=azure_endpoint['api_version'], azure_endpoint=azure_endpoint['endpoint'])
+        else:
+            from openai import OpenAI
+            self.client = OpenAI(api_key=api_key)
+    
+    def __getattr__(self, name):
+        """Delegate attribute access to the self.client object."""
+        return getattr(self.client, name)

+import numpy as np
+import umap
+import hdbscan
+import matplotlib.pyplot as plt
+import pandas as pd
+import plotly.express as px
+import umap.plot
+from copy import deepcopy
+from sklearn.cluster import AgglomerativeClustering
+
+from typing import Tuple
+
+class Clustering_and_DimRed():
+
+    """
+    Class to perform dimensionality reduction with UMAP followed by clustering with HDBSCAN.
+    """
+    def __init__(self,
+             n_dims_umap: int = 5,
+             n_neighbors_umap: int = 15,
+             min_dist_umap: float = 0,
+             metric_umap: str = "cosine",
+             min_cluster_size_hdbscan: int = 30,
+             metric_hdbscan: str = "euclidean",
+             cluster_selection_method_hdbscan: str = "eom",
+             number_clusters_hdbscan: int = None,
+             random_state: int = 42,
+             verbose: bool = True,
+             UMAP_hyperparams: dict = {},
+             HDBSCAN_hyperparams: dict = {}) -> None:
+        """
+        Initializes the clustering and dimensionality reduction parameters for topic modeling.
+
+        Args:
+            n_dims_umap (int, optional): Number of dimensions to reduce to using UMAP.
+            n_neighbors_umap (int, optional): Number of neighbors for UMAP.
+            min_dist_umap (float, optional): Minimum distance for UMAP.
+            metric_umap (str, optional): Metric for UMAP.
+            min_cluster_size_hdbscan (int, optional): Minimum cluster size for HDBSCAN.
+            metric_hdbscan (str, optional): Metric for HDBSCAN.
+            cluster_selection_method_hdbscan (str, optional): Cluster selection method for HDBSCAN.
+            number_clusters_hdbscan (int, optional): Number of clusters for HDBSCAN. If None, HDBSCAN will determine the number of clusters automatically. Ensure that min_cluster_size is not too large to find enough clusters.
+            random_state (int, optional): Random state for UMAP and HDBSCAN.
+            verbose (bool, optional): Whether to print progress.
+            UMAP_hyperparams (dict, optional): Additional hyperparameters for UMAP.
+            HDBSCAN_hyperparams (dict, optional): Additional hyperparameters for HDBSCAN.
+        """
+
+
+        # do some checks on the input arguments 
+        assert n_dims_umap > 0, "n_dims_umap must be greater than 0"
+        assert n_neighbors_umap > 0, "n_neighbors_umap must be greater than 0"
+        assert min_dist_umap >= 0, "min_dist_umap must be greater than or equal to 0"
+        assert min_cluster_size_hdbscan > 0, "min_cluster_size_hdbscan must be greater than 0"
+        assert number_clusters_hdbscan is None or number_clusters_hdbscan > 0, "number_clusters_hdbscan must be greater than 0 or None"
+        assert random_state is None or random_state >= 0, "random_state must be greater than or equal to 0"
+
+        self.random_state = random_state
+        self.verbose = verbose
+        self.UMAP_hyperparams = UMAP_hyperparams
+        self.HDBSCAN_hyperparams = HDBSCAN_hyperparams
+
+        # update hyperparameters for UMAP
+        self.UMAP_hyperparams["n_components"] = n_dims_umap
+        self.UMAP_hyperparams["n_neighbors"] = n_neighbors_umap
+        self.UMAP_hyperparams["min_dist"] = min_dist_umap
+        self.UMAP_hyperparams["metric"] = metric_umap
+        self.UMAP_hyperparams["random_state"] = random_state
+        self.UMAP_hyperparams["verbose"] = verbose
+        self.umap = umap.UMAP(**self.UMAP_hyperparams)
+
+        self.HDBSCAN_hyperparams["min_cluster_size"] = min_cluster_size_hdbscan
+        self.HDBSCAN_hyperparams["metric"] = metric_hdbscan
+        self.HDBSCAN_hyperparams["cluster_selection_method"] = cluster_selection_method_hdbscan
+        self.number_clusters_hdbscan = number_clusters_hdbscan
+        self.hdbscan = hdbscan.HDBSCAN(**self.HDBSCAN_hyperparams)
+
+    
+    def reduce_dimensions_umap(self, embeddings: np.ndarray) -> Tuple[np.ndarray, umap.UMAP]:
+        """
+        Reduces dimensions of embeddings using UMAP.
+
+        Args:
+            embeddings (np.ndarray): Embeddings to reduce.
+
+        Returns:
+            tuple: A tuple containing two items:
+                - reduced_embeddings (np.ndarray): Reduced embeddings.
+                - umap_mapper (umap.UMAP): UMAP mapper for transforming new embeddings, especially embeddings of the vocabulary. (MAKE SURE TO NORMALIZE EMBEDDINGS AFTER USING THE MAPPER)
+        """
+
+        mapper = umap.UMAP(**self.UMAP_hyperparams).fit(embeddings)
+        dim_red_embeddings = mapper.transform(embeddings)
+        dim_red_embeddings = dim_red_embeddings/np.linalg.norm(dim_red_embeddings, axis=1).reshape(-1,1)
+        return dim_red_embeddings, mapper
+    
+    def cluster_hdbscan(self, embeddings: np.ndarray) -> np.ndarray:
+        """
+        Cluster embeddings using HDBSCAN.
+        
+        If self.number_clusters_hdbscan is not None, further clusters the data with AgglomerativeClustering to achieve a fixed number of clusters.
+
+        Args:
+            embeddings (np.ndarray): Embeddings to cluster.
+
+        Returns:
+            np.ndarray: Cluster labels.
+        """
+
+        labels = self.hdbscan.fit_predict(embeddings)
+        outliers = np.where(labels == -1)[0]
+
+        if self.number_clusters_hdbscan is not None:
+            clusterer = AgglomerativeClustering(n_clusters=self.number_clusters_hdbscan)  #one cluster for outliers  
+            labels = clusterer.fit_predict(embeddings)
+            labels[outliers] = -1
+
+        # reindex to make the labels consecutive numbers from -1 to the number of clusters. -1 is reserved for outliers
+        unique_labels = np.unique(labels)
+        unique_labels_no_outliers = unique_labels[unique_labels != -1]
+        map2newlabel = {label: i for i, label in enumerate(unique_labels_no_outliers)}
+        map2newlabel[-1] = -1
+        labels = np.array([map2newlabel[label] for label in labels])
+
+        return labels
+    
+    def cluster_and_reduce(self, embeddings: np.ndarray) -> Tuple[np.ndarray, np.ndarray, umap.UMAP]:
+        """
+        Cluster embeddings using HDBSCAN and reduce dimensions with UMAP.
+
+        Args:
+            embeddings (np.ndarray): Embeddings to cluster and reduce.
+
+        Returns:
+            tuple: A tuple containing three items:
+                - reduced_embeddings (np.ndarray): Reduced embeddings.
+                - cluster_labels (np.ndarray): Cluster labels.
+                - umap_mapper (umap.UMAP): UMAP mapper for transforming new embeddings, especially embeddings of the vocabulary. (MAKE SURE TO NORMALIZE EMBEDDINGS AFTER USING THE MAPPER)
+        """
+
+        dim_red_embeddings, umap_mapper = self.reduce_dimensions_umap(embeddings)
+        clusters = self.cluster_hdbscan(dim_red_embeddings)
+        return dim_red_embeddings, clusters, umap_mapper
+    
+    def visualize_clusters_static(self, embeddings: np.ndarray, labels: np.ndarray):
+        """
+        Reduce dimensionality with UMAP to two dimensions and plot the clusters.
+
+        Args:
+            embeddings (np.ndarray): Embeddings for which to plot clustering.
+            labels (np.ndarray): Cluster labels.
+        """
+
+
+        # Reduce dimensionality with UMAP
+        reducer = umap.UMAP(n_components=2, random_state = self.random_state, n_neighbors=30, metric="cosine", min_dist=0)
+        embeddings_2d = reducer.fit_transform(embeddings)
+
+
+        # Create a color palette, then map the labels to the colors.
+        # We add one to the number of unique labels to account for the noise points labelled as -1.
+        palette = plt.cm.get_cmap("tab20", len(np.unique(labels)) + 1)
+        
+        # Create a new figure
+        fig, ax = plt.subplots(figsize=(10, 8))
+
+        outlier_shown_in_legend = False
+
+        # Iterate through all unique labels (clusters and outliers)
+        for label in np.unique(labels):
+            # Find the embeddings that are part of this cluster
+            cluster_points = embeddings_2d[labels == label]
+            
+            # If label is -1, these are outliers. We want to display them in grey.
+            if label == -1:
+                color = 'grey'
+                if not outlier_shown_in_legend:
+                    ax.scatter(cluster_points[:, 0], cluster_points[:, 1], c=color, label='outlier', s = 0.1)
+                    outlier_shown_in_legend = True
+                else:
+                    ax.scatter(cluster_points[:, 0], cluster_points[:, 1], c=color, s = 0.1)
+            else:
+                color = palette(label)
+                # Plot the points in this cluster without a label to prevent them from showing up in the legend
+                ax.scatter(cluster_points[:, 0], cluster_points[:, 1], c=color, s = 0.1)
+            
+        # Add a legend
+        ax.legend()
+
+        # Show the plot
+        plt.show()
+
+
+    def visualize_clusters_dynamic(self, embeddings: np.ndarray, labels: np.ndarray, texts: list[str], class_names: list[str] = None):
+        """
+        Visualize clusters using Plotly and enable hovering over clusters to see the beginning of the texts of the documents.
+
+        Args:
+            embeddings (np.ndarray): Embeddings for which to visualize clustering.
+            labels (np.ndarray): Cluster labels.
+            texts (list[str]): Texts of the documents.
+            class_names (list[str], optional): Names of the classes.
+        """
+
+
+        # Reduce dimensionality with UMAP
+        reducer = umap.UMAP(n_components=2, random_state = self.random_state, n_neighbors=30, metric="cosine", min_dist=0)
+        embeddings_2d = reducer.fit_transform(embeddings)
+
+        df = pd.DataFrame(embeddings_2d, columns=['x', 'y'])
+        df['text'] = [text[:200] for text in texts] 
+        df["class"] = labels
+
+        if class_names is not None:
+            df["class"] = [class_names[label] for label in labels]
+
+        # Create a color palette, then map the labels to the colors.
+        # Exclude the outlier (-1) label from color palette assignment
+        unique_labels = [label for label in np.unique(labels) if label != -1]
+        palette = plt.cm.get_cmap("tab20", len(unique_labels))
+
+        # Create color map
+        color_discrete_map = {label: 'rgb'+str(tuple(int(val*255) for val in palette(i)[:3])) if label != -1 else 'grey' for i, label in enumerate(unique_labels)}
+        color_discrete_map[-1] = 'grey'
+        
+        # plot data points where the color represents the class
+        fig = px.scatter(df, x='x', y='y', hover_data=['text', 'class'], color='class', color_discrete_map=color_discrete_map)
+        
+        fig.update_traces(mode='markers', marker=dict(size=3))  # Optional: Increase the marker size
+
+        # make plot quadratic
+        fig.update_layout(
+        autosize=False,
+        width=1500,
+        height=1500,
+        margin=dict(
+            l=50,   
+            r=50,
+            b=100,
+            t=100,
+            pad=4
+        )
+        )
+        # set title 
+        fig.update_layout(title_text='UMAP projection of the document embeddings', title_x=0.5)
+
+        
+        # show plot
+        fig.show()
+
+
+    def umap_diagnostics(self, embeddings, hammer_edges = False):
+        """
+        Fit UMAP on the provided embeddings and generate diagnostic plots.
+        
+        Params:
+        ------
+        embeddings : array-like
+            The high-dimensional data for UMAP to reduce and visualize.
+        hammer_edges : bool, default False. Is computationally expensive.
+            
+        """
+        new_hyperparams = deepcopy(self.UMAP_hyperparams)
+        new_hyperparams["n_components"] = 2
+        mapper = umap.UMAP(**new_hyperparams).fit(embeddings)
+
+        # 1. Connectivity plot with points
+        print("UMAP Connectivity Plot with Points")
+        umap.plot.connectivity(mapper, show_points=True)
+        plt.show()
+
+        if hammer_edges:
+            # 2. Connectivity plot with edge bundling
+            print("UMAP Connectivity Plot with Hammer Edge Bundling")
+            umap.plot.connectivity(mapper, edge_bundling='hammer')
+            plt.show()
+
+        # 3. PCA diagnostic plot
+        print("UMAP PCA Diagnostic Plot")
+        umap.plot.diagnostic(mapper, diagnostic_type='pca')
+        plt.show()
+
+        # 4. Local dimension diagnostic plot
+        print("UMAP Local Dimension Diagnostic Plot")
+        umap.plot.diagnostic(mapper, diagnostic_type='local_dim')
+        plt.show()

+import nltk
+import string
+import collections
+from tqdm import tqdm
+from typing import List
+import numpy as np
+import re  
+from nltk.tokenize import word_tokenize
+import umap
+from collections import Counter
+import warnings
+
+from typing import List
+
+# make sure the import works even if the package has not been installed and just the files are used
+try:
+    from topicgpt.GetEmbeddingsOpenAI import GetEmbeddingsOpenAI
+except:
+    from GetEmbeddingsOpenAI import GetEmbeddingsOpenAI
+
+nltk.download('stopwords', quiet=True)  # download stopwords
+nltk.download('punkt', quiet=True) # download tokenizer
+
+class ExtractTopWords:
+    
+    def extract_centroids(self, embeddings: np.ndarray, labels: np.ndarray) -> dict:
+        """
+        Extract centroids of clusters.
+
+        Args:
+            embeddings (np.ndarray): Embeddings to cluster and reduce.
+            labels (np.ndarray): Cluster labels. -1 means outlier.
+
+        Returns:
+            dict: Dictionary of cluster labels and their centroids.
+        """
+
+        centroid_dict = {}
+        for label in np.unique(labels):
+            if label != -1:
+                centroid_dict[label] = np.mean(embeddings[labels == label], axis = 0)
+
+        return centroid_dict
+    
+    def extract_centroid(self, embeddings: np.ndarray) -> np.ndarray:
+        """
+        Extract the single centroid of a cluster.
+
+        Args:
+            embeddings (np.ndarray): Embeddings to extract the centroid from.
+
+        Returns:
+            np.ndarray: The centroid of the cluster.
+        """
+
+        return np.mean(embeddings, axis = 0)
+    
+    def compute_centroid_similarity(self, embeddings: np.ndarray, centroid_dict: dict, cluster_label: int) -> np.ndarray:
+        """
+        Compute the similarity of the document embeddings to the centroid of the cluster via cosine similarity.
+
+        Args:
+            embeddings (np.ndarray): Embeddings to cluster and reduce.
+            centroid_dict (dict): Dictionary of cluster labels and their centroids.
+            cluster_label (int): Cluster label for which to compute the similarity.
+
+        Returns:
+            np.ndarray: Cosine similarity of the document embeddings to the centroid of the cluster.
+        """
+
+        centroid = centroid_dict[cluster_label]
+        similarity = np.dot(embeddings, centroid) / (np.linalg.norm(embeddings) * np.linalg.norm(centroid))
+        return similarity
+    
+    def get_most_similar_docs(self, corpus: list[str], embeddings: np.ndarray, labels: np.ndarray, centroid_dict: dict, cluster_label: int, top_n: int = 10) -> List[str]:
+        """
+        Get the most similar documents to the centroid of a cluster.
+
+        Args:
+            corpus (list[str]): List of documents.
+            embeddings (np.ndarray): Embeddings to cluster and reduce.
+            labels (np.ndarray): Cluster labels. -1 means outlier.
+            centroid_dict (dict): Dictionary of cluster labels and their centroids.
+            cluster_label (int): Cluster label for which to compute the similarity.
+            top_n (int, optional): Number of top documents to extract.
+
+        Returns:
+            List[str]: List of the most similar documents to the centroid of a cluster.
+        """
+
+        similarity = self.compute_centroid_similarity(embeddings, centroid_dict, cluster_label)
+        most_similar_docs = [corpus[i] for i in np.argsort(similarity)[-top_n:][::-1]]
+        return most_similar_docs
+    
+    def compute_corpus_vocab(self, 
+                        corpus: list[str],
+                        remove_stopwords: bool = True, 
+                        remove_punction: bool = True, 
+                        min_word_length: int = 3,
+                        max_word_length: int = 20, 
+                        remove_short_words: bool = True, 
+                        remove_numbers: bool = True, 
+                        verbose: bool = True,
+                        min_doc_frequency: int = 3,
+                        min_freq: float = 0.1,
+                        max_freq: float = 0.9) -> list[str]:
+        """
+        Compute the vocabulary of the corpus and perform preprocessing of the corpus.
+
+        Args:
+            corpus (list[str]): List of documents.
+            remove_stopwords (bool, optional): Whether to remove stopwords.
+            remove_punction (bool, optional): Whether to remove punctuation.
+            min_word_length (int, optional): Minimum word length to retain.
+            max_word_length (int, optional): Maximum word length to retain.
+            remove_short_words (bool, optional): Whether to remove short words.
+            remove_numbers (bool, optional): Whether to remove numbers.
+            verbose (bool, optional): Whether to print progress and describe what is happening.
+            min_doc_frequency (int, optional): Minimum number of documents a word should appear in to be considered in the vocabulary.
+            min_freq (float, optional): Minimum frequency percentile of words to be considered in the vocabulary.
+            max_freq (float, optional): Maximum frequency percentile of words to be considered in the vocabulary.
+
+        Returns:
+            list[str]: List of words in the corpus sorted alphabetically.
+        """
+
+        stopwords = set(nltk.corpus.stopwords.words('english'))
+        
+        word_counter = collections.Counter()
+        doc_frequency = collections.defaultdict(set)
+
+        for doc_id, doc in enumerate(tqdm(corpus, disable=not verbose, desc="Processing corpus")):
+            words = nltk.word_tokenize(doc)
+            for word in words:
+                if remove_punction and word in string.punctuation:
+                    continue
+                if remove_stopwords and word.lower() in stopwords:
+                    continue
+                if remove_numbers and re.search(r'\d', word):  # use a regular expression to check for digits
+                    continue
+                if not re.search('[a-zA-Z]', word):  # checks if word contains at least one alphabetic character
+                    continue
+                # remove words that do not begin with an alphabetic character
+                if not word[0].isalpha():
+                    continue
+                if len(word) > max_word_length or (remove_short_words and len(word) < min_word_length):
+                    continue
+                
+                word_lower = word.lower()
+                word_counter[word_lower] += 1
+                doc_frequency[word_lower].add(doc_id)
+
+        total_words = sum(word_counter.values())
+        freq_counter = {word: count / total_words for word, count in word_counter.items()}
+
+        # print most common words and their frequencies
+        if verbose:
+            print("Most common words in the vocabulary:")
+            for word, count in word_counter.most_common(10):
+                print(f"{word}: {count}")
+
+        freq_arr = np.array(list(freq_counter.values()))
+
+        min_freq_value = np.quantile(freq_arr, min_freq, method="lower")
+        max_freq_value = np.quantile(freq_arr, max_freq, method="higher")
+        
+
+        vocab = {}
+
+        for word in freq_counter.keys():
+            if min_freq_value <= freq_counter[word] <= max_freq_value and len(doc_frequency[word]) >= min_doc_frequency:
+                vocab[word] = freq_counter[word]
+
+        vocab = {word for word in freq_counter.keys() 
+                if min_freq_value <= freq_counter[word] <= max_freq_value 
+                and len(doc_frequency[word]) >= min_doc_frequency}
+
+        # Sorting the vocabulary alphabetically
+        vocab = sorted(list(vocab))
+        
+        return vocab
+
+    def compute_words_topics(self, corpus: list[str], vocab: list[str], labels: np.ndarray) -> dict:
+        """
+        Compute the words per topic.
+
+        Args:
+            corpus (list[str]): List of documents.
+            vocab (list[str]): List of words in the corpus sorted alphabetically.
+            labels (np.ndarray): Cluster labels. -1 means outlier.
+
+        Returns:
+            dict: Dictionary of topics and their words.
+        """
+
+
+        # Download NLTK resources (only required once)
+        nltk.download("punkt")
+        vocab = set(vocab)
+
+        words_per_topic = {label: [] for label in np.unique(labels) if label != -1}
+
+        for doc, label in tqdm(zip(corpus, labels), desc="Computing words per topic", total=len(corpus)):
+            if label != -1:
+                words = word_tokenize(doc)
+                for word in words:
+                    if word.lower() in vocab:
+                        words_per_topic[label].append(word.lower())
+
+        return words_per_topic
+                    
+    def embed_vocab_openAI(self, client, vocab: list[str], embedder: GetEmbeddingsOpenAI = None) -> dict[str, np.ndarray]:
+        """
+        Embed the vocabulary using the OpenAI embedding API.
+
+        Args:
+            client: Client.
+            vocab (list[str]): List of words in the corpus sorted alphabetically.
+            embedder (GetEmbeddingsOpenAI, optional): Embedding object.
+
+        Returns:
+            dict[str, np.ndarray]: Dictionary of words and their embeddings.
+        """
+
+        vocab = sorted(list(set(vocab)))
+        if embedder is None: 
+            embedder = GetEmbeddingsOpenAI.GetEmbeddingsOpenAI(client)
+        result = embedder.get_embeddings(vocab)
+
+        res_dict = {}
+        for word, emb in zip(vocab, result["embeddings"]):
+            res_dict[word] = emb
+        return res_dict
+    
+    def compute_bow_representation(self, document: str, vocab: list[str], vocab_set: set[str]) -> np.ndarray:
+        """
+        Compute the bag-of-words representation of a document.
+
+        Args:
+            document (str): Document to compute the bag-of-words representation of.
+            vocab (list[str]): List of words in the corpus sorted alphabetically.
+            vocab_set (set[str]): Set of words in the corpus sorted alphabetically.
+
+        Returns:
+            np.ndarray: Bag-of-words representation of the document.
+        """
+
+        bow = np.zeros(len(vocab))
+        words = word_tokenize(document)
+        if vocab_set is None:
+            vocab_set = set(vocab)
+        for word in words:
+            if word.lower() in vocab_set:
+                bow[vocab.index(word.lower())] += 1
+        return bow   
+    
+    def compute_word_topic_mat_old(self, corpus: list[str], vocab: list[str], labels: np.ndarray, consider_outliers: bool = False) -> np.ndarray:
+        """
+        Compute the word-topic matrix.
+
+        Args:
+            corpus (list[str]): List of documents.
+            vocab (list[str]): List of words in the corpus sorted alphabetically.
+            labels (np.ndarray): Cluster labels. -1 means outlier.
+            consider_outliers (bool, optional): Whether to consider outliers when computing the top words. I.e. whether the labels contain -1 to indicate outliers.
+
+        Returns:
+            np.ndarray: Word-topic matrix.
+        """
+
+        if consider_outliers:
+            word_topic_mat = np.zeros(len(vocab), len((np.unique(labels))))
+        else:
+            word_topic_mat = np.zeros((len(vocab), len((np.unique(labels)) - 1)))
+
+        vocab_set = set(vocab)
+        for i, doc in tqdm(enumerate(corpus), desc="Computing word-topic matrix", total=len(corpus)):
+            if labels[i] > - 0.5:
+                bow = self.compute_bow_representation(doc, vocab, vocab_set)
+                idx_to_add = labels[i]
+                word_topic_mat[:, idx_to_add] += bow
+
+        return word_topic_mat
+    
+    def compute_word_topic_mat(self, corpus: list[str], vocab: list[str], labels: np.ndarray, consider_outliers=False) -> np.ndarray:
+        """
+        Compute the word-topic matrix efficiently.
+
+        Args:
+            corpus (list[str]): List of documents.
+            vocab (list[str]): List of words in the corpus, sorted alphabetically.
+            labels (np.ndarray): Cluster labels. -1 indicates outliers.
+            consider_outliers (bool, optional): Whether to consider outliers when computing the top words. Defaults to False.
+
+        Returns:
+            np.ndarray: Word-topic matrix.
+        """
+
+
+        corpus_arr = np.array(corpus) 
+
+        if consider_outliers:
+            word_topic_mat = np.zeros((len(vocab), len((np.unique(labels)))))
+        else:
+            word_topic_mat = np.zeros((len(vocab), len((np.unique(labels)))))
+        
+        for i, label in tqdm(enumerate(np.unique(labels)), desc="Computing word-topic matrix", total=len(np.unique(labels))):
+            topic_docs = corpus_arr[labels == label]
+            topic_doc_string = " ".join(topic_docs)
+            topic_doc_words = word_tokenize(topic_doc_string)
+            topic_doc_counter = Counter(topic_doc_words)
+
+            word_topic_mat[:, i] = np.array([topic_doc_counter.get(word, 0) for word in vocab])
+        
+        return word_topic_mat
+
+    def extract_topwords_tfidf(self, word_topic_mat: np.ndarray, vocab: list[str], labels: np.ndarray, top_n_words: int = 10) -> dict:
+        """
+        Extract the top words for each topic using a class-based tf-idf score.
+
+        Args:
+            word_topic_mat (np.ndarray): Word-topic matrix.
+            vocab (list[str]): List of words in the corpus sorted alphabetically.
+            labels (np.ndarray): Cluster labels. -1 means outlier.
+            top_n_words (int, optional): Number of top words to extract per topic.
+
+        Returns:
+            dict: Dictionary of topics and their top words.
+        """
+
+
+        if min(labels) == -1:
+            word_topic_mat = word_topic_mat[:, 1:]
+
+
+        with warnings.catch_warnings():
+            warnings.filterwarnings("ignore", category=RuntimeWarning)
+            tf = word_topic_mat / np.sum(word_topic_mat, axis=0)
+            idf = np.log(1 + (word_topic_mat.shape[1] / np.sum(word_topic_mat > 0, axis=1)))
+
+            tfidf = tf * idf[:, np.newaxis]
+        
+            # set tfidf to zero if tf is nan (happens if word does not occur in any document or topic does not have any words)
+            tfidf[np.isnan(tf)] = 0
+
+        # extract top words for each topic
+        top_words = {}
+        top_word_scores = {}
+        for topic in np.unique(labels):
+            if topic != -1:
+                indices = np.argsort(-tfidf[:, topic])[:top_n_words]
+                top_words[topic] = [vocab[word_idx] for word_idx in indices]
+                top_word_scores[topic] = [tfidf[word_idx, topic] for word_idx in indices]
+
+
+        return top_words, top_word_scores
+    
+    def compute_embedding_similarity_centroids(self, vocab: list[str], vocab_embedding_dict: dict, umap_mapper: umap.UMAP, centroid_dict: dict, reduce_vocab_embeddings: bool = False, reduce_centroid_embeddings: bool = False) -> np.ndarray:
+        """
+        Compute the cosine similarity of each word in the vocabulary to each centroid.
+
+        Args:
+            vocab (list[str]): List of words in the corpus sorted alphabetically.
+            vocab_embedding_dict (dict): Dictionary of words and their embeddings.
+            umap_mapper (umap.UMAP): UMAP mapper to transform new embeddings in the same way as the document embeddings.
+            centroid_dict (dict): Dictionary of cluster labels and their centroids. -1 means outlier.
+            reduce_vocab_embeddings (bool, optional): Whether to reduce the vocab embeddings with the UMAP mapper.
+            reduce_centroid_embeddings (bool, optional): Whether to reduce the centroid embeddings with the UMAP mapper.
+
+        Returns:
+            np.ndarray: Cosine similarity of each word in the vocab to each centroid. Has shape (len(vocab), len(centroid_dict) - 1).
+        """
+
+        embedding_dim = umap_mapper.n_components
+        centroid_arr = np.zeros((len(centroid_dict), embedding_dim))
+        for i, centroid in enumerate(centroid_dict.values()):
+            centroid_arr[i] = centroid
+        if reduce_centroid_embeddings:
+            centroid_arr = umap_mapper.transform(centroid_arr)
+        
+        centroid_arr = centroid_arr / np.linalg.norm(centroid_arr, axis=1).reshape(-1,1)
+        
+
+        org_embedding_dim = list(vocab_embedding_dict.values())[0].shape[0]
+        vocab_arr = np.zeros((len(vocab), org_embedding_dim))
+        for i, word in enumerate(vocab):
+            vocab_arr[i] = vocab_embedding_dict[word]
+        if reduce_vocab_embeddings:
+            vocab_arr = umap_mapper.transform(vocab_arr)
+
+        vocab_arr = vocab_arr / np.linalg.norm(vocab_arr, axis=1).reshape(-1,1)
+        
+        similarity = vocab_arr @ centroid_arr.T # cosine similarity
+        return similarity
+    
+    def extract_topwords_centroid_similarity(self, word_topic_mat: np.ndarray, vocab: list[str], vocab_embedding_dict: dict, centroid_dict: dict, umap_mapper: umap.UMAP, top_n_words: int = 10, reduce_vocab_embeddings: bool = True, reduce_centroid_embeddings: bool = False, consider_outliers: bool = False) -> tuple[dict, np.ndarray]:
+        """
+        Extract the top words for each cluster by computing the cosine similarity of the words that occur in the corpus to the centroid of the cluster.
+
+        Args:
+            word_topic_mat (np.ndarray): Word-topic matrix.
+            vocab (list[str]): List of words in the corpus sorted alphabetically.
+            vocab_embedding_dict (dict): Dictionary of words and their embeddings.
+            centroid_dict (dict): Dictionary of cluster labels and their centroids. -1 means outlier.
+            umap_mapper (umap.UMAP): UMAP mapper to transform new embeddings in the same way as the document embeddings.
+            top_n_words (int, optional): Number of top words to extract per topic.
+            reduce_vocab_embeddings (bool, optional): Whether to reduce the vocab embeddings with the UMAP mapper.
+            reduce_centroid_embeddings (bool, optional): Whether to reduce the centroid embeddings with the UMAP mapper.
+            consider_outliers (bool, optional): Whether to consider outliers when computing the top words. I.e., whether the labels contain -1 to indicate outliers.
+
+        Returns:
+            dict: Dictionary of topics and their top words.
+            np.ndarray: Cosine similarity of each word in the vocab to each centroid. Has shape (len(vocab), len(centroid_dict) - 1).
+        """
+
+        similarity_mat = self.compute_embedding_similarity_centroids(vocab, vocab_embedding_dict, umap_mapper, centroid_dict, reduce_vocab_embeddings, reduce_centroid_embeddings)
+        top_words = {}
+        top_word_scores = {}
+        
+        if word_topic_mat.shape[1] > len(np.unique(list(centroid_dict.keys()))):	
+            word_topic_mat = word_topic_mat[:, 1:] #ignore outliers
+
+        for i, topic in enumerate(np.unique(list(centroid_dict.keys()))):
+            if topic != -1:
+                topic_similarity_mat = similarity_mat[:, topic] * word_topic_mat[:, topic]
+                top_words[topic] = [vocab[word_idx] for word_idx in np.argsort(-topic_similarity_mat)[:top_n_words]]
+                top_word_scores[topic] = [similarity_mat[word_idx, topic] for word_idx in np.argsort(-similarity_mat[:, topic])[:top_n_words]]
+
+        return top_words, top_word_scores

+from openai import OpenAI
+
+import tiktoken
+from tqdm import tqdm
+import numpy as np
+
+class GetEmbeddingsOpenAI:
+    """
+    This class allows to compute embeddings of text using the OpenAI API.
+    """
+
+    def __init__(self, client, azure_config: dict = {}, embedding_model: str = "text-embedding-ada-002", tokenizer: str = None, max_tokens: int = 8191) -> None:
+        """
+        Constructor of the class.
+
+        Args:
+            client: Client.
+            embedding_model (str, optional): Name of the embedding model to use.
+            tokenizer (str, optional): Name of the tokenizer to use.
+            max_tokens (int, optional): Maximum number of tokens to use.
+
+        Note:
+            By default, the embedding model "text-embedding-ada-002" is used with the corresponding tokenizer "cl100k_base" and a maximum number of tokens of 8191.
+        """
+
+        self.client = client
+        self.embedding_model = embedding_model
+        self.tokenizer_str = tokenizer
+        self.max_tokens = max_tokens
+
+    @staticmethod
+    def num_tokens_from_string(string: str, encoding) -> int:
+        """
+        Returns the number of tokens in a text string.
+
+        Args:
+            string (str): Text string to compute the number of tokens.
+            encoding: A function to encode the string into tokens.
+
+        Returns:
+            int: Number of tokens in the text string.
+        """
+        num_tokens = len(encoding.encode(string))
+        return num_tokens
+
+    def compute_number_of_tokens(self, corpus: list[str]) -> int:
+        """
+        Computes the total number of tokens needed to embed the corpus.
+
+        Args:
+            corpus (list[str]): List of strings to embed, where each element in the list is a document.
+
+        Returns:
+            int: Total number of tokens needed to embed the corpus.
+        """
+
+
+        if self.tokenizer_str is None:
+             tokenizer = tiktoken.encoding_for_model(self.embedding_model)
+
+        else: 
+             tokenizer = tiktoken.get_encoding(self.tokenizer_str)
+
+        num_tokens = 0
+        for document in tqdm(corpus):
+            num_tokens += self.num_tokens_from_string(document, tokenizer)
+
+        return num_tokens
+
+    def split_doc(self, text):
+        """
+        Splits a single document that is longer than the maximum number of tokens into a list of smaller documents.
+
+        Args:
+            self: The instance of the class.
+            text (str): The string to be split.
+
+        Returns:
+            List[str]: A list of strings to embed, where each element in the list is a list of chunks comprising the document.
+        """
+
+        split_text = []
+        split_text.append(text[:self.max_tokens])
+        for i in range(1, len(text) // self.max_tokens):
+            split_text.append(text[i * self.max_tokens:(i + 1) * self.max_tokens])
+        split_text.append(text[(len(text) // self.max_tokens) * self.max_tokens:])
+        return split_text
+
+    def split_long_docs(self, text: list[str]) -> list[list[str]]:
+        """
+        Splits all documents that are longer than the maximum number of tokens into a list of smaller documents.
+
+        Args:
+            self: The instance of the class.
+            text (list[str]): List of strings to embed, where each element in the list is a document.
+
+        Returns:
+            List[list[str]]: A list of lists of strings to embed, where each element in the outer list is a list of chunks comprising the document.
+        """
+
+        if self.tokenizer_str is None:
+            tokenizer = tiktoken.encoding_for_model(self.embedding_model)
+        else:
+            tokenizer = tiktoken.get_encoding(self.tokenizer_str)
+
+
+        split_text = []
+        for document in tqdm(text):
+            if self.num_tokens_from_string(document, tokenizer) > self.max_tokens:
+                split_text.append(self.split_doc(document))
+            else:
+                split_text.append([document])
+        return split_text   
+
+    def make_api_call(self, text: str):
+        """
+        Makes an API call to the OpenAI API to embed a text string.
+
+        Args:
+            self: The instance of the class.
+            text (str): The string to embed.
+
+        Returns:
+            API response: The response from the API.
+        """
+        response = self.client.embeddings.create(input = [text], model = self.embedding_model)
+        return response
+
+
+
+    def get_embeddings_doc_split(self, corpus: list[list[str]], n_tries=3) -> list[dict]:
+        """
+        Computes the embeddings of a corpus for split documents.
+
+        Args:
+            self: The instance of the class.
+            corpus (list[list[str]]): List of strings to embed, where each element is a document represented by a list of its chunks.
+            n_tries (int, optional): Number of tries to make an API call (default is 3).
+
+        Returns:
+            List[dict]: A list of dictionaries, where each dictionary contains the embedding of the document, the text of the document, and a list of errors that occurred during the embedding process.
+        """
+
+        api_res_list = [] 
+        for i in tqdm(range(len(corpus))):
+            chunk_lis = corpus[i]
+            api_res_doc = []
+            for chunk_n, chunk in enumerate(chunk_lis):
+
+                for i in range(n_tries + 1):
+                    try: 
+                        api_res_doc.append(
+                            {"api_res": self.make_api_call(chunk), 
+                            "error": None }
+                         )
+                        break
+                    except Exception as e:
+                            print(f"Error {e} occured for chunk {chunk_n} of document {i}")
+                            print(chunk)
+                            print("Trying again.")
+                            if i == n_tries: 
+                                print("Maximum number of tries reached. Skipping chunk.")
+                                api_res_doc.append(
+                                    {"api_res": None, 
+                                    "error": e })
+
+
+            # average the embeddings of the chunks
+            emb_lis = []
+            for api_res in api_res_doc:
+                if api_res["api_res"] is not None:
+                    emb_lis.append(np.array(api_res["api_res"].data[0].embedding))
+            text = " ".join(chunk_lis)
+            embedding = np.mean(emb_lis, axis = 0)
+            api_res_list.append(
+                {"embedding": embedding, 
+                "text": text, 
+                "errors": [api_res["error"] for api_res in api_res_doc]}
+                )
+        return api_res_list
+
+    def convert_api_res_list(self, api_res_list: list[dict]) -> dict:
+        """
+        Converts the api_res list into a dictionary containing the embeddings as a matrix and the corpus as a list of strings.
+
+        Args:
+            self: The instance of the class.
+            api_res_list (list[dict]): List of dictionaries, where each dictionary contains the embedding of the document, the text of the document, and a list of errors that occurred during the embedding process.
+
+        Returns:
+            dict: A dictionary containing the embeddings as a matrix and the corpus as a list of strings.
+        """
+
+
+        embeddings = np.array([api_res["embedding"] for api_res in api_res_list])
+        corpus = [api_res["text"] for api_res in api_res_list]
+        errors = [api_res["errors"] for api_res in api_res_list]
+        return {"embeddings": embeddings, "corpus": corpus, "errors": errors}
+
+
+    def get_embeddings(self, corpus: list[str]) -> dict:
+        """
+        Computes the embeddings of a corpus.
+
+        Args:
+            self: The instance of the class.
+            corpus (list[str]): List of strings to embed, where each element in the list is a document.
+
+        Returns:
+            dict: A dictionary containing the embeddings as a matrix and the corpus as a list of strings.
+        """
+
+        corpus_split = self.split_long_docs(corpus)
+        corpus_emb = self.get_embeddings_doc_split(corpus_split)
+        self.corpus_emb = corpus_emb
+        res = self.convert_api_res_list(corpus_emb)
+        return res

+from topicgpt.TopicRepresentation import Topic
+
+import unittest
+from sklearn.datasets import fetch_20newsgroups 
+
+from topicgpt.TopicGPT import TopicGPT
+
+    
+import sys
+
+
+class QuickestTopicGPT_prompting(unittest.TestCase):
+    """
+    This class is used to mainly test the prompting functionality of the TopicGPT class.
+    """
+
+
+    @classmethod
+    def setUpClass(cls, sample_size:int = 500):
+        """
+        download the necessary data and only keep a sample of it 
+        params: 
+            client: Client.
+            sample_size: the number of documents to use for the test
+        """
+
+        data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')) #download the 20 Newsgroups dataset
+        corpus = data['data']# just select the first 1000 documents for this example
+        corpus = [doc for doc in corpus if doc != ""]
+        corpus = corpus[:sample_size]
+
+        cls.corpus = corpus
+
+        cls.tm = TopicGPT(client = client, n_topics = 1)
+        cls.tm.fit(cls.corpus)
+
+    def test_repr_topics(self):
+        """
+        test the repr_topics function of the TopicGPT class
+        """
+        print("Testing repr_topics...")
+        self.assertTrue(type(self.tm.repr_topics()) == str)
+
+    def test_promt_knn_search(self):
+        """
+        test the ppromt function that calls knn_search of the TopicPrompting class
+        """
+        print("Testing ppromt_knn_search...")
+        
+        prompt_lis = ["Is topic 0 about Bananas? Use knn Search",
+                      "Is topic 0 about Space? Use knn Search"]
+        
+        for prompt in prompt_lis:
+
+            answer, function_result = self.tm.prompt(prompt)
+
+            print(f"Answer to the prompt '{prompt}' \n is \n '{answer}'")
+
+            self.assertTrue(type(answer) == str)
+            self.assertTrue(type(function_result[0]) == list)
+            self.assertTrue(type(function_result[1]) == list)
+            self.assertTrue(type(function_result[0][0]) == str)
+            self.assertTrue(type(function_result[1][0]) == int)
+
+
+    def test_prompt_split_topic_kmeans_inplace(self):
+        """
+        test the ppromt function that calls split_topic_kmeans of the TopicPrompting class
+        """
+
+        print("Testing ppromt_split_topic_kmeans...")
+
+        prompt_lis = ["Split topic 0 into 2 subtopics using kmeans. Do this inplace"]
+        added_topic_lis_len  = [2]
+
+        old_number_of_topics = len(self.tm.topic_lis)
+
+        for prompt, added_topic_len in zip(prompt_lis, added_topic_lis_len):
+                
+                answer, function_result = self.tm.prompt(prompt)
+    
+                print(f"Answer to the prompt '{prompt}' \n is \n '{answer}'")
+                print("function_result: ", function_result)
+    
+                self.assertTrue(type(answer) == str)
+                self.assertTrue(type(function_result) == list)
+                self.assertTrue(type(function_result[0]) == Topic)
+
+                self.assertTrue(len(self.tm.topic_lis) == old_number_of_topics + added_topic_len -1 )
+                self.assertTrue(self.tm.topic_lis == function_result)
+
+   
+    def test_prompt_combine_topics_inplace(self):
+        """
+        test the prompt function that calls combine_topics of the TopicPrompting class
+        """
+
+        print("Testing ppromt_combine_topics...")
+
+        prompt_lis = ["Combine topic 0 and topic 1 into one topic. Do this inplace"]
+
+        # split topic first
+        self.tm.prompt("Please split topic 0 into two subtopic. Do this inplace.")
+
+        old_number_topics = len(self.tm.topic_lis)
+
+
+
+        for prompt in prompt_lis:
+                
+                answer, function_result = self.tm.prompt(prompt)
+    
+                print(f"Answer to the prompt '{prompt}' \n is \n '{answer}'")
+                print("function_result: ", function_result)
+                print("topic_gpt_topic_list: ", self.tm.topic_lis)
+    
+                self.assertTrue(type(answer) == str)
+                self.assertTrue(type(function_result) == list)
+                self.assertTrue(type(function_result[0]) == Topic)
+                self.assertTrue(self.tm.topic_lis == function_result)
+                self.assertTrue(len(self.tm.topic_lis) == old_number_topics -1)
+
+
+if __name__ == "__main__":
+    
+    for i, arg in enumerate(sys.argv):
+        if arg == "--api-key":
+            api_key = sys.argv.pop(i + 1)
+            sys.argv.pop(i)
+            break
+
+    if api_key is None:
+        print("API key must be provided with --api-key")
+        sys.exit(1)
+
+    
+    unittest.main()

+from topicgpt.TopicRepresentation import Topic
+
+import unittest
+from sklearn.datasets import fetch_20newsgroups 
+
+from topicgpt.TopicGPT import TopicGPT
+
+
+class QuickTestTopicGPT_init_and_fit(unittest.TestCase):
+    """
+    Run some basic tests on TopicGPT that do not require any saved data
+    """
+
+
+    @classmethod
+    def setUpClass(cls, sample_size:int = 500):
+        """
+        download the necessary data and only keep a sample of it 
+        params: 
+            api_key: the openai api key
+            sample_size: the number of documents to use for the test
+        """
+
+        data = fetch_20newsgroups(subset='all', remove=('headers', 'footers', 'quotes')) #download the 20 Newsgroups dataset
+        corpus = data['data']# just select the first 1000 documents for this example
+        corpus = [doc for doc in corpus if doc != ""]
+        corpus = corpus[:sample_size]
+
+        cls.corpus = corpus
+
+    def setUp(self):
+        self.api_key_openai = api_key
+
+
+    def test_init(self):
+        """
+        test the init function of the TopicGPT class
+        """
+        print("Testing init...")
+        topicgpt = TopicGPT(api_key = self.api_key_openai)
+        self.assertTrue(isinstance(topicgpt, TopicGPT))
+
+        topicgpt = TopicGPT(api_key = self.api_key_openai, 
+                            n_topics= 20)
+        self.assertTrue(isinstance(topicgpt, TopicGPT))
+        
+        topicgpt = TopicGPT(api_key = self.api_key_openai, 
+                            n_topics= 20,
+                            corpus_instruction="This is a corpus instruction")
+        self.assertTrue(isinstance(topicgpt, TopicGPT))
+
+        # check if assertions are triggered
+
+        with self.assertRaises(AssertionError):
+            topicgpt = TopicGPT(api_key = None, 
+                                n_topics= 32,
+                                openai_prompting_model="gpt-4",
+                                max_number_of_tokens=8000,
+                                corpus_instruction="This is a corpus instruction")
+
+        with self.assertRaises(AssertionError):
+            topicgpt = TopicGPT(api_key = self.api_key_openai, 
+                                n_topics= 0,
+                                max_number_of_tokens=8000,
+                                corpus_instruction="This is a corpus instruction")
+            
+        with self.assertRaises(AssertionError):
+            topicgpt = TopicGPT(api_key = self.api_key_openai, 
+                                n_topics= 20,
+                                max_number_of_tokens=0,
+                                corpus_instruction="This is a corpus instruction")
+            
+
+    def test_fit(self):
+        """
+        test the fit function of the TopicGPT class
+        """
+        print("Testing fit...")
+
+        def instance_test(topicgpt):
+            topicgpt.fit(self.corpus)
+
+            self.assertTrue(hasattr(topicgpt, "vocab"))
+            self.assertTrue(hasattr(topicgpt, "topic_lis"))
+
+            self.assertTrue(isinstance(topicgpt.vocab, list))
+            self.assertTrue(isinstance(topicgpt.vocab[0], str))
+
+            self.assertTrue(isinstance(topicgpt.topic_lis, list))
+            self.assertTrue(type(topicgpt.topic_lis[0]) == Topic)
+
+            if topicgpt.n_topics is not None:
+                self.assertTrue(len(topicgpt.topic_lis) == topicgpt.n_topics)
+
+            self.assertTrue(topicgpt.topic_lis == topicgpt.topic_prompting.topic_lis)
+            self.assertTrue(topicgpt.vocab == topicgpt.topic_prompting.vocab)
+            self.assertTrue(topicgpt.vocab_embeddings == topicgpt.topic_prompting.vocab_embeddings)
+
+        
+        topicgpt1 = TopicGPT(api_key = self.api_key_openai, n_topics = 1)
+
+        topic_gpt_list = [topicgpt1]
+
+        for topic_gpt in topic_gpt_list:
+            instance_test(topic_gpt)
+
+
+import sys
+
+if __name__ == "__main__":
+    for i, arg in enumerate(sys.argv):
+        if arg == "--api-key":
+            api_key = sys.argv.pop(i + 1)
+            sys.argv.pop(i)
+            break
+
+    if api_key is None:
+        print("API key must be provided with --api-key")
+        sys.exit(1)
+    unittest.main()

+import numpy as np
+import os
+import pickle
+# make sure the import works even if the package has not been installed and just the files are used
+from topicgpt.Clustering import Clustering_and_DimRed
+from topicgpt.ExtractTopWords import ExtractTopWords
+from topicgpt.TopwordEnhancement import TopwordEnhancement
+from topicgpt.GetEmbeddingsOpenAI import GetEmbeddingsOpenAI
+from topicgpt.TopicPrompting import TopicPrompting
+from topicgpt.TopicRepresentation import Topic
+from topicgpt.Client import Client
+import topicgpt.TopicRepresentation as TopicRepresentation
+
+
+embeddings_path= "SavedEmbeddings/embeddings.pkl" #global variable for the path to the embeddings
+
+class TopicGPT:
+    """
+    This is the main class for doing topic modelling with TopicGPT. 
+    """
+
+    def __init__(self,
+             api_key: str = "",
+             azure_endpoint: dict = {},
+             n_topics: int = None,
+             openai_prompting_model: str = "gpt-3.5-turbo-16k",
+             max_number_of_tokens: int = 16384,
+             corpus_instruction: str = "",
+             document_embeddings: np.ndarray = None,
+             vocab_embeddings: dict[str, np.ndarray] = None,
+             embedding_model: str = "text-embedding-ada-002",
+             max_number_of_tokens_embedding: int = 8191,
+             use_saved_embeddings: bool = True,
+             path_saved_embeddings: str = embeddings_path,
+             clusterer: Clustering_and_DimRed = None,
+             n_topwords: int = 2000,
+             n_topwords_description: int = 500,
+             topword_extraction_methods: list[str] = ["tfidf", "cosine_similarity"],
+             compute_vocab_hyperparams: dict = {},
+             enhancer: TopwordEnhancement = None,
+             topic_prompting: TopicPrompting = None,
+             verbose: bool = True) -> None:
+        
+        """
+        Initializes the main class for conducting topic modeling with TopicGPT.
+
+        Args:
+            api_key (str): Your OpenAI API key. Obtain this key from https://beta.openai.com/account/api-keys.
+            n_topics (int, optional): Number of topics to discover. If None, the Hdbscan algorithm (https://pypi.org/project/hdbscan/) is used to determine the number of topics automatically. Otherwise, agglomerative clustering is used. Note that with insufficient data, fewer topics may be found than specified.
+            openai_prompting_model (str, optional): Model provided by OpenAI for topic description and prompts. Refer to https://platform.openai.com/docs/models for available models.
+            max_number_of_tokens (int, optional): Maximum number of tokens to use for the OpenAI API.
+            corpus_instruction (str, optional): Additional information about the corpus, if available, to benefit the model.
+            document_embeddings (np.ndarray, optional): Document embeddings for the corpus. If None, they will be computed using the OpenAI API.
+            vocab_embeddings (dict[str, np.ndarray], optional): Vocabulary embeddings for the corpus in a dictionary format where keys are words and values are embeddings. If None, they will be computed using the OpenAI API.
+            embedding_model (str, optional): Name of the embedding model to use. See https://beta.openai.com/docs/api-reference/text-embedding for available models.
+            max_number_of_tokens_embedding (int, optional): Maximum number of tokens to use for the OpenAI API when computing embeddings.
+            use_saved_embeddings (bool, optional): Whether to use saved embeddings. If True, embeddings are loaded from the file 'SavedEmbeddings/embeddings.pkl' or path_saved_embeddings if different. If False, embeddings are computed using the OpenAI API and saved to the file.
+            path_saved_embeddings (str, optional): Path to the saved embeddings file.
+            clusterer (Clustering_and_DimRed, optional): Clustering and dimensionality reduction object. Find the class in the "Clustering/Clustering" folder. If None, a clustering object with default parameters is used. Note that providing document and vocab embeddings and an embedding object at the same time is not sensible; the number of topics specified in the clusterer will overwrite the n_topics argument.
+            n_topwords (int, optional): Number of top words to extract and save for each topic. Note that fewer top words might be used later.
+            n_topwords_description (int, optional): Number of top words to provide to the LLM (Language Model) to describe the topic.
+            topword_extraction_methods (list[str], optional): List of methods for extracting top words. Available methods include "tfidf", "cosine_similarity", and "topword_enhancement". Refer to the file 'ExtractTopWords/ExtractTopWords.py' for more details.
+            compute_vocab_hyperparams (dict, optional): Hyperparameters for computing vocabulary embeddings. Refer to the file 'ExtractTopWords/ExtractTopWords.py' for more details.
+            enhancer (TopwordEnhancement, optional): Topword enhancement object. Used for describing topics. Find the class in the "TopwordEnhancement/TopwordEnhancement.py" folder. If None, a topword enhancement object with default parameters is used. If an openai model is specified here, it will overwrite the openai_prompting_model argument for topic description.
+            topic_prompting (TopicPrompting, optional): Topic prompting object for formulating prompts. Find the class in the "TopicPrompting/TopicPrompting.py" folder. If None, a topic prompting object with default parameters is used. If an openai model is specified here, it will overwrite the openai_prompting_model argument for topic description.
+            verbose (bool, optional): Whether to print detailed information about the process. This can be overridden by arguments in passed objects.
+        """
+        
+
+
+        # Do some checks on the input arguments
+        assert api_key is not None, "You need to provide an OpenAI API key."
+        assert n_topics is None or n_topics > 0, "The number of topics needs to be a positive integer."
+        assert max_number_of_tokens > 0, "The maximum number of tokens needs to be a positive integer."
+        assert max_number_of_tokens_embedding > 0, "The maximum number of tokens for the embedding model needs to be a positive integer."
+        assert n_topwords > 0, "The number of top words needs to be a positive integer."
+        assert n_topwords_description > 0, "The number of top words for the topic description needs to be a positive integer."
+        assert len(topword_extraction_methods) > 0, "You need to provide at least one topword extraction method."
+        assert n_topwords_description <= n_topwords, "The number of top words for the topic description needs to be smaller or equal to the number of top words."
+
+        self.client = Client(api_key = api_key, azure_endpoint = azure_endpoint)
+
+
+        self.n_topics = n_topics
+        self.openai_prompting_model = openai_prompting_model
+        self.max_number_of_tokens = max_number_of_tokens
+        self.corpus_instruction = corpus_instruction
+        self.document_embeddings = document_embeddings
+        self.vocab_embeddings = vocab_embeddings
+        self.embedding_model = embedding_model
+        self.max_number_of_tokens_embedding = max_number_of_tokens_embedding
+        self.embedder = GetEmbeddingsOpenAI(client = self.client, embedding_model = self.embedding_model, max_tokens = self.max_number_of_tokens_embedding)
+        self.clusterer = clusterer
+        self.n_topwords = n_topwords
+        self.n_topwords_description = n_topwords_description
+        self.topword_extraction_methods = topword_extraction_methods
+        self.compute_vocab_hyperparams = compute_vocab_hyperparams
+        self.enhancer = enhancer
+        self.topic_prompting = topic_prompting	
+        self.use_saved_embeddings = use_saved_embeddings
+        self.verbose = verbose
+
+        self.compute_vocab_hyperparams["verbose"] = self.verbose
+        
+        # if embeddings have already been downloaded to the folder SavedEmbeddings, then load them
+        if self.use_saved_embeddings and os.path.exists(path_saved_embeddings):
+            with open(path_saved_embeddings, "rb") as f:
+                self.document_embeddings, self.vocab_embeddings = pickle.load(f)
+
+        for elem in topword_extraction_methods:
+            assert elem in ["tfidf", "cosine_similarity", "topword_enhancement"], "Invalid topword extraction method. Valid methods are 'tfidf', 'cosine_similarity', and 'topword_enhancement'."
+        
+        if clusterer is None:
+            self.clusterer = Clustering_and_DimRed(number_clusters_hdbscan = self.n_topics, verbose = self.verbose)
+        else:
+            self.n_topics = clusterer.number_clusters_hdbscan
+        
+        if enhancer is None:
+            self.enhancer = TopwordEnhancement(client = self.client, openai_model = self.openai_prompting_model, max_context_length = self.max_number_of_tokens, corpus_instruction = self.corpus_instruction)
+
+        if topic_prompting is None:
+            self.topic_prompting = TopicPrompting(topic_lis = [], client = self.client, openai_prompting_model = self.openai_prompting_model,  max_context_length_promting = 16000, enhancer = self.enhancer, openai_embedding_model = self.embedding_model, max_context_length_embedding = self.max_number_of_tokens_embedding, corpus_instruction = corpus_instruction)
+        
+        self.extractor = ExtractTopWords()
+    
+    def __repr__(self) -> str:
+        repr = "TopicGPT object with the following parameters:\n"
+        repr += "-"*150 + "\n"
+        repr += "n_topics: " + str(self.n_topics) + "\n"
+        repr += "openai_prompting_model: " + self.openai_prompting_model + "\n"
+        repr += "max_number_of_tokens: " + str(self.max_number_of_tokens) + "\n"
+        repr += "corpus_instruction: " + self.corpus_instruction + "\n"
+        repr += "embedding_model: " + self.embedding_model + "\n"
+        repr += "clusterer: " + str(self.clusterer) + "\n"
+        repr += "n_topwords: " + str(self.n_topwords) + "\n"
+        repr += "n_topwords_description: " + str(self.n_topwords_description) + "\n"
+        repr += "topword_extraction_methods: " + str(self.topword_extraction_methods) + "\n"
+        repr += "compute_vocab_hyperparams: " + str(self.compute_vocab_hyperparams) + "\n"
+        repr += "enhancer: " + str(self.enhancer) + "\n"
+        repr += "topic_prompting: " + str(self.topic_prompting) + "\n"
+
+        return repr
+
+    def compute_embeddings(self, corpus: list[str]) -> tuple[np.ndarray, dict[str, np.ndarray]]:
+        """
+        Computes document and vocabulary embeddings for the given corpus.
+
+        Args:
+            corpus (list[str]): List of strings to embed, where each element is a document.
+
+        Returns:
+            tuple: A tuple containing two items:
+                - document_embeddings (np.ndarray): Document embeddings for the corpus, with shape (len(corpus), n_embedding_dimensions).
+                - vocab_embeddings (dict[str, np.ndarray]): Vocabulary embeddings for the corpus, provided as a dictionary where keys are words and values are embeddings.
+        """
+
+        
+        self.document_embeddings = self.embedder.get_embeddings(corpus)["embeddings"]
+
+        self.vocab_embeddings = self.extractor.embed_vocab_openAI(self.client, self.vocab, embedder = self.embedder)
+
+        return self.document_embeddings, self.vocab_embeddings
+    
+    def extract_topics(self, corpus: list[str]) -> list[Topic]:
+        """
+        Extracts topics from the given corpus.
+
+        Args:
+            corpus (list[str]): List of strings to process, where each element represents a document.
+
+        Returns:
+            list[Topic]: A list of Topic objects representing the extracted topics.
+        """
+
+        assert self.document_embeddings is not None and self.vocab_embeddings is not None, "You need to compute the embeddings first."
+
+        if self.vocab is None: 
+            self.vocab = self.extractor.compute_corpus_vocab(self.corpus, **self.compute_vocab_hyperparams)
+        
+        self.topic_lis = TopicRepresentation.extract_topics_no_new_vocab_computation(
+            corpus = corpus,
+            vocab = self.vocab,
+            document_embeddings = self.document_embeddings,
+            clusterer = self.clusterer,
+            vocab_embeddings = self.vocab_embeddings,
+            n_topwords = self.n_topwords,
+            topword_extraction_methods = self.topword_extraction_methods,
+            consider_outliers = True
+        )
+
+        return self.topic_lis
+    
+    def describe_topics(self, topics: list[Topic]) -> list[Topic]:
+        """
+        Names and describes the provided topics using the OpenAI API.
+
+        Args:
+            topics (list[Topic]): List of Topic objects to be named and described.
+
+        Returns:
+            list[Topic]: A list of Topic objects with names and descriptions.
+        """
+
+
+        assert self.topic_lis is not None, "You need to extract the topics first."
+
+        if "cosine_similarity" in self.topword_extraction_methods:
+            topword_method = "cosine_similarity"
+        elif "tfidf" in self.topword_extraction_methods:
+            topword_method = "tfidf"
+        else:
+            raise ValueError("You need to use either 'cosine_similarity' or 'tfidf' as topword extraction method.")
+
+        self.topic_lis = TopicRepresentation.describe_and_name_topics(
+            topics = topics,
+            enhancer = self.enhancer,
+            topword_method= topword_method,
+            n_words = self.n_topwords_description
+        )
+
+        return self.topic_lis
+    
+    def fit(self, corpus: list[str], verbose: bool = True):
+        """
+        Compute embeddings if necessary, extract topics, and describe them.
+
+        Args:
+            corpus (list[str]): List of strings to embed, where each element represents a document.
+            verbose (bool, optional): Whether to print the progress and details of the process.
+        """
+
+        self.corpus = corpus 
+        
+        # remove empty documents
+        len_before_removing = len(self.corpus)
+        while '' in self.corpus:
+            corpus.remove('')
+        len_after_removing = len(self.corpus)
+        if verbose:
+            print("Removed " + str(len_before_removing - len_after_removing) + " empty documents.")
+
+        if self.vocab_embeddings is None:
+            if verbose:
+                print("Computing vocabulary...")
+
+            self.vocab = self.extractor.compute_corpus_vocab(self.corpus, **self.compute_vocab_hyperparams)
+        else:
+            print('Vocab already computed')
+            self.vocab = list(self.vocab_embeddings.keys())
+
+        if self.vocab_embeddings is None or self.document_embeddings is None:  
+            if verbose:
+                print("Computing embeddings...")
+            self.compute_embeddings(corpus = self.corpus)
+        else:
+            print('Embeddings already computed')
+        if verbose: 
+            print("Extracting topics...")
+        self.topic_lis = self.extract_topics(corpus = self.corpus)
+
+        if verbose:
+            print("Describing topics...")
+        self.topic_lis = self.describe_topics(topics = self.topic_lis)
+
+        self.topic_prompting.topic_lis = self.topic_lis
+        self.topic_prompting.vocab_embeddings = self.vocab_embeddings
+        self.topic_prompting.vocab = self.vocab
+
+    def visualize_clusters(self):
+        """
+        Visualizes the identified clusters representing the topics in a scatterplot.
+        """
+
+        assert self.topic_lis is not None, "You need to extract the topics first."
+
+        all_document_embeddings = np.concatenate([topic.document_embeddings_hd for topic in self.topic_lis], axis = 0)
+        all_texts = np.concatenate([topic.documents for topic in self.topic_lis], axis = 0)
+        all_document_indices = np.concatenate([np.repeat(i, topic.document_embeddings_hd.shape[0]) for i, topic in enumerate(self.topic_lis)], axis = 0)
+        class_names = [str(topic) for topic in self.topic_lis]
+
+        self.clusterer.visualize_clusters_dynamic(all_document_embeddings, all_document_indices, all_texts, class_names)
+    
+    def repr_topics(self) -> str:
+        """
+        Returns a string explanation of the topics.
+        """
+
+        assert self.topic_lis is not None, "You need to extract the topics first."
+
+        if "cosine_similarity" in self.topword_extraction_methods:
+            topword_method = "cosine_similarity"
+        elif "tfidf" in self.topword_extraction_methods:
+            topword_method = "tfidf"
+        else:
+            raise ValueError("You need to use either 'cosine_similarity' or 'tfidf' as topword extraction method.")
+
+        repr = ""
+        for topic in self.topic_lis:
+            repr += str(topic) + "\n"
+            repr += "Topic_description: " + topic.topic_description + "\n"
+            repr += "Top words: " + str(topic.top_words[topword_method][:10]) + "\n"
+            repr += "\n"
+            repr += "-"*150 + "\n"
+
+        return repr
+
+    def print_topics(self):
+        """
+        Prints a string explanation of the topics.
+        """
+   
+        print(self.repr_topics())
+
+    def prompt(self, query: str) -> tuple[str, object]:
+        """
+        Prompts the model with the given query.
+
+        Args:
+            query (str): The query to prompt the model with.
+
+        Returns:
+            tuple: A tuple containing two items:
+                - answer (str): The answer from the model.
+                - function_result (object): The result of the function call.
+        
+        Note:
+            Please refer to the TopicPrompting class for more details on available functions for prompting the model.
+        """
+
+
+        result = self.topic_prompting.general_prompt(query)
+
+        answer = result[0][-1].choices[0].message.content
+        function_result = result[1]
+        self.topic_prompting._fix_dictionary_topwords()
+        self.topic_lis = self.topic_prompting.topic_lis
+
+        return answer, function_result
+    
+    def pprompt(self, query: str, return_function_result: bool = True) -> object:
+        """
+        Prompts the model with the given query and prints the answer.
+
+        Args:
+            query (str): The query to prompt the model with.
+            return_function_result (bool, optional): Whether to return the result of the function call by the Language Model (LLM).
+
+        Returns:
+            object: The result of the function call if return_function_result is True, otherwise None.
+        """
+
+
+        answer, function_result = self.prompt(query)
+
+        print(answer)
+
+        if return_function_result:
+            return function_result
+        
+    def save_embeddings(self, path: str = embeddings_path) -> None:
+        """
+        Saves the document and vocabulary embeddings to a pickle file for later re-use.
+
+        Args:
+            path (str, optional): The path to save the embeddings to. Defaults to embeddings_path.
+        """
+
+
+        assert self.document_embeddings is not None and self.vocab_embeddings is not None, "You need to compute the embeddings first."
+
+        # create dictionary if it doesn't exist yet 
+        if not os.path.exists("SavedEmbeddings"):
+            os.makedirs("SavedEmbeddings")
+
+
+        with open(path, "wb") as f:
+            pickle.dump([self.document_embeddings, self.vocab_embeddings], f)
+

+import openai
+from openai import OpenAI
+import numpy as np
+import json
+import tiktoken
+import openai
+from openai import OpenAI
+import re
+import sklearn
+import hdbscan
+from copy import deepcopy
+
+# make sure the import works even if the package has not been installed and just the files are used
+try: 
+    from topicgpt.TopicRepresentation import Topic
+    from topicgpt.TopicRepresentation import extract_and_describe_topic_cos_sim
+    from topicgpt.TopicRepresentation import extract_describe_topics_labels_vocab
+    from topicgpt.TopwordEnhancement import TopwordEnhancement
+except:
+    from TopicRepresentation import Topic
+    from TopicRepresentation import extract_and_describe_topic_cos_sim
+    from TopicRepresentation import extract_describe_topics_labels_vocab
+    from TopwordEnhancement import TopwordEnhancement
+
+
+basic_model_instruction = """You are a helpful assistant. 
+You are excellent at inferring information about topics discovered via topic modelling using information retrieval. 
+You summarize information intelligently. 
+You use the functions you are provided with if applicable.
+You make sure that everything you output is strictly based on the provided text. If you cite documents, give their indices. 
+You always explicitly say if you don't find any useful information!
+You only say that something is contained in the corpus if you are very sure about it!"""
+
+
+class TopicPrompting:
+    """
+    This class allows to formulate prompts and queries against the identified topics to get more information about them
+    """
+
+    def __init__(self, 
+             topic_lis: list[Topic], 
+             client,
+             openai_prompting_model: str = "gpt-3.5-turbo-16k", 
+             max_context_length_promting: int = 16000, 
+             openai_model_temperature_prompting: float = 0.5,
+             openai_embedding_model: str = "text-embedding-ada-002",
+             max_context_length_embedding: int = 8191, 
+             basic_model_instruction: str = basic_model_instruction,
+             corpus_instruction: str = "",
+             enhancer: TopwordEnhancement = None,
+             vocab: list = None,
+             vocab_embeddings: dict = None,
+             random_state: int = 42):
+        """
+        Initialize the object.
+
+        Args:
+            topic_list (list[Topic]): List of Topic objects.
+            client: Client.
+            openai_prompting_model (str, optional): OpenAI model to use for prompting (default is "gpt-3.5-turbo-16k").
+            max_context_length_prompting (int, optional): Maximum context length for the prompting model (default is 16000).
+            openai_model_temperature_prompting (float, optional): Temperature for the prompting model (default is 0.5).
+            openai_embedding_model (str, optional): OpenAI model to use for computing embeddings for similarity search (default is "text-embedding-ada-002").
+            max_context_length_embedding (int, optional): Maximum context length for the embedding model (default is 8191).
+            basic_model_instruction (str, optional): Basic instruction for the prompting model.
+            corpus_instruction (str, optional): Instruction for the prompting model to use the corpus.
+            enhancer (TopwordEnhancement, optional): TopwordEnhancement object for naming and describing the topics (default is None).
+            vocab (list, optional): Vocabulary of the corpus (default is None).
+            vocab_embeddings (dict, optional): Dictionary mapping words to their embeddings (default is None).
+            random_state (int, optional): Random state for reproducibility (default is 42).
+        """
+
+        self.topic_lis = topic_lis
+        self.client = client
+        self.openai_prompting_model = openai_prompting_model
+        self.max_context_length_promting = max_context_length_promting
+        self.openai_model_temperature_prompting = openai_model_temperature_prompting
+        self.openai_embedding_model = openai_embedding_model
+        self.max_context_length_embedding = max_context_length_embedding    
+        self.basic_model_instruction = basic_model_instruction
+        self.corpus_instruction = f" The following information is available about the corpus used to identify the topics: {corpus_instruction}.\n"
+        self.enhancer = enhancer
+        self.vocab = vocab
+        self.vocab_embeddings = vocab_embeddings
+        self.random_state = random_state
+
+
+        self.function_descriptions = {
+                "knn_search": {
+                    "name": "knn_search",
+                    "description": "This function is the best choice to find out if a topic is about a specific subject or keyword or aspects or contains information about it. It should also be used to infer the subtopics of a given topic. Note that it is possible that just useless documents are returned.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_index": {
+                                "type": "integer",
+                                "description": "index of the topic to search in."
+                            },
+                            "query": {
+                                "type": "string",
+                                "description": "query string. Can be a single word or a sentence. Used to create an embedding and search a vector database for the k nearest neighbors."
+                            },
+                            "k": {
+                                "type": "integer",
+                                "description": "number of neighbors to return. Use more neighbors to get a more diverse and comprehensive set of results."
+                            }
+                        },
+                        "required": ["topic_index", "query"]
+
+                    }
+                },
+                "identify_topic_idx": {
+                    "name": "identify_topic_idx",
+                    "description": "This function can be used to identify the index of the topic that the query is most likely about. This is useful if the topic index is needed for other functions. It should NOT be used to find more detailed information on topics. Note that it is possible that the model does not find any topic that fits the query. In this case, the function returns None.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "query": {
+                                "type": "string",
+                                "description": "query string. Can be a single word or a sentence. Used to find the index of the topic that is most likely about the query."
+                            }
+                        },
+                        "required": ["query"]
+
+                    }
+                },
+                "split_topic_kmeans": {
+                    "name": "split_topic_kmeans",
+                    "description": "This function can be used to split a topic into several subtopics using kmeans clustering. Only use this function to actually split topics. The subtopics do not need to be specified and are found automatically via clustering. It returns the topics the original topic has been split into.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_idx": {
+                                "type": "integer",
+                                "description": "index of the topic to split."
+                            },
+                            "n_clusters": {
+                                "type": "integer",
+                                "description": "number of clusters to split the topic into. The more clusters, the more fine-grained the splitting. Typically 2 clusters are used.",
+                                "default": 2
+                            },
+                            "inplace": {
+                                "type": "boolean",
+                                "description": "if True, the topic is split inplace. Otherwise, a new list of topics is created and returned. ALWAYS set inplace to False unless something else is explicitly requested!",
+                                "default": False
+                            }
+                        },
+                        "required": ["topic_idx"]
+                    }
+                },
+                "split_topic_keywords": {
+                    "name": "split_topic_keywords",
+                    "description": "This function can be used to split a topic into subtopics according to the keywords. I.e. a topic about 'machine learning' can be split into a topic about 'supervised learning' and a topic about 'unsupervised learning'. This is achieved by computing the cosine similarity between the keywords and the documents in the topic.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_idx": {
+                                "type": "integer",
+                                "description": "index of the topic to split."
+                            },
+                            "keywords": {
+                                "type": "array",
+                                "items": {
+                                    "type": "string"
+                                },
+                                "minItems": 2,
+                                "description": "keywords to form new subtopics to replace old topic. Needs to be a list of at least two keywords."
+                            },
+                            "inplace": {
+                                "type": "boolean",
+                                "description": "if True, the topic is split inplace. Otherwise, a new list of topics is created and returned. ALWAYS set inplace to False unless something else is explicitly requested!",
+                                "default": False
+                            }
+                        },
+                        "required": ["topic_idx", "keywords"]
+                    }
+                },
+                "split_topic_single_keyword": {
+                    "name": "split_topic_single_keyword",
+                    "description": "This function can be used to split a topic into the main topic and an additional subtopic. I.e. a topic about 'machine learning' can be split into a topic about 'machine learning' and a topic about 'supervised learning.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_idx": {
+                                "type": "integer",
+                                "description": "index of the topic to split."
+                            },
+                            "keyword": {
+                                "type": "string",
+                                "description": "keyword to form new subtopic besides old main topic. Needs to be a single keyword."
+                            },
+                            "inplace": {
+                                "type": "boolean",
+                                "description": "if True, the topic is split inplace. Otherwise, a new list of topics is created and returned. ALWAYS set inplace to False unless something else is explicitly requested!",
+                                "default": False
+                            }
+                        },
+                        "required": ["topic_idx", "keyword"]
+                    }
+                },
+                "combine_topics": {
+                    "name": "combine_topics",
+                    "description": "This function can be used to combine several topics into one topic. It returns the newly formed topic and removes the old topics from the list of topics.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_idx_lis": {
+                                "type": "array",
+                                "items": {
+                                    "type": "integer"
+                                },
+                                "minItems": 2,
+                                "description": "list of topic indices to combine."
+                            },
+                            "inplace": {
+                                "type": "boolean",
+                                "description": "if True, the topic is split inplace. Otherwise, a new list of topics is created and returned. ALWAYS set inplace to False unless something else is explicitly requested!",
+                                "default": False
+                            }
+                        },
+                        "required": ["topic_idx_lis"]
+                    }
+                },
+                "add_new_topic_keyword": {
+                    "name": "add_new_topic_keyword",
+                    "description": "This function can be used to globally create a new topic based on a keyword. This is useful if the keyword is not contained in any of the topics. The new topic is created by finding the documents that are closest to the keyword and then taking away those documents from the other topics. Note that this method is computationally expensive and should only be used if splitting another topic is unavoidable.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "keyword": {
+                                "type": "string",
+                                "description": "keyword to form new topic. Needs to be a single keyword."
+                            },
+                            "inplace": {
+                                "type": "boolean",
+                                "description": "if True, the topic is split inplace. Otherwise, a new list of topics is created and returned. ALWAYS set inplace to False unless something else is explicitly requested!",
+                                "default": False
+                            }
+
+                        },
+                        "required": ["keyword"]
+                    }
+                },
+                "delete_topic": {
+                    "name": "delete_topic",
+                    "description": "This function can be used to delete a topic and assign the documents of this topic to the other topics based on centroid similarity. This is useful if the topic is not needed anymore. Note that this method is computationally expensive.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_idx": {
+                                "type": "integer",
+                                "description": "index of the topic to delete."
+                            },
+                            "inplace": {
+                                "type": "boolean",
+                                "description": "if True, the topic is split inplace. Otherwise, a new list of topics is created and returned. ALWAYS set inplace to False unless something else is explicitly requested!",
+                                "default": False
+                            }
+
+                        },
+                        "required": ["topic_idx"]
+                    }
+                },
+                "get_topic_information": {
+                    "name": "get_topic_information",
+                    "description": "This function can be used to get information about several topics. This function can be used to COMPARE topics or to get an overview over them. It returns a list of dictionaries containing the topic index and information about the topics.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_idx_lis": {
+                                "type": "array",
+                                "items": {
+                                    "type": "integer"
+                                },
+                                "minItems": 1,
+                                "description": "list of topic indices to get information about."
+                            }
+                        },
+                        "required": ["topic_idx_lis"]
+                    }
+                },
+                "split_topic_hdbscan": {
+                    "name": "split_topic_hdbscan",
+                    "description": "This function can be used to split a topic into several subtopics using hdbscan clustering. This method should be used if the number of clusters to split the topic into is not known.",
+                    "parameters": {
+                        "type": "object",
+                        "properties": {
+                            "topic_idx": {
+                                "type": "integer",
+                                "description": "index of the topic to split."
+                            },
+                            "min_cluster_size": {
+                                "type": "integer",
+                                "description": "minimum number of documents in a cluster. The higher the number, the more fine-grained the splitting.",
+                                "default": 10
+                            },
+                            "inplace": {
+                                "type": "boolean",
+                                "description": "if True, the topic is split inplace. Otherwise, a new list of topics is created and returned. ALWAYS set inplace to False unless something else is explicitly requested!",
+                                "default": False
+                            }
+                        },
+                        "required": ["topic_idx"]
+                    }
+                }
+        }
+
+        self.functionNames2Functions = {
+            "knn_search": self._knn_search_openai,
+            "identify_topic_idx": self._identify_topic_idx_openai,
+            "split_topic_kmeans": self._split_topics_kmeans_openai,
+            "split_topic_keywords": self._split_topic_keywords_openai,
+            "split_topic_single_keyword": self._split_topic_single_keyword_openai,
+            "combine_topics": self._combine_topics_openai,
+            "add_new_topic_keyword": self._add_new_topic_keyword_openai,
+            "delete_topic": self._delete_topic_openai,
+            "get_topic_information": self._get_topic_information_openai,
+            "split_topic_hdbscan": self._split_topic_hdbscan_openai
+        }
+
+    def reindex_topics(self) -> None:
+        """
+        Reindexes the topics in self.topic_list to assign correct new indices.
+
+        This method updates the indices of topics within the instance's topic list to ensure they are correctly ordered.
+
+        Returns:
+            None
+        """
+
+        for idx, topic in enumerate(self.topic_lis):
+            topic.topic_idx = idx
+
+    def reindex_topic_lis(self, topic_list: list[Topic]) -> list[Topic]:
+        """
+        Reindexes the topics in the provided topic list to assign correct new indices.
+
+        This method updates the indices of topics within the given topic list to ensure they are correctly ordered.
+
+        Args:
+            topic_list (list[Topic]): The list of Topic objects to reindex.
+
+        Returns:
+            list[Topic]: The reindexed list of Topic objects.
+        """
+
+        for idx, topic in enumerate(topic_list):
+            topic.topic_idx = idx
+        return topic_list
+
+    def show_topic_lis(self) -> str:
+        """
+        Returns a string representation of the list of topics.
+
+        This method generates a human-readable string representation of the topics in the instance's topic list.
+
+        Returns:
+            str: A string containing the representation of the list of topics.
+        """
+
+        self.reindex_topics()
+        res = ""
+        for idx, topic in enumerate(self.topic_lis):
+            res += str(topic)
+
+        print(res)
+
+    def get_topic_lis(self) -> list[Topic]:
+        """
+        Returns the list of topics stored in the instance.
+
+        This method retrieves and returns the list of topics associated with the instance.
+
+        Returns:
+            list[Topic]: The list of Topic objects.
+        """
+
+        self.reindex_topics()
+        return self.topic_lis
+
+    def set_topic_lis(self, topic_list: list[Topic]) -> None:
+        """
+        Sets the list of topics for the instance.
+
+        This method updates the list of topics associated with the instance to the provided list.
+
+        Args:
+            topic_list (list[Topic]): The list of Topic objects to set.
+
+        Returns:
+            None
+        """
+
+        self.topic_lis = topic_list
+        self.reindex_topics()
+
+    def knn_search(self, topic_index: int, query: str, k: int = 20, doc_cutoff_threshold: int = 1000) -> tuple[list[str], list[int]]:
+        """
+        Finds the k nearest neighbors of the query in the given topic based on cosine similarity in the original embedding space.
+
+        Args:
+            topic_index (int): Index of the topic to search within.
+            query (str): Query string.
+            k (int, optional): Number of neighbors to return (default is 20).
+            doc_cutoff_threshold (int, optional): Maximum number of tokens per document. Afterwards, the document is cut off (default is 1000).
+
+        Returns:
+            tuple: A tuple containing two lists -
+                - A list of top k documents (as strings).
+                - A list of indices corresponding to the top k documents in the topic.
+        """
+
+        topic = self.topic_lis[topic_index]
+
+        query_embedding = self.client.embeddings.create(input = [query], model = self.openai_embedding_model)["data"][0]["embedding"]
+
+        query_similarities = topic.document_embeddings_hd @ query_embedding / (np.linalg.norm(topic.document_embeddings_hd, axis = 1) * np.linalg.norm(query_embedding))
+
+        topk_doc_indices = np.argsort(query_similarities)[::-1][:k]
+        topk_docs = [topic.documents[i] for i in topk_doc_indices]
+
+        # cut off documents that are too long
+        max_number_tokens = self.max_context_length_promting - len(tiktoken.encoding_for_model(self.openai_prompting_model).encode(self.basic_model_instruction + " " + self.corpus_instruction)) - 100
+        n_tokens = 0
+        for i, doc in enumerate(topk_docs):
+            encoded_doc = tiktoken.encoding_for_model(self.openai_prompting_model).encode(doc)
+            n_tokens += len(encoded_doc[:doc_cutoff_threshold])
+            if n_tokens > max_number_tokens:
+                topk_docs = topk_docs[:i]
+                topk_doc_indices = topk_doc_indices[:i]
+                break
+            if len(encoded_doc) > doc_cutoff_threshold:
+                encoded_doc = encoded_doc[:doc_cutoff_threshold]
+                topk_docs[i] = tiktoken.encoding_for_model(self.openai_prompting_model).decode(encoded_doc)
+
+
+
+
+        return topk_docs, [int(elem) for elem in topk_doc_indices]
+
+    def prompt_knn_search(self, llm_query: str, topic_index: int = None, n_tries: int = 3) -> tuple[str, tuple[list[str], list[int]]]:
+        """
+        Uses the Language Model (LLM) to answer the llm_query based on the documents belonging to the topic.
+
+        Args:
+            llm_query (str): Query string for the Language Model (LLM).
+            topic_index (int, optional): Index of the topic object. If None, the topic is inferred from the query.
+            n_tries (int, optional): Number of tries to get a valid response from the LLM (default is 3).
+
+        Returns:
+            tuple: A tuple containing two elements -
+                - A string representing the answer from the LLM.
+                - A tuple containing two lists -
+                    - A list of top k documents (as strings).
+                    - A list of indices corresponding to the top k documents in the topic.
+        """
+
+        messages = [
+            {
+                "role": "system",
+                "content": self.basic_model_instruction + " " + self.corpus_instruction
+            },
+            {
+                "role": "user",
+                "content": llm_query
+            }
+            ]
+        for _ in range(n_tries):
+            try: 
+                response_message = self.client.chat.completions.create(model = self.openai_prompting_model,
+                messages = messages,
+                functions = [self.function_descriptions["knn_search"]],
+                function_call = "auto")["choices"][0]["message"]
+
+                # Step 2: check if GPT wanted to call a function
+                function_call = response_message.get("function_call")
+                if function_call is not None:
+                    #print("GPT wants to the call the function: ", function_call)
+                    # Step 3: call the function
+                    # Note: the JSON response may not always be valid; be sure to handle errors
+
+                    function_name = function_call["name"]
+                    function_to_call = self.functionNames2Functions[function_name]
+                    function_args = json.loads(function_call["arguments"])
+                    if topic_index is not None:
+                        function_args["topic_index"] = topic_index
+                    function_response = function_to_call(**function_args)
+                    function_response_json = function_response[0]
+                    function_response_return_output = function_response[1]
+
+
+
+                    # Step 4: send the info on the function call and function response to GPT
+                    messages.append(response_message)  # extend conversation with assistant's reply
+
+
+                    messages.append(
+                        {
+                            "role": "function",
+                            "name": function_name,
+                            "content": function_response_json,
+                        }
+                    )  # extend conversation with function response
+
+                    #print(messages)
+                    second_response = self.client.chat.completions.create(model=self.openai_prompting_model,
+                    messages=messages)  # get a new response from GPT where it can see the function response
+            except (TypeError, ValueError, openai.APIError, openai.APIConnectionError) as error:
+                print("Error occured: ", error)
+                print("Trying again...")
+
+            return second_response, function_response_return_output
+
+    def identify_topic_idx(self, query: str, n_tries: int = 3) -> int:
+        """
+        Identifies the index of the topic that the query is most likely about.
+
+        This method uses a Language Model (LLM) to determine which topic best fits the query description. If the LLM does not find any topic that fits the query, None is returned.
+
+        Args:
+            query (str): Query string.
+            n_tries (int, optional): Number of tries to get a valid response from the LLM (default is 3).
+
+        Returns:
+            int: The index of the topic that the query is most likely about. If no suitable topic is found, None is returned.
+        """
+
+
+        topic_descriptions_str = ""
+        for i, topic in enumerate(self.topic_lis):
+            description = topic.topic_description
+            description = f"""Topic index: {i}: \n {description} \n \n"""
+            topic_descriptions_str += description
+
+        system_prompt = f"""You are a helpful assistant."""
+
+        user_prompt = f""" Please find the index of the topic that is about the following query: {query}. 
+        Those are the given topics: '''{topic_descriptions_str}'''.
+        Please make sure to reply ONLY with an integer number between 0 and {len(self.topic_lis) - 1}!
+        Reply with -1 if you don't find any topic that fits the query!
+        Always explicitly say if you don't find any useful information by replying with -1! If in doubt, say that you did not find any useful information!
+        Reply in the following format: "The topic index is: <index>"""
+
+        messages = [
+            {
+                "role": "system",
+                "content": system_prompt
+            },
+            {
+                "role": "user",
+                "content": user_prompt
+            }
+            ]
+        for _ in range(n_tries):
+            try: 
+                response_message = self.client.chat.completions.create(model = self.openai_prompting_model,
+                messages = messages)["choices"][0]["message"]
+
+            except (TypeError, ValueError, openai.APIError, openai.APIConnectionError) as error:
+                print("Error occured: ", error)
+                print("Trying again...")
+
+
+
+        response_text = response_message["content"]
+        # find integer number in response text
+        try:
+            match = re.search(r'(-?\d+)', response_text)
+            topic_index = int(match.group(1))
+        except:  # in case the LLM does not find any topic that fits the query, return None
+            topic_index = None
+
+
+        if topic_index is None:
+            raise ValueError("No integer number found in response text! The model gave the following response: ", response_text)
+
+        if topic_index == -1: 
+            return None
+        else:
+            return topic_index
+
+    def split_topic_new_assignments(self, topic_idx: int, new_topic_assignments: np.ndarray, inplace: bool = False) -> list[Topic]:
+        """
+        Splits a topic into new topics based on new topic assignments.
+
+        Note that this method only computes topwords based on the cosine-similarity method because tf-idf topwords need expensive computation on the entire corpus. 
+        The topwords of the old topic are also just split among the new ones. No new topwords are computed in this step.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            new_topic_assignments (np.ndarray): New topic assignments for the documents in the topic.
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+
+        if self.vocab_embeddings is None:
+            raise(ValueError("Need to provide vocab_embeddings to Topic prompting class to split a topic!"))
+        if self.enhancer is None:
+            raise(ValueError("Need to provide enhancer to Topic prompting class to split a topic!"))
+
+        vocab_embedding_dict = self.vocab_embeddings
+        enhancer = self.enhancer
+
+        old_topic = self.topic_lis[topic_idx]
+
+        assert len(new_topic_assignments) == len(old_topic.documents), "new_topic_assignments must have the same length as the number of documents in the topic!"
+
+        # create new topics
+        new_topics = []
+        for i in np.unique(new_topic_assignments):
+            docs = [old_topic.documents[j] for j in range(len(old_topic.documents)) if new_topic_assignments[j] == i]
+            docs_embeddings = old_topic.document_embeddings_hd[new_topic_assignments == i]
+            words_raw = []
+            for doc in docs:
+                words_raw += doc.split(" ")
+            words_raw = set(words_raw)
+            words = [word for word in old_topic.words if word in words_raw]
+
+            new_topic = extract_and_describe_topic_cos_sim(
+                documents_topic = docs,
+                document_embeddings_topic = docs_embeddings,
+                words_topic = words,
+                vocab_embeddings = vocab_embedding_dict,
+                umap_mapper = old_topic.umap_mapper,
+                enhancer=enhancer,
+                n_topwords = 2000
+            )
+            new_topic.topic_idx = len(self.topic_lis) + i + 1
+            new_topics.append(new_topic)
+
+        new_topic_lis = self.topic_lis.copy()
+        new_topic_lis.pop(topic_idx)
+        new_topic_lis += new_topics
+        new_topic_lis = self.reindex_topic_lis(new_topic_lis)
+
+        if inplace:
+            self.topic_lis = new_topic_lis
+
+        return new_topic_lis
+
+    def split_topic_kmeans(self, topic_idx: int, n_clusters: int = 2, inplace: bool = False) -> list[Topic]:
+        """
+        Splits an existing topic into several subtopics using k-means clustering on the document embeddings of the topic.
+
+        Note that no new topwords are computed in this step, and the topwords of the old topic are just split among the new ones. Additionally, only the cosine-similarity method for topwords extraction is used.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            n_clusters (int, optional): Number of clusters to split the topic into (default is 2).
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+
+        old_topic = self.topic_lis[topic_idx]
+        embeddings = old_topic.document_embeddings_ld  # embeddings to split into clusters
+
+        kmeans_res = sklearn.cluster.KMeans(n_clusters = n_clusters, random_state = self.random_state, n_init = "auto").fit(embeddings)
+        cluster_labels = kmeans_res.labels_
+        new_topics = self.split_topic_new_assignments(topic_idx, cluster_labels, inplace)
+
+        return new_topics
+
+    def split_topic_hdbscan(self, topic_idx: int, min_cluster_size: int = 100, inplace: bool = False) -> list[Topic]:
+        """
+        Splits an existing topic into several subtopics using HDBSCAN clustering on the document embeddings of the topic.
+
+        This method does not require specifying the number of clusters to split. Note that no new topwords are computed in this step, and the topwords of the old topic are just split among the new ones. Additionally, only the cosine-similarity method for topwords extraction is used.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            min_cluster_size (int, optional): Minimum cluster size to split the topic into (default is 100).
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+
+        old_topic = self.topic_lis[topic_idx]
+        embeddings = old_topic.document_embeddings_ld
+
+        clusterer = hdbscan.HDBSCAN(min_cluster_size = min_cluster_size, prediction_data = True)
+        clusterer.fit(embeddings)
+        cluster_labels = clusterer.labels_
+        new_topics = self.split_topic_new_assignments(topic_idx, cluster_labels, inplace)
+
+        new_topics = self.reindex_topic_lis(new_topics)
+
+        if inplace:
+            self.topic_lis = new_topics
+
+        return new_topics
+
+    def split_topic_keywords(self, topic_idx: int, keywords: str, inplace: bool = False) -> list[Topic]:
+        """
+        Splits the topic into subtopics according to the provided keywords.
+
+        This is achieved by computing the cosine similarity between the keywords and the documents in the topic. Note that no new topwords are computed in this step, and the topwords of the old topic are just split among the new ones. Additionally, only the cosine-similarity method for topwords extraction is used.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            keywords (str): Keywords to split the topic into. Needs to be a list of at least two keywords.
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+        assert len(keywords) > 1, "Need at least two keywords to split the topic! Otherwise use the split_topic_single_keyword function!"
+        keyword_embeddings = []
+        for keyword in keywords:
+            keyword_embeddings.append(self.client.embeddings.create(input = [keyword], model = self.openai_embedding_model)["data"][0]["embedding"])
+        keyword_embeddings = np.array(keyword_embeddings)
+
+        old_topic = self.topic_lis[topic_idx]
+        document_embeddings = old_topic.document_embeddings_hd
+
+        document_embeddings = document_embeddings / np.linalg.norm(document_embeddings, axis = 1)[:, np.newaxis]
+        keyword_embeddings = keyword_embeddings / np.linalg.norm(keyword_embeddings, axis = 1)[:, np.newaxis]
+        similarities = document_embeddings @ keyword_embeddings.T
+        new_topic_assignments = np.argmax(similarities, axis = 1)
+
+        # if the topic cannot be split, i.e. all documents are assigned the same label, raise an error
+        if len(np.unique(new_topic_assignments)) == 1:
+            raise ValueError(f"The topic cannot be split into the subtopics {keywords}. All documents are assigned the same label!")
+
+        new_topics = self.split_topic_new_assignments(topic_idx, new_topic_assignments, inplace = inplace)
+
+        new_topics = self.reindex_topic_lis(new_topics)
+
+        if inplace:
+            self.topic_lis = new_topics
+
+        return new_topics
+
+    def split_topic_single_keyword(self, topic_idx: int, keyword: str, inplace: bool = False) -> list[Topic]:
+        """
+        Splits the topic with a single keyword.
+
+        This method splits the topic such that all documents closer to the original topic name stay in the old topic, while all documents closer to the keyword are moved to the new topic. Note that no new topwords are computed in this step, and the topwords of the old topic are just split among the new ones. Additionally, only the cosine-similarity method for topwords extraction is used.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            keyword (str): Keyword to split the topic into.
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+        keywords = [self.topic_lis[topic_idx].topic_name, keyword]
+
+        res = self.split_topic_keywords(topic_idx, keywords, inplace)
+
+        return res
+
+    def combine_topics(self, topic_idx_lis: list[int], inplace: bool = False) -> list[Topic]:
+        """
+        Combines several topics into one topic.
+
+        This method combines the specified topics into a single topic. Note that no new topwords are computed in this step, and the topwords of the old topics are just combined. Additionally, only the cosine-similarity method for topwords extraction is used.
+
+        Args:
+            topic_idx_list (list[int]): List of topic indices to combine.
+            inplace (bool, optional): If True, the topics are combined in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            list of Topic: A list of new topics resulting from the combination.
+        """
+
+        new_topic_docs = []
+        new_topic_words = []
+        new_topic_document_embeddings_hd = []
+
+        for topic_idx in topic_idx_lis:
+            topic = self.topic_lis[topic_idx]
+            new_topic_docs += topic.documents
+            new_topic_words += topic.words
+            new_topic_document_embeddings_hd.append(topic.document_embeddings_hd)
+
+        new_topic_document_embeddings_hd = np.concatenate(new_topic_document_embeddings_hd, axis = 0)
+
+        new_topic = extract_and_describe_topic_cos_sim(
+            documents_topic = new_topic_docs,
+            document_embeddings_topic = new_topic_document_embeddings_hd,
+            words_topic = new_topic_words,
+            vocab_embeddings = self.vocab_embeddings,
+            umap_mapper = self.topic_lis[0].umap_mapper,
+            enhancer=self.enhancer,
+            n_topwords = 2000
+        )
+
+        new_topic.topic_idx = len(self.topic_lis) + 1
+        new_topic_lis = self.topic_lis.copy()
+
+        for topic_idx in sorted(topic_idx_lis, reverse = True):
+            new_topic_lis.pop(topic_idx)
+        new_topic_lis.append(new_topic)
+        new_topic_lis = self.reindex_topic_lis(new_topic_lis)
+
+
+        if inplace:
+            self.topic_lis = new_topic_lis
+            self.reindex_topics()
+
+        return new_topic_lis
+
+    def add_new_topic_keyword(self, keyword: str, inplace: bool = False, rename_new_topic: bool = False) -> list[Topic]:
+        """
+        Create a new topic based on a keyword and recompute topic topwords.
+
+        This method removes all documents belonging to other topics from them and adds them to the new topic. It computes new topwords using both the tf-idf and the cosine-similarity method.
+
+        Args:
+            keyword (str): Keyword to create the new topic from.
+            inplace (bool, optional): If True, the topic is updated in place. Otherwise, a new list of topics is created and returned (default is False).
+            rename_new_topic (bool, optional): If True, the new topic is renamed to the keyword (default is False).
+
+        Returns:
+            list of Topic: A list of new topics, including the newly created topic and the modified old ones.
+        """
+
+        umap_mapper = self.topic_lis[0].umap_mapper
+
+        keyword_embedding_hd = self.client.embeddings.create(input = [keyword], model = self.openai_embedding_model)["data"][0]["embedding"]
+        keyword_embedding_hd = np.array(keyword_embedding_hd).reshape(1, -1)
+        keyword_embedding_ld = umap_mapper.transform(keyword_embedding_hd)[0]
+
+        old_centroids_ld = []
+        for topic in self.topic_lis:
+            old_centroids_ld.append(topic.centroid_ld)
+        old_centroids_ld = np.array(old_centroids_ld)
+
+        # assign documents to new centroid (keyword_embedding_ld) iff they are closer to the new centroid than to their old centroid
+
+        new_doc_topic_assignments = []
+        doc_lis = []
+
+        new_topic_idx = len(self.topic_lis)
+        for i, topic in enumerate(self.topic_lis):
+            doc_lis += topic.documents
+            document_embeddings = topic.document_embeddings_ld
+            cos_sim_old_centroid = document_embeddings @ old_centroids_ld[i] / (np.linalg.norm(document_embeddings, axis = 1) * np.linalg.norm(old_centroids_ld[i]))
+            cos_sim_new_centroid = document_embeddings @ keyword_embedding_ld / (np.linalg.norm(document_embeddings, axis = 1) * np.linalg.norm(keyword_embedding_ld))
+            new_centroid_is_closer = cos_sim_new_centroid > cos_sim_old_centroid
+
+            new_document_assignments = np.where(new_centroid_is_closer, new_topic_idx, i)
+            new_doc_topic_assignments.append(new_document_assignments)
+
+        new_doc_topic_assignments = np.concatenate(new_doc_topic_assignments, axis = 0)
+
+        assert len(doc_lis) == len(new_doc_topic_assignments), "Number of documents must be equal to the number of document assignments!"
+
+        new_embeddings_hd = []
+        new_embeddings_ld = []
+
+        for topic in self.topic_lis:
+            new_embeddings_hd.append(topic.document_embeddings_hd)
+            new_embeddings_ld.append(topic.document_embeddings_ld)
+
+        new_embeddings_hd = np.concatenate(new_embeddings_hd, axis = 0)
+        new_embeddings_ld = np.concatenate(new_embeddings_ld, axis = 0)
+
+        new_topics = extract_describe_topics_labels_vocab(
+            corpus = doc_lis,
+            document_embeddings_hd = new_embeddings_hd,
+            document_embeddings_ld = new_embeddings_ld,
+            labels = new_doc_topic_assignments,
+            vocab = self.vocab,
+            umap_mapper = umap_mapper,
+            vocab_embeddings = self.vocab_embeddings, 
+            enhancer = self.enhancer
+        )
+
+        if rename_new_topic:
+            new_topics[-1].topic_name = keyword
+
+        new_topics = self.reindex_topic_lis(new_topics)
+
+        if inplace:
+            self.topic_lis = new_topics
+
+        return new_topics
+
+    def delete_topic(self, topic_idx: int, inplace: bool = False) -> list[Topic]:
+        """
+        Deletes a topic with the given index from the list of topics and recomputes topwords and representations of the remaining topics.
+
+        This method assigns the documents of the deleted topic to the remaining topics.
+
+        Args:
+            topic_idx (int): Index of the topic to delete.
+            inplace (bool, optional): If True, the topic is deleted in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            list of Topic: A list of new topics resulting from the deletion.
+        """
+
+
+        topic_lis_new = deepcopy(self.topic_lis)
+        topic_lis_new.pop(topic_idx)
+
+        old_centroids_ld = []
+        for topic in topic_lis_new:
+            old_centroids_ld.append(topic.centroid_ld)
+
+        old_centroids_ld = np.array(old_centroids_ld)
+
+        document_embeddings_ld = []
+
+        for topic in self.topic_lis:
+            document_embeddings_ld.append(topic.document_embeddings_ld)
+
+        document_embeddings_ld = np.concatenate(document_embeddings_ld, axis = 0) # has shape (n_documents, n_topics)
+
+        centroid_similarities = document_embeddings_ld @ old_centroids_ld.T / (np.linalg.norm(document_embeddings_ld, axis = 1)[:, np.newaxis] * np.linalg.norm(old_centroids_ld, axis = 1))
+        new_topic_assignments = np.argmax(centroid_similarities, axis = 1)
+
+        new_embeddings_hd = []
+        new_embeddings_ld = []
+
+        for topic in self.topic_lis:
+            new_embeddings_hd.append(topic.document_embeddings_hd)
+            new_embeddings_ld.append(topic.document_embeddings_ld)
+
+        new_embeddings_hd = np.concatenate(new_embeddings_hd, axis = 0)
+        new_embeddings_ld = np.concatenate(new_embeddings_ld, axis = 0)
+
+        doc_lis = []
+        for topic in self.topic_lis:
+            doc_lis += topic.documents
+
+
+
+        new_topics = extract_describe_topics_labels_vocab(
+            corpus = doc_lis,
+            document_embeddings_hd = new_embeddings_hd,
+            document_embeddings_ld = new_embeddings_ld,
+            labels = new_topic_assignments,
+            vocab = self.vocab,
+            umap_mapper = self.topic_lis[0].umap_mapper,
+            vocab_embeddings = self.vocab_embeddings,
+            enhancer = self.enhancer
+        )
+
+        new_topics = self.reindex_topic_lis(new_topics)
+
+        if inplace:
+            self.topic_lis = new_topics
+
+        return new_topics
+
+    def get_topic_information(self, topic_idx_lis: list[int], max_number_topwords: int = 500) -> dict:
+        """
+        Get detailed information on topics by their indices.
+
+        This function returns a dictionary where the keys are the topic indices, and the values are strings describing the topics. The description includes a maximum of max_number_topwords topwords.
+
+        Args:
+            topic_idx_list (list[int]): List of topic indices to compare.
+            max_number_topwords (int, optional): Maximum number of topwords to include in the description of the topics (default is 500).
+
+        Returns:
+            dict: A dictionary with topic indices as keys and their descriptions as values.
+        """
+
+        max_number_tokens = self.max_context_length_promting - len(tiktoken.encoding_for_model(self.openai_prompting_model).encode(self.basic_model_instruction + " " + self.corpus_instruction)) - 100
+
+        topic_info = {} # dictionary with the topic indices as keys and the topic descriptions as values
+
+        for topic_idx in topic_idx_lis:
+            topic = self.topic_lis[topic_idx]
+            topic_info[topic_idx] = topic.topic_description
+
+            topic_str = f"""
+            Topic index: {topic_idx}
+            Topic name: {topic.topic_name}
+            Topic description: {topic.topic_description}
+            Topic topwords: {topic.top_words["cosine_similarity"][:max_number_topwords]}"""
+
+            topic_info[topic_idx] = topic_str
+
+        # prune all topic descriptions to the maximum number of tokens by taking away the last word until the description fits
+
+        max_number_tokens_per_topic = max_number_tokens // len(topic_idx_lis)
+        tiktoken_encodings = {idx: tiktoken.encoding_for_model(self.openai_prompting_model).encode(topic_info[idx]) for idx in topic_idx_lis}
+        pruned_encodings = {idx: tiktoken_encodings[idx][:max_number_tokens_per_topic] for idx in topic_idx_lis}
+
+        topic_info = {idx: tiktoken.encoding_for_model(self.openai_prompting_model).decode(pruned_encodings[idx]) for idx in topic_idx_lis}
+
+        return topic_info
+
+    def _knn_search_openai(self, topic_index: int, query: str, k: int = 20) -> tuple[str, (list[str], list[int])]:
+        """
+        A version of the knn_search function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            topic_index (int): Index of the topic to search in.
+            query (str): Query string.
+            k (int, optional): Number of neighbors to return (default is 20).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            tuple: A tuple containing two lists -
+                - A list of top k documents (as strings).
+                - A list of indices corresponding to the top k documents in the topic.
+        """
+
+        topk_docs, topk_doc_indices = self.knn_search(topic_index, query, k)
+        json_obj = json.dumps({
+            "top-k documents": topk_docs,
+            "indices of top-k documents": list(topk_doc_indices)
+        })
+        return json_obj, (topk_docs, topk_doc_indices)
+
+    def _identify_topic_idx_openai(self, query: str, n_tries: int = 3) -> tuple[str, int]:
+        """
+        A version of the identify_topic_idx function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            query (str): Query string.
+            n_tries (int, optional): Number of tries to get a valid response from the LLM (default is 3).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            int: The topic index.
+        """
+
+        topic_index = self.identify_topic_idx(query, n_tries)
+        json_obj = json.dumps({
+            "topic index": topic_index
+        })
+        return json_obj, topic_index
+
+    def _split_topic_hdbscan_openai(self, topic_idx: int, min_cluster_size: int = 10, inplace: bool = False) -> tuple[str, list[Topic]]:
+        """
+        A version of the split_topic_hdbscan function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            min_cluster_size (int, optional): Minimum cluster size to split the topic into (default is 10).
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+        new_topics = self.split_topic_hdbscan(topic_idx, min_cluster_size, inplace)
+        json_obj = json.dumps({
+            "new topics": [topic.to_dict() for topic in new_topics][-len(new_topics):]
+        })
+        return json_obj, new_topics
+
+    def _split_topics_kmeans_openai(self, topic_idx: list[int], n_clusters: int = 2, inplace: bool = False) -> tuple[str, list[Topic]]:
+        """
+        A version of the split_topic_kmeans function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            topic_idx (list[int]): List of indices of the topics to split.
+            n_clusters (int, optional): Number of clusters to split each topic into (default is 2).
+            inplace (bool, optional): If True, the topics are split in place. Otherwise, new lists of topics are created and returned (default is False).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+        new_topics = self.split_topic_kmeans(topic_idx, n_clusters, inplace)
+        json_obj = json.dumps({
+            "new topics": [topic.to_dict() for topic in new_topics][-n_clusters:]
+        })
+        return json_obj, new_topics
+
+    def _split_topic_keywords_openai(self, topic_idx: int, keywords: str, inplace: bool = False) -> tuple[str, list[Topic]]:
+        """
+        A version of the split_topic_keywords function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            keywords (str): Keywords to split the topic into. Needs to be a list of at least two keywords.
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+        new_topics = self.split_topic_keywords(topic_idx, keywords, inplace)
+        json_obj = json.dumps({
+            "new topics": [topic.to_dict() for topic in new_topics][-len(keywords):]
+        })
+        return json_obj, new_topics
+
+    def _split_topic_single_keyword_openai(self, topic_idx: int, keyword: str, inplace: bool = False) -> tuple[str, list[Topic]]:
+        """
+        A version of the split_topic_single_keyword function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            topic_idx (int): Index of the topic to split.
+            keyword (str): Keyword to split the topic into.
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            list of Topic: A list of new topics resulting from the split.
+        """
+
+        new_topics = self.split_topic_single_keyword(topic_idx, keyword, inplace)
+        json_obj = json.dumps({
+            "new topics": [topic.to_dict() for topic in new_topics][-2:]
+        })
+        return json_obj, new_topics
+
+    def _combine_topics_openai(self, topic_idx_lis: list[int], inplace: bool = False) -> tuple[str, list[Topic]]:
+        """
+        A version of the combine_topics function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            topic_idx_lis (list[int]): List of topic indices to combine.
+            inplace (bool, optional): If True, the topics are combined in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            list of Topic: A list of new topics resulting from the combination.
+        """
+
+        new_topics = self.combine_topics(topic_idx_lis, inplace)
+        json_obj = json.dumps({
+            "new topics": [topic.to_dict() for topic in new_topics][-1]
+        })
+        return json_obj, new_topics
+
+    def _add_new_topic_keyword_openai(self, keyword: str, inplace: bool = False, rename_new_topic: bool = False) -> tuple[str, list[Topic]]:
+        """
+        A version of the add_new_topic_keyword function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            keyword (str): Keyword to create the new topic from.
+            inplace (bool, optional): If True, the topic is split in place. Otherwise, a new list of topics is created and returned (default is False).
+            rename_new_topic (bool, optional): If True, the new topic is renamed to the keyword (default is False).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            list of Topic: A list of new topics resulting from the operation.
+        """
+
+        new_topics = self.add_new_topic_keyword(keyword, inplace, rename_new_topic)
+        json_obj = json.dumps({
+            "new topics": [topic.to_dict() for topic in new_topics][-1]
+        })
+        return json_obj, new_topics
+
+    def _delete_topic_openai(self, topic_idx: int, inplace: bool = False) -> tuple[str, list[Topic]]:
+        """
+        A version of the delete_topic function that returns a JSON file to be used with the OpenAI API.
+
+        Args:
+            topic_idx (int): Index of the topic to delete.
+            inplace (bool, optional): If True, the topic is deleted in place. Otherwise, a new list of topics is created and returned (default is False).
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            list of Topic: A list of topics after the deletion operation.
+        """
+
+        new_topics = self.delete_topic(topic_idx, inplace)
+        json_obj = json.dumps({
+            f"Topics after deleting the one with index {topic_idx}": [topic.to_dict() for topic in new_topics]
+        })
+        return json_obj, new_topics
+
+    def _get_topic_information_openai(self, topic_idx_lis: list[int]) -> tuple[str, dict]:
+        """
+        A version of the get_topic_information function that returns a JSON file suitable for use with the OpenAI API.
+
+        Args:
+            topic_idx_lis (list[int]): List of topic indices to compare.
+
+        Returns:
+            json: JSON object to be used with the OpenAI API.
+            dict: A dictionary containing detailed information about the specified topics.
+        """
+
+        topic_info = self.get_topic_information(topic_idx_lis)
+        json_obj = json.dumps({
+            "topic info": topic_info
+        })
+        return json_obj, topic_info
+
+    def _fix_dictionary_topwords(self):
+        """
+        Fix an issue with the topic representation where the topwords are nested within another dictionary in the actual dictionary defining them.
+        """
+
+        for topic in self.topic_lis:
+            if type(topic.top_words["cosine_similarity"]) == dict:
+                topic.top_words["cosine_similarity"] = topic.top_words["cosine_similarity"][0]
+
+    def general_prompt(self, prompt: str, n_tries: int = 2) -> tuple[list[str], object]:
+        """
+        Prompt the Language Model (LLM) with a general prompt and return the response. Allow the LLM to call any function defined in the class.
+
+        Use n_tries in case the LLM does not provide a valid response.
+
+        Args:
+            prompt (str): Prompt string.
+            n_tries (int, optional): Number of tries to get a valid response from the LLM (default is 2).
+
+        Returns:
+            list of str: Response messages from the LLM.
+            object: Response of the invoked function.
+        """
+
+        messages = [
+            {
+                "role": "system",
+                "content": self.basic_model_instruction + " " + self.corpus_instruction
+            },
+            {
+                "role": "user",
+                "content": prompt
+            }
+            ]
+
+        functions = [self.function_descriptions[key] for key in self.function_descriptions.keys()]
+        for _ in range(n_tries):
+            try: 
+                response_message = self.client.chat.completions.create(model = self.openai_prompting_model,
+                messages = messages,
+                functions = functions,
+                function_call = "auto").choices[0].message
+
+                # Step 2: check if GPT wanted to call a function
+                function_call = response_message.function_call
+                if function_call is not None:
+                    print("GPT wants to the call the function: ", function_call)
+                    # Step 3: call the function
+                    # Note: the JSON response may not always be valid; be sure to handle errors
+
+                    function_name = function_call.name
+                    function_to_call = self.functionNames2Functions[function_name]
+                    function_args = json.loads(function_call.arguments)
+                    function_response = function_to_call(**function_args)
+                    function_response_json = function_response[0]
+                    function_response_return_output = function_response[1]
+
+                    # Step 4: send the info on the function call and function response to GPT
+                    messages.append(response_message)  # extend conversation with assistant's reply
+
+                    messages.append(
+                        {
+                            "role": "function",
+                            "name": function_name,
+                            "content": function_response_json,
+                        }
+                    )  # extend conversation with function response
+
+                    second_response = self.client.chat.completions.create(model=self.openai_prompting_model,
+                    messages=messages)  # get a new response from GPT where it can see the function response
+            except (TypeError, ValueError, openai.APIError, openai.APIConnectionError) as error:
+                print("Error occured: ", error)
+                print("Trying again...")
+
+        return [response_message, second_response], function_response_return_output

+import numpy as np
+import umap
+import sys
+import os
+import inspect
+from tqdm import tqdm
+import umap
+import json
+
+# make sure the import works even if the package has not been installed and just the files are used
+
+from topicgpt.Clustering import Clustering_and_DimRed
+from topicgpt.ExtractTopWords import ExtractTopWords
+from topicgpt.TopwordEnhancement import TopwordEnhancement
+
+class Topic:
+    """
+    class to represent a topic and all its attributes
+    """
+
+    def __init__(self, 
+             topic_idx: str, 
+             documents: list[str], 
+             words: dict[str, int],
+             centroid_hd: np.ndarray = None, 
+             centroid_ld: np.ndarray = None,
+             document_embeddings_hd: np.ndarray = None,
+             document_embeddings_ld: np.ndarray = None,
+             document_embedding_similarity: np.ndarray = None,
+             umap_mapper: umap.UMAP = None,
+             top_words: dict[str, list[str]] = None,
+             top_word_scores: dict[str, list[float]] = None
+             ) -> None:
+        """
+        Represents a topic and all its attributes.
+
+        Args:
+            topic_idx (str): Index or name of the topic.
+            documents (list[str]): List of documents in the topic.
+            words (dict[str, int]): Dictionary of words and their counts in the topic.
+            centroid_hd (np.ndarray, optional): Centroid of the topic in high-dimensional space.
+            centroid_ld (np.ndarray, optional): Centroid of the topic in low-dimensional space.
+            document_embeddings_hd (np.ndarray, optional): Embeddings of documents in high-dimensional space that belong to this topic.
+            document_embeddings_ld (np.ndarray, optional): Embeddings of documents in low-dimensional space that belong to this topic.
+            document_embedding_similarity (np.ndarray, optional): Similarity array of document embeddings to the centroid in low-dimensional space.
+            umap_mapper (umap.UMAP, optional): UMAP mapper object to map from high-dimensional space to low-dimensional space.
+            top_words (dict[str, list[str]], optional): Dictionary of top words in the topic according to different metrics.
+            top_word_scores (dict[str, list[float]], optional): Dictionary of how representative the top words are according to different metrics.
+        """
+
+        # do some checks on the input
+
+        assert len(documents) == len(document_embeddings_hd) == len(document_embeddings_ld) == len(document_embedding_similarity), "documents, document_embeddings_hd, document_embeddings_ld and document_embedding_similarity must have the same length"
+        assert len(documents) > 0, "documents must not be empty"
+        assert len(words) > 0, "words must not be empty"
+
+
+        self.topic_idx = topic_idx
+        self.documents = documents
+        self.words = words
+        self.centroid_hd = centroid_hd
+        self.centroid_ld = centroid_ld
+        self.document_embeddings_hd = document_embeddings_hd
+        self.document_embeddings_ld = document_embeddings_ld
+        self.document_embedding_similarity = document_embedding_similarity
+        self.umap_mapper = umap_mapper
+        self.top_words = top_words
+        self.top_word_scores = top_word_scores
+
+        self.topic_name = None # initialize the name of the topic as none
+
+    def __str__(self) -> str:
+
+        if self.topic_idx and self.topic_name is None:
+            repr = f"Topic {hash(self)}\n"
+        if self.topic_name is None:
+            repr = f"Topic: {self.topic_idx}\n"
+        else: 
+            repr = f"Topic {self.topic_idx}: {self.topic_name}\n"
+        
+        return repr
+    
+    def __repr__(self) -> str:
+        return self.__str__()
+    
+    def to_json(self) -> str:
+        """
+        return a json representation of the topic
+        """
+        repr_dict = {
+            "topic_idx": self.topic_idx,
+            "topic_name": self.topic_name,
+            "topic_description": self.topic_description
+        }
+
+        json_object = json.dumps(repr_dict, indent = 4)
+        return json_object
+    
+    def to_dict(self) -> dict:
+        """
+        return a dict representation of the topic
+        """
+        repr_dict = {
+            "topic_idx": int(self.topic_idx),
+            "topic_name": self.topic_name,
+            "topic_description": self.topic_description
+        }
+        return repr_dict
+    
+    def set_topic_name(self, name:str):
+        """
+        add a name to the topic
+        params:
+            name: name of the topic
+        """
+        self.topic_name = name
+
+    def set_topic_description(self, text: str):
+        """
+        add a text description to the topic
+        params:
+            text: text description of the topic
+        """
+        self.topic_description = text
+
+def topic_to_json(topic: Topic) -> str:
+    """
+    Return a JSON representation of the topic.
+
+    Args:
+        topic (Topic): The topic object to convert to JSON.
+
+    Returns:
+        str: A JSON string representing the topic.
+    """
+    repr_dict = {
+        "topic_idx": topic.topic_idx,
+        "topic_name": topic.topic_name,
+        "topic_description": topic.topic_description
+    }
+
+    json_object = json.dumps(repr_dict, indent = 4)
+    return json_object
+
+def topic_lis_to_json(topics: list[Topic]) -> str:
+    """
+    Return a JSON representation of a list of topics.
+
+    Args:
+        topics (list[Topic]): The list of topic objects to convert to JSON.
+
+    Returns:
+        str: A JSON string representing the list of topics.
+    """
+    repr_dict = {}
+    for topic in topics:
+        repr_dict[topic.topic_idx] = {
+            "topic_name": topic.topic_name,
+            "topic_description": topic.topic_description
+        }
+
+    json_object = json.dumps(repr_dict, indent = 4)
+    return json_object
+
+@staticmethod
+def extract_topics(corpus: list[str], document_embeddings: np.ndarray, clusterer: Clustering_and_DimRed, vocab_embeddings: np.ndarray, n_topwords: int = 2000, topword_extraction_methods: list[str] = ["tfidf", "cosine_similarity"], compute_vocab_hyperparams: dict = {}) -> list[Topic]:
+    """
+    Extracts topics from the given corpus using the provided clusterer object on the document embeddings.
+
+    Args:
+        corpus (list[str]): List of documents.
+        document_embeddings (np.ndarray): Embeddings of the documents.
+        clusterer (Clustering_and_DimRed): Clustering and dimensionality reduction object to cluster the documents.
+        vocab_embeddings (np.ndarray): Embeddings of the vocabulary.
+        n_topwords (int, optional): Number of top-words to extract from the topics (default is 2000).
+        topword_extraction_methods (list[str], optional): List of methods to extract top-words from the topics. 
+            Can contain "tfidf" and "cosine_similarity" (default is ["tfidf", "cosine_similarity"]).
+        compute_vocab_hyperparams (dict, optional): Hyperparameters for the top-word extraction methods.
+
+    Returns:
+        list[Topic]: List of Topic objects representing the extracted topics.
+    """
+
+    for elem in topword_extraction_methods:
+        if elem not in ["tfidf", "cosine_similarity"]:
+            raise ValueError("topword_extraction_methods can only contain 'tfidf' and 'cosine_similarity'")
+    if topword_extraction_methods == []:
+        raise ValueError("topword_extraction_methods cannot be empty")
+
+    dim_red_embeddings, labels, umap_mapper = clusterer.cluster_and_reduce(document_embeddings)  # get dimensionality reduced embeddings, their labels and the umap mapper object
+
+    unique_labels = np.unique(labels)  # In case the cluster labels are not consecutive numbers, we need to map them to consecutive 
+    label_mapping = {label: i for i, label in enumerate(unique_labels[unique_labels != -1])}
+    label_mapping[-1] = -1
+    labels = np.array([label_mapping[label] for label in labels])
+
+    extractor = ExtractTopWords()
+    centroid_dict = extractor.extract_centroids(document_embeddings, labels)  # get the centroids of the clusters
+    centroid_arr = np.array(list(centroid_dict.values()))
+    if centroid_arr.ndim == 1:
+        centroid_arr = centroid_arr.reshape(-1, 1)
+    dim_red_centroids = umap_mapper.transform(np.array(list(centroid_dict.values())))  # map the centroids to low dimensional space
+    
+    dim_red_centroid_dict = {label: centroid for label, centroid in zip(centroid_dict.keys(), dim_red_centroids)}
+
+    vocab = extractor.compute_corpus_vocab(corpus, **compute_vocab_hyperparams)  # compute the vocabulary of the corpus
+
+    word_topic_mat = extractor.compute_word_topic_mat(corpus, vocab, labels, consider_outliers = False)  # compute the word-topic matrix of the corpus
+    if "tfidf" in topword_extraction_methods:
+        tfidf_topwords, tfidf_dict = extractor.extract_topwords_tfidf(word_topic_mat = word_topic_mat, vocab = vocab, labels = labels, top_n_words = n_topwords)  # extract the top-words according to tfidf
+    if "cosine_similarity" in topword_extraction_methods:
+        cosine_topwords, cosine_dict = extractor.extract_topwords_centroid_similarity(word_topic_mat = word_topic_mat, vocab = vocab, vocab_embedding_dict = vocab_embeddings, centroid_dict= dim_red_centroid_dict, umap_mapper = umap_mapper, top_n_words = n_topwords, reduce_vocab_embeddings = True, reduce_centroid_embeddings = False, consider_outliers = False)
+                                                                                     
+    topics = []
+    for i, label in enumerate(np.unique(labels)):
+        if label < -0.5: # dont include outliers
+            continue
+        topic_idx = f"{label}"
+        documents = [doc for j, doc in enumerate(corpus) if labels[j] == label]
+        embeddings_hd = document_embeddings[labels == label]
+        embeddings_ld = dim_red_embeddings[labels == label]
+        centroid_hd = centroid_dict[label]
+        centroid_ld = dim_red_centroids[label]
+        
+        centroid_similarity = np.dot(embeddings_ld, centroid_ld)/(np.linalg.norm(embeddings_ld, axis = 1)*np.linalg.norm(centroid_ld))
+        similarity_sorting = np.argsort(centroid_similarity)[::-1]
+        documents = [documents[i] for i in similarity_sorting]
+        embeddings_hd = embeddings_hd[similarity_sorting]
+        embeddings_ld = embeddings_ld[similarity_sorting]
+
+        if type(cosine_topwords[label]) == dict:
+            cosine_topwords[label] = cosine_topwords[label][0]
+
+        top_words = {
+            "tfidf": tfidf_topwords[label] if "tfidf" in topword_extraction_methods else None,
+            "cosine_similarity": cosine_topwords[label] if "cosine_similarity" in topword_extraction_methods else None
+        }
+        top_word_scores = {
+            "tfidf": tfidf_dict[label] if "tfidf" in topword_extraction_methods else None,
+            "cosine_similarity": cosine_dict[label] if "cosine_similarity" in topword_extraction_methods else None
+        }
+
+        topic = Topic(topic_idx = topic_idx,
+                        documents = documents,
+                        words = vocab,
+                        centroid_hd = centroid_hd,
+                        centroid_ld = centroid_ld,
+                        document_embeddings_hd = embeddings_hd,
+                        document_embeddings_ld = embeddings_ld,
+                        document_embedding_similarity = centroid_similarity,
+                        umap_mapper = umap_mapper,
+                        top_words = top_words, 
+                        top_word_scores = top_word_scores
+                        )
+                      
+        topics.append(topic)
+    
+    return topics
+
+@staticmethod
+def extract_topics_no_new_vocab_computation(corpus: list[str], vocab: list[str], document_embeddings: np.ndarray, clusterer: Clustering_and_DimRed, vocab_embeddings: np.ndarray, n_topwords: int = 2000, topword_extraction_methods: list[str] = ["tfidf", "cosine_similarity"], consider_outliers: bool = False) -> list[Topic]:
+    """
+    Extracts topics from the given corpus using the provided clusterer object on the document embeddings. 
+    This version does not compute the vocabulary of the corpus and instead uses the provided vocabulary.
+
+    Args:
+        corpus (list[str]): List of documents.
+        vocab (list[str]): Vocabulary of the corpus.
+        document_embeddings (np.ndarray): Embeddings of the documents.
+        clusterer (Clustering_and_DimRed): Clustering and dimensionality reduction object to cluster the documents.
+        vocab_embeddings (np.ndarray): Embeddings of the vocabulary.
+        n_topwords (int, optional): Number of top-words to extract from the topics (default is 2000).
+        topword_extraction_methods (list[str], optional): List of methods to extract top-words from the topics. 
+            Can contain "tfidf" and "cosine_similarity" (default is ["tfidf", "cosine_similarity"]).
+        consider_outliers (bool, optional): Whether to consider outliers during topic extraction (default is False).
+
+    Returns:
+        list[Topic]: List of Topic objects representing the extracted topics.
+    """
+
+
+    for elem in topword_extraction_methods:
+        if elem not in ["tfidf", "cosine_similarity"]:
+            raise ValueError("topword_extraction_methods can only contain 'tfidf' and 'cosine_similarity'")
+    if topword_extraction_methods == []:
+        raise ValueError("topword_extraction_methods cannot be empty")
+
+    dim_red_embeddings, labels, umap_mapper = clusterer.cluster_and_reduce(document_embeddings)  # get dimensionality reduced embeddings, their labels and the umap mapper object
+
+    unique_labels = np.unique(labels)  # In case the cluster labels are not consecutive numbers, we need to map them to consecutive 
+    label_mapping = {label: i for i, label in enumerate(unique_labels[unique_labels != -1])}
+    label_mapping[-1] = -1
+    labels = np.array([label_mapping[label] for label in labels])
+
+    extractor = ExtractTopWords()
+    centroid_dict = extractor.extract_centroids(document_embeddings, labels)  # get the centroids of the clusters
+
+    centroid_arr = np.array(list(centroid_dict.values()))
+    if centroid_arr.ndim == 1:
+        centroid_arr = centroid_arr.reshape(-1, 1)
+    dim_red_centroids = umap_mapper.transform(np.array(list(centroid_dict.values())))  # map the centroids to low dimensional space
+
+    dim_red_centroid_dict = {label: centroid for label, centroid in zip(centroid_dict.keys(), dim_red_centroids)}
+
+    word_topic_mat = extractor.compute_word_topic_mat(corpus, vocab, labels, consider_outliers = consider_outliers)  # compute the word-topic matrix of the corpus
+    if "tfidf" in topword_extraction_methods:
+        tfidf_topwords, tfidf_dict = extractor.extract_topwords_tfidf(word_topic_mat = word_topic_mat, vocab = vocab, labels = labels, top_n_words = n_topwords)  # extract the top-words according to tfidf
+    if "cosine_similarity" in topword_extraction_methods:
+        cosine_topwords, cosine_dict = extractor.extract_topwords_centroid_similarity(word_topic_mat = word_topic_mat, vocab = vocab, vocab_embedding_dict = vocab_embeddings, centroid_dict= dim_red_centroid_dict, umap_mapper = umap_mapper, top_n_words = n_topwords, reduce_vocab_embeddings = True, reduce_centroid_embeddings = False, consider_outliers = True)
+                                                                                           
+    topics = []
+    for i, label in enumerate(np.unique(labels)):
+        if label < -0.5: # dont include outliers
+            continue
+        topic_idx = f"{label}"
+        documents = [doc for j, doc in enumerate(corpus) if labels[j] == label]
+        embeddings_hd = document_embeddings[labels == label]
+        embeddings_ld = dim_red_embeddings[labels == label]
+        centroid_hd = centroid_dict[label]
+        centroid_ld = dim_red_centroids[label]
+        
+        centroid_similarity = np.dot(embeddings_ld, centroid_ld)/(np.linalg.norm(embeddings_ld, axis = 1)*np.linalg.norm(centroid_ld))
+        similarity_sorting = np.argsort(centroid_similarity)[::-1]
+        documents = [documents[i] for i in similarity_sorting]
+        embeddings_hd = embeddings_hd[similarity_sorting]
+        embeddings_ld = embeddings_ld[similarity_sorting]
+
+        try:
+            if type(cosine_topwords[label]) == dict:
+                cosine_topwords[label] = cosine_topwords[label][0]
+        except:
+            pass
+
+        top_words = {
+            "tfidf": tfidf_topwords[label] if "tfidf" in topword_extraction_methods else None,
+            "cosine_similarity": cosine_topwords[label] if "cosine_similarity" in topword_extraction_methods else None
+        }
+        top_word_scores = {
+            "tfidf": tfidf_dict[label] if "tfidf" in topword_extraction_methods else None,
+            "cosine_similarity": cosine_dict[label] if "cosine_similarity" in topword_extraction_methods else None
+        }
+
+        topic = Topic(topic_idx = topic_idx,
+                        documents = documents,
+                        words = vocab,
+                        centroid_hd = centroid_hd,
+                        centroid_ld = centroid_ld,
+                        document_embeddings_hd = embeddings_hd,
+                        document_embeddings_ld = embeddings_ld,
+                        document_embedding_similarity = centroid_similarity,
+                        umap_mapper = umap_mapper,
+                        top_words = top_words, 
+                        top_word_scores = top_word_scores
+                        )
+                      
+        topics.append(topic)
+    
+    return topics
+
+@staticmethod
+def extract_and_describe_topics(corpus: list[str], document_embeddings: np.ndarray, clusterer: Clustering_and_DimRed, vocab_embeddings: np.ndarray, enhancer: TopwordEnhancement, n_topwords: int = 2000, n_topwords_description: int = 500, topword_extraction_methods: list[str] = ["tfidf", "cosine_similarity"], compute_vocab_hyperparams: dict = {}, topword_description_method: str = "cosine_similarity") -> list[Topic]:
+    """
+    Extracts topics from the given corpus using the provided clusterer object on the document embeddings and describes/names them using the given enhancer object.
+
+    Args:
+        corpus (list[str]): List of documents.
+        document_embeddings (np.ndarray): Embeddings of the documents.
+        clusterer (Clustering_and_DimRed): Clustering and dimensionality reduction object to cluster the documents.
+        vocab_embeddings (np.ndarray): Embeddings of the vocabulary.
+        enhancer (TopwordEnhancement): Enhancer object for enhancing top-words and generating descriptions/names for topics.
+        n_topwords (int, optional): Number of top-words to extract from the topics (default is 2000).
+        n_topwords_description (int, optional): Number of top-words to use from the extracted topics for description and naming (default is 500).
+        topword_extraction_methods (list[str], optional): List of methods to extract top-words from the topics. 
+            Can contain "tfidf" and "cosine_similarity" (default is ["tfidf", "cosine_similarity"]).
+        compute_vocab_hyperparams (dict, optional): Hyperparameters for the top-word extraction methods.
+        topword_description_method (str, optional): Method to use for top-word extraction for description/naming. 
+            Can be "tfidf" or "cosine_similarity" (default is "cosine_similarity").
+
+    Returns:
+        list[Topic]: List of Topic objects representing the extracted and described topics.
+    """
+
+    print("Extracting topics...")
+    topics = extract_topics(corpus, document_embeddings, clusterer, vocab_embeddings, n_topwords, topword_extraction_methods, compute_vocab_hyperparams)
+    print("Describing topics...")
+    topics = describe_and_name_topics(topics, enhancer, topword_description_method, n_topwords_description)
+    return topics
+
+@staticmethod
+def extract_topics_labels_vocab(corpus: list[str], document_embeddings_hd: np.ndarray, document_embeddings_ld: np.ndarray, labels: np.ndarray, umap_mapper: umap.UMAP, vocab_embeddings: np.ndarray, vocab: list[str] = None, n_topwords: int = 2000, topword_extraction_methods: list[str] = ["tfidf", "cosine_similarity"]) -> list[Topic]:
+    """
+    Extracts topics from the given corpus using the provided labels that indicate the topics (no -1 for outliers). Vocabulary is already computed.
+
+    Args:
+        corpus (list[str]): List of documents.
+        document_embeddings_hd (np.ndarray): Embeddings of the documents in high-dimensional space.
+        document_embeddings_ld (np.ndarray): Embeddings of the documents in low-dimensional space.
+        labels (np.ndarray): Labels indicating the topics.
+        umap_mapper (umap.UMAP): UMAP mapper object to map from high-dimensional space to low-dimensional space.
+        vocab_embeddings (np.ndarray): Embeddings of the vocabulary.
+        vocab (list[str], optional): Vocabulary of the corpus (default is None).
+        n_topwords (int, optional): Number of top-words to extract from the topics (default is 2000).
+        topword_extraction_methods (list[str], optional): List of methods to extract top-words from the topics. 
+            Can contain "tfidf" and "cosine_similarity" (default is ["tfidf", "cosine_similarity"]).
+
+    Returns:
+        list[Topic]: List of Topic objects representing the extracted topics.
+    """
+
+    for elem in topword_extraction_methods:
+        if elem not in ["tfidf", "cosine_similarity"]:
+            raise ValueError("topword_extraction_methods can only contain 'tfidf' and 'cosine_similarity'")
+    if topword_extraction_methods == []:
+        raise ValueError("topword_extraction_methods cannot be empty")
+    
+    if vocab is None:
+        extractor = ExtractTopWords()
+        vocab = extractor.compute_corpus_vocab(corpus)  # compute the vocabulary of the corpus
+    
+    extractor = ExtractTopWords()
+    centroid_dict = extractor.extract_centroids(document_embeddings_hd, labels)  # get the centroids of the clusters
+    
+    centroid_arr = np.array(list(centroid_dict.values()))
+    if centroid_arr.ndim == 1:
+        centroid_arr = centroid_arr.reshape(-1, 1)
+    dim_red_centroids = umap_mapper.transform(np.array(list(centroid_dict.values())))  # map the centroids to low dimensional space
+
+    word_topic_mat = extractor.compute_word_topic_mat(corpus, vocab, labels, consider_outliers = False)  # compute the word-topic matrix of the corpus
+
+    dim_red_centroid_dict = {label: centroid for label, centroid in zip(centroid_dict.keys(), dim_red_centroids)}
+
+    if "tfidf" in topword_extraction_methods:
+        tfidf_topwords, tfidf_dict = extractor.extract_topwords_tfidf(word_topic_mat = word_topic_mat, vocab = vocab, labels = labels, top_n_words = n_topwords)  # extract the top-words according to tfidf
+    if "cosine_similarity" in topword_extraction_methods:
+        cosine_topwords, cosine_dict = extractor.extract_topwords_centroid_similarity(word_topic_mat = word_topic_mat, vocab = vocab, vocab_embedding_dict = vocab_embeddings, centroid_dict= dim_red_centroid_dict, umap_mapper = umap_mapper, top_n_words = n_topwords, reduce_vocab_embeddings = True, reduce_centroid_embeddings = False, consider_outliers = False)
+                                                                                                 
+    topics = []
+    for i, label in enumerate(np.unique(labels)):
+        if label < -0.5: # dont include outliers
+            continue
+        topic_idx = f"{label}"
+        documents = [doc for j, doc in enumerate(corpus) if labels[j] == label]
+        embeddings_hd = document_embeddings_hd[labels == label]
+        embeddings_ld = document_embeddings_ld[labels == label]
+        centroid_hd = centroid_dict[label]
+        centroid_ld = dim_red_centroids[label]
+        
+        centroid_similarity = np.dot(embeddings_ld, centroid_ld)/(np.linalg.norm(embeddings_ld, axis = 1)*np.linalg.norm(centroid_ld))
+        similarity_sorting = np.argsort(centroid_similarity)[::-1]
+        documents = [documents[i] for i in similarity_sorting]
+        embeddings_hd = embeddings_hd[similarity_sorting]
+        embeddings_ld = embeddings_ld[similarity_sorting]
+
+        if type(cosine_topwords[label]) == dict:
+            cosine_topwords[label] = cosine_topwords[label][0]
+        top_words = {
+            "tfidf": tfidf_topwords[label] if "tfidf" in topword_extraction_methods else None,
+            "cosine_similarity": cosine_topwords[label] if "cosine_similarity" in topword_extraction_methods else None
+        }
+        top_word_scores = {
+            "tfidf": tfidf_dict[label] if "tfidf" in topword_extraction_methods else None,
+            "cosine_similarity": cosine_dict[label] if "cosine_similarity" in topword_extraction_methods else None
+        }
+
+        topic = Topic(topic_idx = topic_idx,
+                        documents = documents,
+                        words = vocab,
+                        centroid_hd = centroid_hd,
+                        centroid_ld = centroid_ld,
+                        document_embeddings_hd = embeddings_hd,
+                        document_embeddings_ld = embeddings_ld,
+                        document_embedding_similarity = centroid_similarity,
+                        umap_mapper = umap_mapper,
+                        top_words = top_words, 
+                        top_word_scores = top_word_scores
+                        )
+                      
+        topics.append(topic)
+    
+    return topics
+
+@staticmethod
+def extract_describe_topics_labels_vocab(
+    corpus: list[str],
+    document_embeddings_hd: np.ndarray,
+    document_embeddings_ld: np.ndarray,
+    labels: np.ndarray,
+    umap_mapper: umap.UMAP,
+    vocab_embeddings: np.ndarray,
+    enhancer: TopwordEnhancement,
+    vocab: list[str] = None,
+    n_topwords: int = 2000,
+    n_topwords_description: int = 500,
+    topword_extraction_methods: list[str] = ["tfidf", "cosine_similarity"],
+    topword_description_method: str = "cosine_similarity"
+) -> list[Topic]:
+    """
+    Extracts topics from the given corpus using the provided labels that indicate the topics (no -1 for outliers). Vocabulary is already computed.
+    Describe and name the topics with the given enhancer object.
+
+    Args:
+        corpus (list[str]): List of documents.
+        document_embeddings_hd (np.ndarray): Embeddings of the documents in high-dimensional space.
+        document_embeddings_ld (np.ndarray): Embeddings of the documents in low-dimensional space.
+        labels (np.ndarray): Labels indicating the topics.
+        umap_mapper (umap.UMAP): UMAP mapper object to map from high-dimensional space to low-dimensional space.
+        vocab_embeddings (np.ndarray): Embeddings of the vocabulary.
+        enhancer (TopwordEnhancement): Enhancer object to enhance the top-words and generate the description.
+        vocab (list[str], optional): Vocabulary of the corpus (default is None).
+        n_topwords (int, optional): Number of top-words to extract from the topics (default is 2000).
+        n_topwords_description (int, optional): Number of top-words to use from the extracted topics for the description and the name (default is 500).
+        topword_extraction_methods (list[str], optional): List of methods to extract top-words from the topics. 
+            Can contain "tfidf" and "cosine_similarity" (default is ["tfidf", "cosine_similarity"]).
+        topword_description_method (str, optional): Method to use for top-word extraction. Can be "tfidf" or "cosine_similarity" (default is "cosine_similarity").
+
+    Returns:
+        list[Topic]: List of Topic objects representing the extracted topics.
+    """
+
+    topics = extract_topics_labels_vocab(corpus, document_embeddings_hd, document_embeddings_ld, labels, umap_mapper, vocab_embeddings, vocab, n_topwords, topword_extraction_methods)
+    topics = describe_and_name_topics(topics, enhancer, topword_description_method, n_topwords_description)
+    return topics
+
+@staticmethod
+def extract_topic_cos_sim(
+    documents_topic: list[str],
+    document_embeddings_topic: np.ndarray,
+    words_topic: list[str],
+    vocab_embeddings: dict,
+    umap_mapper: umap.UMAP,
+    n_topwords: int = 2000
+) -> Topic:
+    """
+    Create a Topic object from the given documents and embeddings by computing the centroid and the top-words.
+    Only uses cosine-similarity for top-word extraction.
+
+    Args:
+        documents_topic (list[str]): List of documents in the topic.
+        document_embeddings_topic (np.ndarray): High-dimensional embeddings of the documents in the topic.
+        words_topic (list[str]): List of words in the topic.
+        vocab_embeddings (dict): Embeddings of the vocabulary.
+        umap_mapper (umap.UMAP): UMAP mapper object to map from high-dimensional space to low-dimensional space.
+        n_topwords (int, optional): Number of top-words to extract from the topics (default is 2000).
+
+    Returns:
+        Topic: Topic object representing the extracted topic.
+    """
+
+    topword_extraction_methods = ["cosine_similarity"]
+    extractor = ExtractTopWords()
+    centroid_hd = extractor.extract_centroid(document_embeddings_topic)
+    centroid_ld = umap_mapper.transform(centroid_hd.reshape(1, -1))[0]
+
+    labels = np.zeros(len(documents_topic), dtype = int) #everything has label 0   
+
+    word_topic_mat = extractor.compute_word_topic_mat(documents_topic, words_topic, labels, consider_outliers = False)  # compute the word-topic matrix of the corpus
+    if "cosine_similarity" in topword_extraction_methods:
+        cosine_topwords, cosine_dict = extractor.extract_topwords_centroid_similarity(word_topic_mat = word_topic_mat, vocab = words_topic, vocab_embedding_dict = vocab_embeddings, centroid_dict= {0: centroid_ld}, umap_mapper = umap_mapper, top_n_words = n_topwords, reduce_vocab_embeddings = True, reduce_centroid_embeddings = False, consider_outliers = False)
+
+    
+
+    top_words = {
+        "cosine_similarity": cosine_topwords if "cosine_similarity" in topword_extraction_methods else None
+    }
+    top_word_scores = {
+        "cosine_similarity": cosine_dict if "cosine_similarity" in topword_extraction_methods else None
+    }
+
+    document_embeddings_hd = document_embeddings_topic
+    document_embeddings_ld = umap_mapper.transform(document_embeddings_hd)
+    document_embedding_similarity = np.dot(document_embeddings_ld, centroid_ld)/(np.linalg.norm(document_embeddings_ld, axis = 1)*np.linalg.norm(centroid_ld)) # is this correct???
+
+    topic = Topic(topic_idx = None,
+                documents = documents_topic,	
+                words = words_topic,
+                centroid_hd = centroid_hd,
+                centroid_ld = centroid_ld,
+                document_embeddings_hd = document_embeddings_hd,
+                document_embeddings_ld = document_embeddings_ld,
+                document_embedding_similarity = document_embedding_similarity,
+                umap_mapper = umap_mapper,
+                top_words = top_words,
+                top_word_scores = top_word_scores
+                )
+    
+    return topic
+
+@staticmethod
+def extract_and_describe_topic_cos_sim(
+    documents_topic: list[str],
+    document_embeddings_topic: np.ndarray,
+    words_topic: list[str],
+    vocab_embeddings: dict,
+    umap_mapper: umap.UMAP,
+    enhancer: TopwordEnhancement,
+    n_topwords: int = 2000,
+    n_topwords_description=500
+) -> Topic:
+    """
+    Create a Topic object from the given documents and embeddings by computing the centroid and the top-words.
+    Only use cosine-similarity for top-word extraction.
+    Describe and name the topic with the given enhancer object.
+
+    Args:
+        documents_topic (list[str]): List of documents in the topic.
+        document_embeddings_topic (np.ndarray): High-dimensional embeddings of the documents in the topic.
+        words_topic (list[str]): List of words in the topic.
+        vocab_embeddings (dict): Embeddings of the vocabulary.
+        umap_mapper (umap.UMAP): UMAP mapper object to map from high-dimensional space to low-dimensional space.
+        enhancer (TopwordEnhancement): Enhancer object to enhance the top-words and generate the description.
+        n_topwords (int, optional): Number of top-words to extract from the topics (default is 2000).
+        n_topwords_description (int, optional): Number of top-words to use from the extracted topics for the description and the name (default is 500).
+
+    Returns:
+        Topic: Topic object representing the extracted and described topic.
+    """
+    topic = extract_topic_cos_sim(documents_topic, document_embeddings_topic, words_topic, vocab_embeddings, umap_mapper, n_topwords)
+    topic = describe_and_name_topics([topic], enhancer, "cosine_similarity", n_topwords_description)[0]
+    return topic
+
+    topic = extract_topic_cos_sim(documents_topic, document_embeddings_topic, words_topic, vocab_embeddings, umap_mapper, n_topwords)
+    topic = describe_and_name_topics([topic], enhancer, "cosine_similarity", n_topwords_description)[0]
+    return topic
+
+@staticmethod
+def describe_and_name_topics(
+    topics: list[Topic],
+    enhancer: TopwordEnhancement,
+    topword_method="tfidf",
+    n_words=500
+) -> list[Topic]:
+    """
+    Describe and name the topics using the OpenAI API with the given enhancer object.
+
+    Args:
+        topics (list[Topic]): List of Topic objects.
+        enhancer (TopwordEnhancement): Enhancer object to enhance the top-words and generate the description.
+        topword_method (str, optional): Method to use for top-word extraction. Can be "tfidf" or "cosine_similarity" (default is "tfidf").
+        n_words (int, optional): Number of topwords to extract for the description and the name (default is 500).
+
+    Returns:
+        list[Topic]: List of Topic objects with the description and name added.
+    """
+
+    if topword_method not in ["tfidf", "cosine_similarity"]:
+        raise ValueError("topword_method can only be 'tfidf' or 'cosine_similarity'")
+   
+    for topic in tqdm(topics):
+        tws = topic.top_words[topword_method]
+        try: 
+            topic_name = enhancer.generate_topic_name_str(tws, n_words = n_words)
+            topic_description = enhancer.describe_topic_topwords_str(tws, n_words = n_words)
+        except Exception as e:
+            print(f"Error in topic {topic.topic_idx}: {e}")
+            print("Trying again...")
+            topic_name = enhancer.generate_topic_name_str(tws, n_words = n_words)
+            topic_description = enhancer.describe_topic_topwords_str(tws, n_words = n_words)
+
+
+        topic.set_topic_name(topic_name)
+        topic.set_topic_description(topic_description)
+        
+    return topics
+