main_notebook.ipynb

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 1,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pickle\n",
    "from tqdm.notebook import tqdm\n",
    "from conversation import create_coder, create_reviewer, create_refiner, start_conversation"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Trabalho de RL - Qual o Melhor Prompt para Iterar sobre a Geração de Código?\n",
    "Nosso trabalho de RL é sobre descobrir quais os melhores prompts para iterar sobre a geração de \n",
    "código de LLMs.\n",
    "\n",
    "## Como funciona\n",
    "Criamos uma conversa com 3 participantes: Coder, Reviewer e Refiner.\n",
    "Cada participante é responsável por enviar um prompt para o LLM, e inputar os resultados do LLM no\n",
    "ambiente (aqui chamado de 'conversa').\n",
    "\n",
    "### Coder\n",
    "O Coder é responsável por escrever o código.  \n",
    "Ele envia o prompt inicial, descrevendo o problema a ser resolvido. Definimos que todos os nossos\n",
    "problemas serão de limpeza de uma base de dados csv.  \n",
    "O Coder só participa da conversa 1 vez (no início) e, por isso, não o definimos como um agente RL.\n",
    "Ao invés disso, ele é programado para iterar por todos os prompts n vezes, e avaliamos os resultados\n",
    "das conversas com cada prompt inicial posteriormente.\n",
    "\n",
    "### Reviewer\n",
    "O Reviewer é responsável por avaliar o código gerado pelo LLM.  \n",
    "Ele envia um prompt solicitando a avaliação do código gerado pelo LLM. Ele pode essa avaliação\n",
    "sempre após a geração de um código que não tem nota superior à nota terminal.  \n",
    "O Reviewer é um agente RL, e seu objetivo é maximizar a nota do código gerado pelo LLM.\n",
    "\n",
    "### Refiner\n",
    "O Refiner é responsável por refinar o código gerado pelo LLM.\n",
    "Ele envia um prompt solicitando a melhoria do código gerado pelo LLM. Ele pode essa avaliação\n",
    "sempre após uma revisão do Reviewer.  \n",
    "O Refiner é um agente RL, e seu objetivo é maximizar a nota do código gerado pelo LLM.\n",
    "\n",
    "### Prompt\n",
    "Para cada participante, geramos prompts que iam de 1 a $n$ nas **escalas** das seguintes **propriedades**:\n",
    "- Clareza;\n",
    "- Comprimento;\n",
    "- Especificidade, e\n",
    "- Complexidade.\n",
    "\n",
    "Isso totalizou até 20 prompts diferentes para cada participante. Para diminuir o espaço de ações,\n",
    "optamos por usar uma estratégia mais simples:\n",
    "\n",
    "- Para cada prompt (**comprimento da escala** x **número de propriedades**) do **Coder**;\n",
    "    - Para cada **propriedade** do **Reviewer**;\n",
    "        - Para cada **propriedade** do **Refiner**;\n",
    "            - Geramos $m$ conversas onde:\n",
    "                1. O Coder envia o prompt e adiciona o código inicial;\n",
    "                2. O código é avaliado (se a nota não for terminal, prossegue);\n",
    "                3. O Reviewer escolhe um dos $n$ prompts da **propriedade** e adiciona a revisão;\n",
    "                4. O Refiner escolhe um dos $n$ prompts da **propriedade** e adiciona a melhoria.\n",
    "                5. Se o comprimento da conversa não for terminal, volta ao passo 2.\n",
    "\n",
    "### Avaliação do Código\n",
    "O código é avaliado por um LLM usando a bibliteca `instructor`. Pedimos que o código receba uma nota\n",
    "de 0 a 100 para a sua corretude e legibilidade, bem como uma curta explicação do porquê da nota \n",
    "(esse comentário é adicionado posteriormente à conversa).  \n",
    "Se a nota média for superior a 95, a conversa é terminada pois consideramos que o código é bom o\n",
    "suficiente."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [],
   "source": [
    "prompts_coder = [\n",
    "    {   \"index\": 0,\n",
    "        \"prompt\": \"I have a dirty dataset. Develop a code to clean it.\"\n",
    "    },\n",
    "    {   \"index\": 1,\n",
    "        \"prompt\": \"I will provide a dataset with some issues that need to be addressed. You should import the CSV file located at 'csv_data/imdb_sample_10.csv' and develop a well structured code to perform the necessary corrections.\",\n",
    "    },\n",
    "    {   \"index\": 2,\n",
    "        \"prompt\": \"I have a dataset that requires cleaning. The file path is 'csv_data/imdb_sample_10.csv'. Please ensure the following tasks are carried out: 1. Import the CSV file; 2. check for and address the following issues: missing data, duplicate record, formatting inconsistencies, typographical errors, data encoding issues, etc; 3. Save the cleaned dataset in a file called 'processed_imdb_sample_10.csv' and return it. Ensure that the code is well structured, efficient, and utilizes popular Python libraries such as pandas and numpy. The code needs to work perfectly when I run it with my database in the specified path.\",\n",
    "    },\n",
    "    {   \"index\": 3,\n",
    "        \"prompt\": \"You will be provided with a dataset contained in a CSV file located at 'csv_data/imdb_sample_10.csv'. Your task is to import this CSV file and develop a well-structured Python script to address the issues within the dataset. To achieve this, follow these steps: 1. Import the necessary libraries (e.g., pandas, numpy); 2. Load the CSV file into a DataFrame; 3. Identify and document the issues present in the dataset (e.g., missing values, incorrect data types, duplicate entries); 4. Develop code to correct each identified issue systematically; 5. Ensure that your script is modular, separating the correction steps into different functions where appropriate; 6. Validate the corrections made to ensure data integrity; 7. Please complete the task by following the steps outlined above and provide comments within your code to explain each correction made.\"\n",
    "    }\n",
    "]\n",
    "\n",
    "reviewer_questions = {\n",
    "    \"Question 1\": [\n",
    "        \"You will assume the role of a code reviewer who will receive a script designed to clean a dataset and will need to evaluate the quality of this code. Focus on answering the following questions: Is the code functional? Are there any structural or syntax errors that would prevent its execution? Is the code well-documented? Is the code applicable to the dataset it is intended to clean? Be thorough in your review, imagining that you are responsible for a project and are guiding a programmer on your team.\",\n",
    "        \"You will assume the role of a code reviewer who will receive a script designed to clean a dataset and will need to evaluate the quality of this code. Be concise in your evaluation, keeping it brief and condensed as much as possible.\"\n",
    "    ],\n",
    "    \"Question 2\": [\n",
    "        \"You will assume the role of a code reviewer who will receive a script designed to clean a dataset and will need to evaluate the quality of this code. Analyze the code and verify if it is consistent, functional, applicable to the dataset, and well-structured. Condense the possible areas for improvement and provide your feedback at the end.\",\n",
    "        \"You will assume the role of a code reviewer who will receive a script designed to clean a dataset and will need to evaluate the quality of this code. Evaluate the following criteria and suggest possible improvements: 1. Is the code functional? 2. Is the code concise? 3. Is the code easy to interpret? 4. Is the code well-documented? 5. Is the path to the CSV file correct? (The correct path should be 'csv_data/imdb_sample_10.csv') 6. Is the returned code continuous, meaning it is not separated into multiple cells? 7. Is the code saving the modified CSV? 8. Do the columns referenced in the CSV actually exist?\"\n",
    "    ]\n",
    "}\n",
    "\n",
    "refiner_questions = {\n",
    "    \"Question 1\": [\n",
    "        \"You will assume the role of a code refiner, receiving a script and a list of recommended corrections to be made in it. Ensure that all suggested changes are applied in a coherent way, preserving the functionality of the original code.\",\n",
    "        \"You will assume the role of a code refiner, receiving a script and a list of recommended corrections to be made. The core of the script is the cleaning of a dirty dataset, and implementing the modifications is crucial to ensure that all steps, from exporting the CSV to returning the well modified CSV, are carried out correctly. Therefore, make sure that all suggested changes are applied coherently, preserving the functionality of the original code.\"\n",
    "    ],\n",
    "    \"Question 2\": [\n",
    "        \"You will assume the role of a code refiner, receiving a script and a list of recommended corrections to be made in it. Ensure that all suggested changes are applied in a coherent way, preserving the functionality of the original code.\",\n",
    "        \"\"\n",
    "    ]\n",
    "}           "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from rl.code_evaluator import CodeEvaluator\n",
    "# coder = create_coder(prompts_coder)\n",
    "# coder_selected_prompt = prompts_coder[0]\n",
    "# reviewer = create_reviewer(reviewer_questions[\"Question 1\"])\n",
    "# refiner = create_refiner(refiner_questions[\"Question 1\"])\n",
    "# evaluator = CodeEvaluator(environment=None, prompt=\"Evaluate the code quality\", name=\"Code Evaluator\")\n",
    "\n",
    "# MAX_TURNS = 3\n",
    "# TOT_CONVERSATIONS = 1\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [],
   "source": [
    "# from tqdm.notebook import trange, tqdm\n",
    "# from coder import Coder\n",
    "# from rl.llm_agent import LLMAgent\n",
    "# from rl.environment import Environment\n",
    "# from rl.code_evaluator import CodeEvaluator, CSV_PATH\n",
    "# from rl.policies import EpsilonGreedyPolicy\n",
    "# from rl.utils import compute_delta_grade, is_terminate_grade\n",
    "\n",
    "# with open(CSV_PATH, \"r\") as f:\n",
    "#     csv_data = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "# global evaluator, csv_data\n",
    "# environment = Environment()\n",
    "\n",
    "# # Set the environment for all agents\n",
    "# for agent in [coder, reviewer, refiner, evaluator]:\n",
    "#     agent.environment = environment\n",
    "\n",
    "# # Adding csv in the first coder prompt\n",
    "# coder_prompt_dict = coder_selected_prompt.copy()\n",
    "# coder_prompt_dict[\"prompt\"] += f\"\\n\\nFile content:\\n\\n{csv_data}\"\n",
    "# coder.add_message(coder_prompt_dict)\n",
    "\n",
    "# # Start the conversation\n",
    "# last_grade = None\n",
    "# for turn in tqdm(range(MAX_TURNS), desc=\"Conv. turns\", position=0, leave=False):\n",
    "#     print(f\"Turn {turn + 1}\")\n",
    "#     # Evaluates the code and status of the CSV\n",
    "#     grade = evaluator.evaluate_code()\n",
    "    \n",
    "#     # Check if the score is enough to close\n",
    "#     if is_terminate_grade(grade):\n",
    "#         break\n",
    "#     # If it is the first turn, reward the coder\n",
    "#     elif last_grade is None:\n",
    "#         coder.first_reward(grade)\n",
    "#     # If it is not the first turn, reward refiner and reviewer    \n",
    "#     else:\n",
    "#         delta_grade = compute_delta_grade(last_grade, grade)\n",
    "#         refiner.reward(delta_grade)\n",
    "#         reviewer.reward(delta_grade)\n",
    "    \n",
    "#     # Add reviewer and refiner messages to the conversation\n",
    "#     reviewer.add_message()\n",
    "#     refiner.add_message()\n",
    "\n",
    "#     last_grade = grade\n",
    "\n",
    "# # Reward the Coder with the latest review\n",
    "# coder.final_reward(last_grade)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 1/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "21d9c0e84cb144a3949348f1fda05ead",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 2/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6139b2346b4445f09e5c00106858e4d1",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 3/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1fafb7d5c82a48beb942f879cdf4f7d9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversation Skipped: 10 validation errors for CodeEvaluation\n",
      "is_code_functional\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_consise\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_easily_readable\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_documented\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_csv_path_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_all_grouped\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_saving_csv\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "are_columns_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "overall_grade\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "explanation\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "Combination 4/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "99613655046d4237b3301751c0d3281b",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 5/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "0d9c9496c10c48aca9ba3744909ad440",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 6/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "187e78f9f288428aa14ce8ee798ff787",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 7/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "103124fdbdb540da9daa85a7e73b212e",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 8/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2dca281b180545b888eff6bdc22529f9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 9/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "357dbb65a35449b29c1b58d84719cd65",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversation Skipped: 10 validation errors for CodeEvaluation\n",
      "is_code_functional\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_consise\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_easily_readable\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_documented\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_csv_path_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_all_grouped\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_saving_csv\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "are_columns_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "overall_grade\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "explanation\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "Combination 10/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "2ba3af819e8b49258c4a79f9f88e0fea",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Combination 11/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "cd58462f75c44c5b92e72de0864845bd",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversation Skipped: 10 validation errors for CodeEvaluation\n",
      "is_code_functional\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_consise\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_easily_readable\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_documented\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_csv_path_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_all_grouped\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_saving_csv\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "are_columns_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "overall_grade\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "explanation\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "Combination 12/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "e929b2ad97ac42f99c9f8fee4eac845f",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversation Skipped: 10 validation errors for CodeEvaluation\n",
      "is_code_functional\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_consise\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_easily_readable\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_documented\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_csv_path_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_all_grouped\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_saving_csv\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "are_columns_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "overall_grade\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "explanation\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "Combination 13/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "7fb0166faf6145988fea76387585ad31",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversation Skipped: 10 validation errors for CodeEvaluation\n",
      "is_code_functional\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_consise\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_easily_readable\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_documented\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_csv_path_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_all_grouped\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_saving_csv\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "are_columns_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "overall_grade\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "explanation\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "Combination 14/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "6ee24bcf77a14ca287d370fb502b6260",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversation Skipped: 8 validation errors for CodeEvaluation\n",
      "is_code_functional\n",
      "  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/bool_type\n",
      "is_code_consise\n",
      "  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/bool_type\n",
      "is_code_easily_readable\n",
      "  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/bool_type\n",
      "is_code_documented\n",
      "  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/bool_type\n",
      "is_code_all_grouped\n",
      "  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/bool_type\n",
      "is_code_saving_csv\n",
      "  Input should be a valid boolean [type=bool_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/bool_type\n",
      "overall_grade\n",
      "  Input should be a valid integer [type=int_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/int_type\n",
      "explanation\n",
      "  Input should be a valid string [type=string_type, input_value=None, input_type=NoneType]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/string_type\n",
      "Combination 15/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1d0237f9b4b243abbd13472abbeb4c7d",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Conversation Skipped: 10 validation errors for CodeEvaluation\n",
      "is_code_functional\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_consise\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_easily_readable\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_documented\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_csv_path_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_all_grouped\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "is_code_saving_csv\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "are_columns_correct\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "overall_grade\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "explanation\n",
      "  Field required [type=missing, input_value={}, input_type=dict]\n",
      "    For further information visit https://errors.pydantic.dev/2.9/v/missing\n",
      "Combination 16/16\r"
     ]
    },
    {
     "data": {
      "application/vnd.jupyter.widget-view+json": {
       "model_id": "1abb39b2b2f44f74bf61470640f31ea9",
       "version_major": 2,
       "version_minor": 0
      },
      "text/plain": [
       "Conv. turns:   0%|          | 0/4 [00:00<?, ?it/s]"
      ]
     },
     "metadata": {},
     "output_type": "display_data"
    },
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'models/coder.txt'",
     "output_type": "error",
     "traceback": [
      "\u001b[1;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[1;31mFileNotFoundError\u001b[0m                         Traceback (most recent call last)",
      "Cell \u001b[1;32mIn[6], line 43\u001b[0m\n\u001b[0;32m     40\u001b[0m                 \u001b[38;5;28;01mexcept\u001b[39;00m \u001b[38;5;167;01mException\u001b[39;00m \u001b[38;5;28;01mas\u001b[39;00m e:\n\u001b[0;32m     41\u001b[0m                    \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mConversation Skipped: \u001b[39m\u001b[38;5;132;01m{\u001b[39;00me\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m---> 43\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m \u001b[38;5;28;43mopen\u001b[39;49m\u001b[43m(\u001b[49m\u001b[38;5;124;43mf\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mmodels/coder.txt\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[38;5;124;43mw\u001b[39;49m\u001b[38;5;124;43m\"\u001b[39;49m\u001b[43m)\u001b[49m \u001b[38;5;28;01mas\u001b[39;00m file:\n\u001b[0;32m     44\u001b[0m     file\u001b[38;5;241m.\u001b[39mwrite(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mstart rewards: \u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;241m+\u001b[39m\u001b[38;5;28mstr\u001b[39m(coder\u001b[38;5;241m.\u001b[39mstart_rewards))\n\u001b[0;32m     45\u001b[0m     file\u001b[38;5;241m.\u001b[39mwrite(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;130;01m\\n\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n",
      "File \u001b[1;32mc:\\Users\\lagua\\OneDrive\\Documentos\\git-folders\\RL-code-generation-with-multiple-agents\\.venv\\Lib\\site-packages\\IPython\\core\\interactiveshell.py:324\u001b[0m, in \u001b[0;36m_modified_open\u001b[1;34m(file, *args, **kwargs)\u001b[0m\n\u001b[0;32m    317\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m file \u001b[38;5;129;01min\u001b[39;00m {\u001b[38;5;241m0\u001b[39m, \u001b[38;5;241m1\u001b[39m, \u001b[38;5;241m2\u001b[39m}:\n\u001b[0;32m    318\u001b[0m     \u001b[38;5;28;01mraise\u001b[39;00m \u001b[38;5;167;01mValueError\u001b[39;00m(\n\u001b[0;32m    319\u001b[0m         \u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mIPython won\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124mt let you open fd=\u001b[39m\u001b[38;5;132;01m{\u001b[39;00mfile\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m by default \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    320\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mas it is likely to crash IPython. If you know what you are doing, \u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    321\u001b[0m         \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124myou can use builtins\u001b[39m\u001b[38;5;124m'\u001b[39m\u001b[38;5;124m open.\u001b[39m\u001b[38;5;124m\"\u001b[39m\n\u001b[0;32m    322\u001b[0m     )\n\u001b[1;32m--> 324\u001b[0m \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[43mio_open\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfile\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43margs\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[38;5;241;43m*\u001b[39;49m\u001b[43mkwargs\u001b[49m\u001b[43m)\u001b[49m\n",
      "\u001b[1;31mFileNotFoundError\u001b[0m: [Errno 2] No such file or directory: 'models/coder.txt'"
     ]
    }
   ],
   "source": [
    "from rl.code_evaluator import CodeEvaluator\n",
    "\n",
    "\n",
    "MAX_TURNS = 4\n",
    "TOT_CONVERSATIONS = 1\n",
    "coder = create_coder(prompts_coder)\n",
    "\n",
    "comb = 0\n",
    "total_combs = len(prompts_coder) * len(reviewer_questions) * len(refiner_questions)\n",
    "for i, coder_prompt_dict in enumerate(prompts_coder):\n",
    "    for j, rev_prompts in enumerate(reviewer_questions.values()):\n",
    "        for k, ref_prompts in enumerate(refiner_questions.values()):\n",
    "            reviewer = create_reviewer(rev_prompts)\n",
    "            refiner = create_refiner(ref_prompts)\n",
    "            evaluator = CodeEvaluator(environment=None, prompt=\"Evaluate the code quality\", name=\"Code Evaluator\")\n",
    "            \n",
    "            comb += 1\n",
    "            print(f\"Combination {comb}/{total_combs}\", end=\"\\r\")\n",
    "            for l in range(TOT_CONVERSATIONS):\n",
    "                try:\n",
    "                    environment, rev_hist, ref_hist, last_grade = start_conversation(\n",
    "                        coder, \n",
    "                        coder_prompt_dict, \n",
    "                        reviewer, \n",
    "                        refiner, \n",
    "                        MAX_TURNS\n",
    "                    )\n",
    "                    # Salva a conversa\n",
    "                    with open(f\"conversations/conv_{i}_{j}_{k}_{l}.md\", \"w\", encoding=\"utf-8\") as file:\n",
    "                        for message in environment.messages:\n",
    "                            file.write(f\"**{message['role']}**: {message['content']}\\n\")\n",
    "                    with open(f\"conversations/conversation_final_grades.txt\", \"a\") as file:\n",
    "                        file.write(f\"Conversation {i}_{j}_{k}_{l}: {last_grade}\\n\")\n",
    "                    \n",
    "                    # Salva os históricos\n",
    "                    with open(f\"history/reviewer_{i}_{j}_{k}_{l}.txt\", \"w\") as file:\n",
    "                        file.write(str(rev_hist))\n",
    "                    with open(f\"history/refiner_{i}_{j}_{k}_{l}.txt\", \"w\") as file:\n",
    "                        file.write(str(ref_hist))\n",
    "                except Exception as e:\n",
    "                   print(f\"Conversation Skipped: {e}\")\n",
    "\n",
    "with open(f\"results/coder.txt\", \"w\") as file:\n",
    "    file.write(\"start rewards: \"+str(coder.start_rewards))\n",
    "    file.write(\"\\n\")\n",
    "    file.write(\"end rewards: \"+str(coder.end_rewards))\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open(f\"results/coder.txt\", \"w\") as file:\n",
    "    file.write(\"start rewards: \"+str(coder.start_rewards))\n",
    "    file.write(\"\\n\")\n",
    "    file.write(\"end rewards: \"+str(coder.end_rewards))"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": ".venv",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.12.7"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 2
}