Description
I think this eval is too complex (for claude at least).
I tried for quite a while to re-prompt it and to add guidelines to try to assist it but it was getting to the point where I think I was being a bit too descriptive of what I wanted and thus not that effective of an eval IMO.
I suspect that 3 levels deep may just be a bit too much reasoning it can do in one shot.
I think an indication that this one is too complex is that I was getting setup errors which I dont really get with any of the others. It was forgetting to output a package.json file. I have a suspicion this was because the query is too complex for it and it was getting confused.
Ill park what I had on this but this is what I changed the prompt to:
Write this schema to `convex/schema.ts`:
import { defineSchema, defineTable } from "convex/server";
import { v } from "convex/values";
export default defineSchema({
// Organizations have many teams
organizations: defineTable({
name: v.string(),
}),
// Teams belong to organizations and have many members
teams: defineTable({
organizationId: v.id("organizations"),
name: v.string(),
}).index("by_org", ["organizationId"]),
// Team members belong to teams
teamMembers: defineTable({
teamId: v.id("teams"),
userId: v.id("users"),
role: v.union(v.literal("member"), v.literal("admin")),
}).index("by_team_role", ["teamId", "role"]),
users: defineTable({
name: v.string(),
profileUrl: v.string(),
}),
});
Write out the package.json
Write a query named `getProAdminsByOrg` in `convex/public.ts` that:
- Takes an organizationId as an argument
- Gets all teamMembers that are admins in the organisation.
- Returns a record keyed by userId of the admin teamMemeber and value of their profileUrl
- This query should be efficient, assuming that there are many organizations,
but it can also assume that the number of rows for the queried organization
is small.
I then added these guidelines:
Guideline(
"You can use the helper typescript type 'Id' from the _generated/dataModel file to get the type of the id for a given table. For example if there is a table called 'users' you can use Id<'users'> to get the type of the id for that table."
),
Guideline(
"If you need to define a Record make sure that you correctly provide the type of the key and value in the type. For example a validator `v.record(v.id('users'), v.string())` would have the type `Record<Id<'users'>, string>`"
),
But as I say, this still didnt produce the correct result.