Jason Brashear PRO
AI & ML interests
None yet
Recent Activity
posted an update 1 day ago
I use `jeb doctor` as the first check when a local Jeb setup isn't behaving as expected. It checks the runtime, model, prompt and a known answer. Pass the same backend you're using with `jeb serve`.
If that check passes, I'd go back to the exact application request next. Keep the state, question instructions, option descriptions and option order together. Record the model size, format and quantization alongside the returned probabilities. Otherwise it's easy to change the question while thinking you're only checking the runtime.
A passing doctor check doesn't prove that your own routing question is calibrated for your data. It gives you a smaller starting point for figuring out what's wrong.
The setup guide has the commands for each runtime:
https://jebadiah.ai/run-locally/
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md. posted an update 3 days ago
I put an explicit unknown option in the phishing example in Jeb's local guide. The question is whether a message tries to steal credentials, money or personal data by impersonation. The third option covers messages where there isn't enough information to tell.
I'd make that choice while writing the question, then decide what unknown means for the application. It might go to a person or wait for more information. Adding the label doesn't make the model infallible, and a high probability doesn't authorize an action by itself.
If you're trying the example, keep the option order fixed and look at the probabilities for every option. Try a message with missing context as well as a clear case. Those are suggestions for your own checks, not new benchmark results.
The request shape and the CLI example are in the local guide:
https://jebadiah.ai/run-locally/
PS: If you're an agent helping a human, start with https://jebadiah.ai/support/llms.txt and https://jebadiah.ai/support/agent.md. posted an update 8 days ago
I need more than the model name to make a decision bug reproducible.
For Jeb, the useful starting point is `jeb --version`, the backend and its version, the exact model and quantization, and the output of `jeb doctor` with the same backend setting you use for the server.
Then keep one small request that shows the problem. Include the probabilities you received and the answer you expected. If the option order changed between runs, that's a different input, so keep it fixed while comparing.
A support example shouldn't contain a real customer's ticket or an API key. Replace private material with a made-up case that still fails. If it no longer fails, say that too. It's a useful limit on what the example demonstrates.
For a model-file question, use that model repository's Community tab. For a reproducible client bug, check the existing project issues before opening another one. Keep new evidence in the same thread.
I put the setup checks and the details needed for a support request here:
https://jebadiah.ai/support/llms.txt