Building Agentic AI Applications with LLMs Assessment

Learning and Certification Courses and Workshops
1

Hello-

I am struggling to complete the assessment for this course. The assessor seems to be generating json errors. Example:

EXCEPTIONS: 
[!] [Case 0] Invalid json output: {
  'properties': {
    'feedback': {
      'description': 'Feedback on the submission and suggestions for improvement, noting strengths and weaknesses',
      'title': 'Feedback',
      'type': 'string'
    },
    'score': {
      'description': 'The score awarded for the submission (0-10)',
      'title': 'Score',
      'type': 'integer'
    }
  },
  'required': ['feedback', 'score'],
  'title': 'EvaluationTrace',
  'type': 'object'
}

FEEDBACK: 
The submission demonstrates a good understanding of the key differences between LangGraph and LangChain. However, the response could be improved by providing more concrete examples and specific details to support the claims made. Additionally, the use of citations could be more comprehensive, as only a few sources are mentioned.

SCORE: 7

Rationale: 
The submission provides a clear and concise overview of the core differences between LangGraph and LangChain. The response is well-structured, and the use of bullet points makes it easy to follow. However, the submission could benefit from more concrete examples and specific details to support the claims made. Additionally, the use of citations could be more comprehensive, as only a few sources are mentioned.

The submission demonstrates a good understanding of the key differences between LangGraph and LangChain, and the response is well-structured. However, the submission could be improved by providing more concrete examples and specific details to support the claims made.

And when no error is present the feedback is extremely limited:
MESSAGES:

  • [Case 3][Quality None/10] None
  • [Case 3][Penalty -6] Result “less than ok”
  • [Case 5][Quality None/10] None
  • [Case 5][Penalty -6] Result “less than ok”
  • [Case 7][Quality None/10] None
  • [Case 7][Penalty -6] Result “less than ok”

Not sure if these are also silent errors. Would appreciate some clarification on whether this is the expected result as I’m unable to get to the 80 score despite the submission looking ok from my standpoint. Thanks for the assistance.

2

Hey @j2angst. It sounds like there might be some issue with the autograder’s LLM endpoint failing (which would mean that the deployment in build.nvidia.com is working unexpectedly). The former issue is propagating back the exception to let you know that the model failed to give back the proper output.

You should actually be getting these messages back, since they explain why points are being lost. Did you manage to get this to work? I can dig into the weeds and see if I can make the grader model specifiable through your api.

3

Odd question, I believe the autograder logs trace to the Jaeger interface. Curious if this pointed to the issue.

4

Hey appreciate the reply! I have not got it working yet - still getting the same output today. I’m not familiar with the Jaeger interface I’m assuming I would access that from within the NVIDIA environment? Happy to take a look if you have instructions. Thanks for the help.

5

No worries! I can dig into it tomorrow on my end. If you have it pulled up, it would be on the “:16686/search” address, but I wouldn’t kickstart just to check as it’s more of a diagnostic/for-your-information interface. If it’s something hanging up behind the scenes, it would explain it but may not be enough to solve it.

6

This is the output from running open from url and passing in “:16686/search”. I couldn’t get curl to work from the terminal - and unfortunately the output did not change when I ran it at the start of the session and after making a submission so doubt it’s particularly helpful. No rush but do let me know if you find anything on your end or if I should run something else. Appreciate the help!

<!doctype html>JupyterLab {“allow_hidden_files”: false, “appName”: “JupyterLab”, “appNamespace”: “lab”, “appSettingsDir”: “/usr/local/share/jupyter/lab/settings”, “appUrl”: “/lab”, “appVersion”: “4.4.6”, “baseUrl”: “/lab/”, “buildAvailable”: true, “buildCheck”: true, “cacheFiles”: false, “copyAbsolutePath”: false, “delete_to_trash”: true, “devMode”: false, “disabledExtensions”: , “exposeAppInBrowser”: false, “extensionManager”: {“can_install”: true, “install_path”: “/usr/local”, “name”: “PyPI”}, “extraLabextensionsPath”: , “federated_extensions”: [{“entrypoints”: null, “extension”: “./extension”, “load”: “static/remoteEntry.5cbb9d2323598fbda535.js”, “name”: “jupyterlab_pygments”, “style”: “./style”}, {“entrypoints”: null, “extension”: “./extension”, “load”: “static/remoteEntry.84740c0bf9a6442a86fd.js”, “name”: “jupyterlab-jupytext”}, {“entrypoints”: null, “extension”: “./extension”, “load”: “static/remoteEntry.35b6c65bd99dab37b910.js”, “name”: “@jupyter-widgets/jupyterlab-manager”}, {“entrypoints”: null, “extension”: “./extension”, “load”: “static/remoteEntry.c3551e86769ef3973316.js”, “name”: “ '@'hadim/jupyter-archive”, “style”: “./style”}, {“entrypoints”: null, “extension”: “./extension”, “load”: “static/remoteEntry.09935b5f2738ef3c32f2.js”, “name”: “@jupyter-server/resource-usage”, “style”: “./style”}], “fullAppUrl”: “/lab/lab”, “fullLabextensionsUrl”: “/lab/lab/extensions”, “fullLicensesUrl”: “/lab/lab/api/licenses”, “fullListingsUrl”: “/lab/lab/api/listings”, “fullMathjaxUrl”: “``https://cdnjs.cloudflare.com/ajax/libs/mathjax/2.7.7/MathJax.js”``, “fullSettingsUrl”: “/lab/lab/api/settings”, “fullStaticUrl”: “/lab/static/lab”, “fullThemesUrl”: “/lab/lab/api/themes”, “fullTranslationsApiUrl”: “/lab/lab/api/translations”, “fullTreeUrl”: “/lab/lab/tree”, “fullWorkspacesApiUrl”: “/lab/lab/api/workspaces”, “ignorePlugins”: , “labextensionsPath”: [“/root/.local/share/jupyter/labextensions”, “/usr/local/share/jupyter/labextensions”, “/usr/share/jupyter/labextensions”], “labextensionsUrl”: “/lab/extensions”, “licensesUrl”: “/lab/api/licenses”, “listingsUrl”: “/lab/api/listings”, “mathjaxConfig”: “TeX-AMS_HTML-full,Safe”, “mode”: “multiple-document”, “news”: {“disabled”: false}, “notebookStartsKernel”: true, “notebookVersion”: “[2, 17, 0]”, “preferredPath”: “/”, “quitButton”: true, “rootUri”: “file:///dli/task”, “schemasDir”: “/usr/local/share/jupyter/lab/schemas”, “serverRoot”: “/dli/task”, “settingsUrl”: “/lab/api/settings”, “staticDir”: “/usr/local/share/jupyter/lab/static”, “store_id”: 4, “templatesDir”: “/usr/local/share/jupyter/lab/static”, “terminalsAvailable”: true, “themesDir”: “/usr/local/share/jupyter/lab/themes”, “themesUrl”: “/lab/api/themes”, “token”: “e913f361dc9ce9b520c2b594a3bbe955”, “translationsApiUrl”: “/lab/api/translations”, “treePath”: “:16686/search”, “treeUrl”: “/lab/tree”, “untracked_message_types”: [“comm_info_request”, “comm_info_reply”, “kernel_info_request”, “kernel_info_reply”, “shutdown_request”, “shutdown_reply”, “interrupt_request”, “interrupt_reply”, “debug_request”, “debug_reply”, “stream”, “display_data”, “update_display_data”, “execute_input”, “execute_result”, “error”, “status”, “clear_output”, “debug_event”, “input_request”, “input_reply”], “userSettingsDir”: “/root/.jupyter/lab/user-settings”, “virtualDocumentsUri”: “file:///dli/task/.virtual_documents”, “workspace”: “default”, “workspacesApiUrl”: “/lab/api/workspaces”, “workspacesDir”: “/root/.jupyter/lab/workspaces”, “wsUrl”: “”} /* Remove token from URL. */
(function () {
var location = window.location;
var search = location.search;

// If there is no query string, bail.
if (search.length <= 1) {
  return;
}

// Rebuild the query string without the `token`.
var query = '?' + search.slice(1).split('&')
  .filter(function (param) { return param.split('=')[0] !== 'token'; })
  .join('&');

// Rebuild the URL with the new query string.
var url = location.origin + location.pathname +
  (query !== '?' ? query : '') + location.hash;

if (url === location.href) {
  return;
}

window.history.replaceState({ }, '', url);

})();

7

Hey @j2angst. Sorry about the delay. With the holidays and other things, the thread kinda fell off on my end. Anyways, I did submit an update and I think it should mostly be working now. There has been some changes in the setup, and the new system should be much more resilient (and you should also be able to set the LLM if the default one doesn’t work).

8

I’m struggling with this assessment as well. Is this fix for everyone that access the assessment notebook. Everything appears to be running correctly but the assessment does not pick up traces questions and responses

9

Hey @vkudlay no worries at all just tested it out and I’m no longer getting the json errors and I am getting feedback for each question now. Really appreciate your assistance on this thank you!