On AI Knowledge Acquisition from Humans
Even though LLMs are trained on terabytes of data, by no means it contains all human knowledge, not to mention that a great amount of human knowledge is not easily available on text. The ones that are available on text, could also be contracting and creating undesirable effect in production. Thus, there’s always the need to tune the AI with specific knowledge so it can perform tasks in a way that’s desirable for the customer.
Recently, I have had some chance to work with Henry Truong, our knowledgeable and experienced advisor at Juji, on some AI agents that need constant update with new knowledge. Henry is passionate in using AI for training and education and has found Juji as a great platform for that.
From my discussion with Henry, I realize that a lot of work he has done is to transfer knowledge from domain experts to AI agents like Juji chatbots. While I am impressed by his methodological way of gathering and formulating the experts’ knowledge, I am worried about the huge amount of time he had to spend on doing this. To decentralize AI agents and make Human AI interaction smoother, I believe we will need to automate such knowledge acquisition process.
The first thing we want to do is to understand the process. Knowledge can be tacit or explicit. Some are hidden and only surfaces when you get your hand dirty and familiar with the underlying assumptions. Here’s the general procedure that Henry takes:
Gather the knowledge from the expert(s) through conversation, and review and audit an AI agent’s pass performance.
Formulate the knowledge given the context, documents and requirements.
Insert the formulated knowledge into the AI agent.
Personalize the knowledge when presenting to the end users.
Review the end user experience to identify any issues or additional knowledge needed. When necessary, repeat the process.
Ideally, step 4 is handled by the AI agent automatically, which is the case for Juji. Step 1 will most likely always require some human in the loop to provide knowledge and instruction. Whereas, step 2 and 3 are something we would like to automate and streamline.