Purpose

This article answers the frequently asked questions about Eightfold Copilot and its capabilities.

What is the Copilot architecture?

Here is the basic Copilot architecture:

In the Pre-Processing step, we first check the moderation in the user query for every request, and then insert the context based on the page/flow that the user is currently viewing. This may use a technique similar to the Traditional Vector DB Retrieval Augmented Generation (RAG).

Is there a reason why Post-Processing evaluation cannot be done offline?

We are currently doing all the evaluations offline. We may perform a few critical evaluations online (real-time) in the future (if necessary).

Is calibrating/training a custom model mandatory for Copilot to adapt to Talent Management data?

No calibration/training of a custom model is required.

Is there a robust guardrail against PII and Proprietary Information leaks?

Yes, we have an infrastructure to actively mask any PIIs. Data transmitted to OpenAI via Eightfold can only be used for services, and not for any training purposes.

How does the LLM deal with unclear data in talent profiles, and how does it make decisions when information is missing or conflicting?

Copilot has an inbuilt mechanism to assess the model's performance in various areas such as diversity of responses, tonal accuracy, factual correctness, and contextual relevance.

Is there a dashboard/visualization to track the LLM performance over time?

We perform the quality evaluations as part of the release certification and are constantly working to evolve this process. As this is a cutting-edge and evolving space, we are planning to create and share offline reports of the model performance.

What are the KPIs used to evaluate the effectiveness of the GenAI solution?

We measure attributes related to tonality, diversity, context, and more.

How does Eightfold ‘Evaluate for bias capability work?

To spot bias in job postings, the Copilot looks for words or phrases that could unfairly target people based on factors such as gender, race, age, disability, sexual orientation, religion, or mental health. These biased phrases might be obvious or hidden within the wording.

For every biased phrase found, Copilot generates:

• A new, neutral phrase that still describes the job's needs for skills, experience (both in the field and how many years), education, and duties.
• An explanation of why the new phrase is better at being inclusive and how it counteracts any hidden biases from the biased phrase.

How do you test Copilot’s suggestions?

Eightfold ensures that Copilot works well for everyone by testing it in two ways: automated and human testing.

• Automated testing checks how Copilot performs against specific standards such as diversity, tone, staying on topic, and avoiding incorrect information. Each test gives a score, which we compare to a target score.
• Human testing is like pretending to use Copilot in real situations. We look at how well it works, how easy it is to use, and how accurate it is. We write down the results and fix any issues that are found.

Can customers include their models within Copilot?

Customers cannot integrate their own models within Copilot.

Can customers integrate their data for copilot to be more personalized?

Currently, this is only possible with the Candidate Copilot, where it can guide users to particular pages to answer their questions.

Eightfold is working on a bigger plan to make Copilot even better. Soon, it will be able to access a knowledge base and answer questions from both candidates and employees.

Can customers specify candidate profile linked sections to be considered for Profile Summary?

No, currently, the profile summary has two sections:

• General summary: this is a brief description of their work experience and might mention important points.
• Activity summary: this part shows their history with the organization, such as past applications, scheduled interviews, and outcomes.

Can we enable Copilot in production for a small group of users for a pilot test?

To enable Copilot for a certain set of users, we need a list of their email addresses. Customers can provide this list to their Eightfold CSM, who will work with the internal teams to get this enabled.

Can Copilot capabilities be selectively enabled in an environment?

Customers can select which Copilot capabilities they want to use. However, some capabilities are connected with each other. For example, the CopilotOne workflow ‘Help me invite applicants’ relies on the ‘Help me write email’ capability. This means that they can't be turned on or off separately.

How do the peer candidates work in the Debrief feature?

Peer candidates are individuals who have previously been interviewed by the same interviewer for a similar or identical position. Their profiles are used by recruiters and hiring managers to compare and prioritize other candidate profiles.

Is Copilot a feature or a product?

Copilot is branded as a separate product line, however, all Copilot’s features aim to support existing workflows across other product lines - TA, TM, PCS and others.

Do Copilot capabilities support internationalization?

While Copilot can process queries/prompts in different languages, its responses are currently only limited to English.

What are the plans for continuous improvement and updating of the LLM?

We continue upgrading our models after performing the necessary quality certifications.