Healthy Friction in Job Recommender Systems

欢迎收听《数据怀疑论者》，这档播客探讨推荐系统的方法、应用场景及其影响。

Welcome to Data Skeptic, a podcast exploring the methods, use cases, and consequences of recommender systems.

欢迎收听《数据怀疑论者》推荐系统系列的又一期节目。

Welcome to another installment of data skeptic recommender systems.

今天，我们讨论的论文是《创造健康摩擦：确定求职推荐解释的利益相关者需求》。

Today, we're discussing the paper creating healthy friction, determining stakeholder requirements of job recommendation explanations.

这里有很多值得深入探讨的好内容。

A lot of good things to unpack here.

首先，最引人注目的是推荐系统在求职中的有趣应用。

First and foremost, the interesting application of recommender systems in job hunting.

而且，正如我们在讨论中会提到的，求职者并不是唯一相关的参与者。

And as we get into in the discussion, it's not just the job seeker who's a relevant participant here.

我们今天的嘉宾罗恩及其合著者采用了一种基于图的学习技术。

Our guest today, Roan, and his coauthors use a graph based learning technique.

但更重要的是，他们专注于解释为什么给每个用户提供这些特定的推荐。

But more important than that, they focus on giving explanations for why the particular recommendations were given to each user.

这里比较直观的想法是，如果你解释为什么给出某个推荐，那应该是有好处的。

The rather intuitive idea here is that if you explain why a particular recommendation was given, that should be advantageous.

用户要么能理解背景并更欣赏它，要么他们能更快找到工作，至少是看起来有资格且感兴趣的工作。

Either the user might understand the context and appreciate it more, or maybe they'll find a job faster, at least one that they seem qualified for and interested in.

但这些解释的存在却带来了一个出人意料的结果。

But the presence of those explanations has a surprising outcome.

我就不剧透结果了。

I won't spoil the punchline.

我们会在采访中聊到，采访马上开始。

We'll get into that during the interview, which should start right about now.

我的荷兰语名字是Roan Schellingerhout，我来自马斯特里赫特大学。

My name in Dutch is Roan Schellingerhout, and I am from Maastricht University.

我是那里的四年级博士生。

I'm a fourth year PhD student there.

用英语说，我的名字可以叫Roan。

And in English, can say my name as Roan.

在荷兰语和英语里，名字的发音不一样。

It's different in Dutch and English.

那你能简单说说你在那边研究什么吗？

And can you share a few details on what you're studying there?

我现在在读博士。

So I'm currently doing a PhD.

具体的研究方向是可解释的多方利益相关者推荐系统。

The exact topic is explainable multi stakeholder recommender systems.

在我们的案例里，我们特别关注招聘，也就是人力资源这块。

And in our case, we're looking specifically at recruitment, so HR.

你能详细讲讲多方利益相关者推荐系统吗？

Could you zoom in on multi stakeholder recommendation systems?

这个前缀我平时不太见到。

That's not a prefix I encounter too often.

基本上，这就是一个推荐系统，但里面有好几方利益相关者都需要使用这个系统。

So it's basically just a recommender system in which there are multiple stakeholders that have to use the system.

在招聘的场景下，就是求职者，还有公司，公司里做HR的人，还有招聘人员，他们可能在公司工作，也可能在第三方机构工作。

So in the case of recruitment, that's job seekers, but also companies, people working in HR at companies, and recruiters who can either work at a company or at a third party.

所以总的来说，多利益相关者推荐系统就是需要平衡多个利益相关者需求的推荐系统。

So in general, a, yeah, multistakeholder recommender system is just a recommender system where multiple stakeholder needs need to be balanced.

我邀请你来是想讨论那篇论文，叫《创造健康摩擦：确定职位推荐解释的利益相关者需求》，我们肯定会有很多后续问题要聊。

And I invited you to come on to discuss the paper, creating healthy friction determining stakeholder requirements of job recommendation explanations, which obviously, we're gonna have a lot of follow ups about.

但我想在深入讨论之前，我注意到你在推荐系统和个性化领域还有很多其他发表。

But I thought before we jump right in, I've noticed you've got also a lot of other publications in and around the recommender system and personalization space.

你能谈谈你广泛的研究兴趣吗？

Could you talk about your sort of broad research interests?

你都在关注哪些方面？

What all are you taking a a look at?

这最初是从我本科论文开始的。

So it started initially during my bachelor's thesis.

那时候，我开始写一篇论文，研究YouTube推荐系统如何处理阴谋论内容，基本上就是推荐系统如何处理那些在某些情况下技术上不允许出现在平台上的内容。

There, I started working on a paper that looked into the YouTube recommender systems treatment of conspiracy content, basically how the recommender system dealt with content that wasn't technically allowed to be on the platform in some cases.

那时候，我试图打开那个黑箱，这让我对可解释性产生了兴趣。

There, I was kind of trying to open up the black box, which sparked my interest for explainability.

然后在我的硕士论文期间，我和任仕达合作过。

And then during my master's thesis, I worked with Randstad.

那是一家招聘公司，我记得是在美国。

It's a recruitment company, I think, in The United States.

它叫Monster.com。

It's called monster.com.

我觉得那是他们在美国的姊妹公司。

I think that's their sister company in US.

在那里，我研究了可解释的职业路径预测。

There, I worked on explainable career path prediction.

也就是说，根据一个人到目前为止的职业生涯，尝试以可解释的方式预测他们的下一份工作。

So given someone's career up until now, tried to predict what their next job will be in an explainable manner.

我对可解释性和推荐系统都很感兴趣，而且当时主要关注招聘领域。

I had interest in both explainability and recommender systems and then while focusing on recruitment.

所以，当我开始在马斯特里赫特大学读博士时，我就想把所有这些结合起来。

So So then when I started my PhD at Maastricht University, I tried to combine all of those.

所以我们花了很多精力研究如何满足非专业用户对可解释性的需求。

So we look a lot into how we can meet explainability needs mainly of lay users.

基本上就是那些没有太多技术背景的人，对，我们要确保他们能理解并运用工作推荐的解释。

So, basically, people who don't have a lot of technical expertise, yeah, how we can make sure that they are able to understand and work with explanations for job recommendations.

嗯，如果他们没有技术背景，你大概不会想给他们看SHAP值。

Well, if they have no technical expertise, you probably don't want to give them a SHAP value.

那你给他们看什么？

What do you give them?

对。

Yes.

没错，确实如此。

Exactly, indeed.

对，用SHAP和Lime这些工具，如果对方是数据科学家，那当然没问题，但普通人可不行。

Yeah, working with SHAP and Lime and whatever, that's perfectly fine if you're dealing with data scientists, of course, but not with your average Joe.

在我博士期间做的第一篇论文里，我们问的正是那个问题：我们该给他们看什么？

In the first paper that we worked on during my PhD, we kind of asked that exact question, what do we give them?

所以在那次研究中，我们给他们看了文字解释，就是纯文字，或者柱状图，这当然已经更接近Lime和Schap的方向了，还有一种我们展示的是基于图形的解释。

So in that case, we showed them a textual explanation, so just a written text, or bar charts, which are already going a bit more in the direction of Lime and Schap, of course, or one that we showed was a graph based explanation.

所以，是的，基本上就是一个由节点和边连接起来的网络。

So, yeah, it's basically a network of nodes and edges that are connected.

然后在那里，你可以看到一条路径，从“因为你有这个经验，它和这个子领域相关，而这个子领域对这个工作很重要”这样延伸出来。

And then in there, you can see kind of a path going from, oh, because you have this experience that's related to this subfield, which is relevant for this job.

所以我们展示的就是这些类型的解释。

So those were the kinds of explanations that we showed.

我们发现，对大多数普通用户来说，文字解释是最容易理解的。

And we found that for most lay users, textual explanations are the easiest to deal with.

就是一段连贯的文字，能让你明白为什么这个工作适合你。

Just a running text, yeah, gives you an overview of why a job is fit for you.

好的，我很想逐一了解这些解释的用户体验。

Well, I'd love to go through the sort of user experience of each of those.

也许我们可以从纯文本解释开始。

Maybe we can start with just textual explanations.

我想象自己是一个在网站上的求职者。

I'm imagining myself a job seeker on a website.

通常，我会先搜索一下，然后得到一堆列表或匹配项，我想你可以这么叫它们。

Typically, I've done some search and then I've got a bunch of listings or matches, I guess you'd call them.

文本解释会出现在哪里，它会告诉我什么呢？

Where does the textual explanation appear for me and what might it tell me?

具体位置当然取决于提供者。

The exact location, that would be up to the provider, of course.

但我们关注的是，当你已经根据文本、描述等有了一份简短的心仪工作列表时，点击它们时，也会给出文本解释。

But what we focused on was that when you already have a bit of a short list of potential jobs that you're interested on just based on the the text and maybe the description and whatever, Then when you click on them, it also gives the the textual explanation.

在那里，基本上就是应该包含的内容，至少是招聘人员可能会告诉你的。

And there, yeah, it's basically what is supposed to be in there, at least is what a recruiter might also tell you.

就像，哦，你有这种教育背景，这很合适。

So it's like, oh, you have this type of education which is suitable.

你有这样的工作经验，因为这样那样的原因，很适合这份工作，我们觉得你能在这个环境里表现好。

You have this work experience which fits this job because of x, y, and z reason, and we think you'll do well in this environment.

就是一些你期望从招聘人员那里听到的通用说法。

Just general arguments that you would expect from a recruiter as well.

那你们是怎么生成这些解释的？

And how do you end up generating the explanations?

我们主要关注的是使用知识图谱，然后把这些知识图谱输入到大型语言模型中，以生成容易理解的解释。

What we are focusing on mainly is using knowledge graphs and then feeding those knowledge graphs into large language models to get easy to understand explanations.

因为我们发现，使用基于图谱的解释可能非常困难，因为它们又相当技术化。

Because what we found out was that using those graph based explanations can be very difficult because they're, again, pretty technical.

同时有很多事情在发生。

There's a lot of things going on at the same time.

所以，如果我们基本上把图谱解释，比如图谱的JSON版本，输入到LLM中，然后告诉LLM，基于这个，提供一个容易理解的解释。

So if we then feed the graph explanation basically into an LLM, like a JSON version of the graph, and then tell the LLM, based on this, provide an easy to understand explanation.

LLM的输出就是展示给用户的内容。

The output of the LLM is then what is shown to the user as well.

你之前也提到了柱状图。

You'd mentioned bar charts as well.

那种情况下用户体验怎么样？

What's the user experience like in that case?

我们当时主要想看看，柱状图在解释说明方面算是行业标准，或者说最常见的做法。

There, we basically wanted to see bar charts are kind of the industry standard when it comes to explanations or the most common, I guess, when it comes to explanations.

所以我们想看看，如果把我们习惯用的东西展示给普通用户，就是一般的老百姓，他们实际会有什么反应。

So we wanted to see, yeah, if you'd present what we're used to to a lay user, to an average Joe, how do they actually respond?

结果我们发现，普通用户其实很不喜欢柱状图。

And what we found is that lay users really do not like bar charts.

他们说，我看不懂这上面画的是什么。

They said, I don't understand what I'm looking at.

我不知道该拿这个怎么办。

I don't know what I'm supposed to be doing with this.

他们基本上看了一眼，一脸困惑，然后就转去看文字说明或者基于图形的解释了。

They basically looked at it, were confused, and then moved on to to the textual or the the graph based explanation.

所以我们发现，除了稍微辅助一下文字说明之外，这些图表基本上没什么用。

So we found that there's very little use generally other than maybe supporting texts a little bit.

所以我们当时做的是，先有文字解释，然后在里面高亮一些对应系统特征的术语。

So what we did was at one point, we had textual explanations, and then in there, we would highlight certain terms that corresponded to features in the system.

然后接下来，文字会说，哦，你的教育背景很重要。

And then next, the text, it would say, oh, your education was very important.

然后会有一个柱状图。

And then there would be a bar chart.

然后在“教育”标签下面，你可以看到柱子挺高的。

And then under the label education, you could see that the bar was quite large.

所以它算是总结了文字，但更多是辅助性的，而不是本身提供信息。

So it kind of summarized the text, but it's more supported than actually being informative itself.

嗯，你提到你收到了一些用户的反馈。

Well, you've mentioned you had some feedback from users.

你得到了比金子还宝贵的东西，就是你项目的真实用户反馈数据。

You've got something I consider to be worth more than gold, actual user response data for your project.

你能稍微讲讲你做的那个研究吗？

Could you talk a little bit about the study you conducted?

在这个健康摩擦的研究里，我们基本上搭建了一个环境，让用户能查看职位列表和解释。

So for this healthy friction study, what we did was we basically created a environment where users could look at job listings and explanations.

我们有两种不同的解释，一种是由真正可解释的推荐系统生成的，另一种就是瞎编的。

We had two different types of explanations, one type that was generated by an actually explainable recommender system, the and other one were just nonsense.

那些就是随机生成的解释，里面的数值全是乱来的。

They were just randomly generated explanations where all of the values were just totally random.

然后我们邀请了不同利益相关者类型的人来参与。

And what we then did was invite different individuals of the the different stakeholder types.

我们邀请了10个求职者、10个招聘人员，还有10个HR，也就是公司代表，来实际使用这个系统。

So we invited 10 job seekers, 10 recruiters, and 10 HR people, company representatives is what we called them, to actually engage with that system.

我们基本上就是说，嘿。

We basically just said, hey.

系统在这儿。

Here is the system.

尽量少给支持，自己摸索，因为应该像真实世界的例子一样。

Try to figure it out with as little support as possible because it should be like a real world example.

最后，我们让他们根据解释选一个工作，比如你觉得哪个最合适，然后还让他们做几个主观评价，比如你觉得推荐的可信度怎么样，有用性如何，诸如此类。

Then at the end, we ask them to pick a job based on the explanation, like which one do you think is the best fit, and then also ask them for a couple of subjective evaluations, like how trustworthy do you think the recommendations were, how useful were they, stuff like that.

这就是我们用的方法。

So that was the the approach we used.

嗯，从求职者的角度来看，我会看工作和解释。

Well, thinking about it from the job seeker's perspective, I'm gonna look at the job and the explanation.

我是从自己的角度去看，还是你给一份候选人的简历，然后我扮演一个角色？

Am I looking at it from my own point of view or is are you giving me like a candidate resume and I'm role playing a bit?

这种情况下，我们确实做了一些角色扮演。

So in this case, we we did some some role playing.

我们让所有用户读一份简历或一个职位列表，然后告诉他们假装成公司或代表的那个人，然后替他们找工作或候选人。

We had all of the users read either a resume or a job listing and then told them to pretend to be the company or the the individual who that represented and then look for a job or a candidate on their behalf.

嗯，随机解释，我猜会是一个好的基线，我应该衡量可解释版本应该会好一些。

Well, random explanation, I presume, would be a good baseline, and I should measure that the explainable version should be somewhat better.

你是怎么测量的？

How do you go about measuring it?

我们设了客观变量和主观变量。

So what we that we had objective variables and subjective variables.

客观变量主要是基于随机解释和真实解释。

So the objective variables was basically based on the random and the the real explanations.

人们最终是否做出了正确的决定？

Do people end up making the correct decision?

他们是否选择了最合适的工作？

Do they choose the best fitting job, basically?

因为是从五份工作或候选人中挑选。

Because it was a list of five jobs or candidates that they had to pick from.

另外，客观方面，我们测了他们花了多长时间，多少分钟，多少秒才做出决定。

And then also, objectively, we determined how long does it take them, how many minutes, how many seconds do they take before they end up reaching a decision.

主观指标就是我刚才提到的那些。

And then the subjective metrics were what I mentioned.

所以基本上就是，你觉得这个系统有多可信？

So it was basically, like, how trustworthy do you feel the system is?

你觉得这些解释和整个系统有多大用处？

How useful do you feel that the explanations and the the system in general was?

还有我们说的透明度。

And how we call it the transparency.

就是，你有多清楚模型到底在做什么？

Like, how well do you understand what the model was actually doing?

就是，这些解释到底有多容易理解？

Like, how understandable basically were the explanations?

嗯，你之前提到有一个正确的候选人。

Well, you'd mentioned there was a correct candidate.

就我个人经验来说，我遇到过很多次，可能有两个候选人非常强，几乎就像抛硬币一样难选。

In my personal experience, I've been in a lot of cases where maybe there are two candidates that are very strong and it's almost a coin toss.

那你怎么确保在一个非算术问题里，只有一个正确答案呢？

How do you ensure that there's exactly one correct answer in a nonarithmetic problem?

这确实是真的。

That is definitely true.

在招聘这件事上，显然永远没有一个完全正确的答案。

There's obviously never a, like, fully correct answer when it comes to recruitment.

这在求职推荐系统里也是一样。

That's also in job recommender systems.

这可能会很模糊，因为归根结底，大部分都是基于真实招聘人员的数据。

It can be pretty fuzzy because at the end of the day, most of it is based on data made by actual recruiters.

但是，是的，招聘人员也是人，所以他们不可能什么都完美地知道，或者不可能什么都完全准确地知道。

But, yeah, recruiters are humans, so they can't know everything perfectly or can't know everything exactly right.

所以在这篇论文里，我们使用了一个在线数据集。

So in the the paper, we used a an online dataset.

但那里有基于某人是否……我记得有多个级别，从他们是否申请了工作，是否有面试，还是真的被录用了，诸如此类。

But in there, there were ground truth values based on whether someone I think there were multiple levels going from whether they applied for the job, they had a job interview, or they actually got accepted to the job, something like that.

级别最高、真实值最高的选项被认为是正确答案。

The option that had the highest level, the highest ground truth value was considered to be the correct answer.

那你的研究对象有没有大致还原出那个结果？

And did the subjects in your study sort of faithfully reproduce that?

还是说他们在选对答案方面有一定的准确度？

Or is there some level of accuracy in picking the correct one?

哦，不不不。

Oh, no, no.

绝对没有。

Definitely not.

在研究中，我们确实看到人们会做出错误的决定。

In the study, we actually saw that people made incorrect decision.

他们通常选的是次优的选项。

They usually picked the second best option.

所以那是一个。

So that was one.

我觉得分数是从零到三，零是完全不匹配，三是最匹配。

I think the scores went from zero to three, being like no match whatsoever up to a perfect match.

然后他们选的是第二高的那个，我觉得。

And they picked the one that was the second highest, I think.

一个真实值是一分，另一个是三分，他们经常选那个一分作为真实值。

One that scored a one as a ground truth and one that scored a three, and they often picked the one that scored a one as the ground truth value.

就是次优选项。

The second best option.

有意思。

Interesting.

那么，在那些看到可解释性解释和随机生成的解释的受试者之间，你看到了哪些不同的地方？

Well, what differentiators did you see when looking across the subjects that saw the explainable explanations that were generated versus those that saw the randomly generated explanations?

实际上，我们看到的差异比预期的要少。

So we actually saw fewer differences than we expected.

比如，我们事先以为，哦，这会是天壤之别。

Like, in advance, we thought, oh, this will be night and day.

比如，他们应该能很明显地分辨出哪个是真的，哪个是随机的。

Like, they'll very obviously be able to tell which is real and which is random.

但结果人们其实没怎么注意到差别。

But it turned out people actually didn't notice all that much of a difference.

有一些统计上不显著的趋势，尤其是在主观变量方面，他们稍微更喜欢真实的解释，但差异相当小。

There were some non statistically significant trends, especially with the subjective variables, that they liked the real explanations a little bit better, but the differences were pretty minor.

我们其实很好奇，为什么人们分辨不出差别，即使差别很大，因为至少作为研究者，对我来说，哪个是真的，哪个是随机的，非常明显。

We actually were really curious why are people not telling the difference even though it's very big because it was to me, at least as the researcher, it was very obvious which one was the real one and which one was the random one.

所以当我们观察他们的行为时，我们录了屏，还采访了他们，让他们在使用系统时大声说出想法。

So when we looked into their behavior, we screen recorded everything, and we interviewed them, asked to talk aloud while working with the system.

所以我们基本上发现，人们看了解释，然后基本上把它当作信息来源，而不是做出某个决定的理由。

So what we basically found out was that people were looking at the explanations and then basically using it as a source of information, but not as a reason to come to a certain decision.

所以他们看着解释，然后说，我不知道。

So they were looking at the explanation, and they said, I don't know.

你学的是经济学。

You studied economics.

这真的很适合你。

This is a really good fit for you.

然后他们就会读一下，然后说，哦，这个人学过经济学。

And they would just read and be like, oh, this person studied economics.

我觉得那意味着他们会非常适合这份工作。

I think that would mean they would be a very good fit for this job.

所以基本上，他们会把解释当作信息来源，但最后还是自己做决定。

So, basically, they would use the explanations as an information source but then still make their own decision.

不管系统在解释里怎么说，他们都会用自己的经验，比如作为招聘人员、求职者或公司代表，来得出结论。

Like, regardless of what the system was saying in the explanation, they would use their own experience as a recruiter or a job seeker or a company representative to come to conclusion.

所以我们看到，随机解释和真实解释之间的结果差异其实没有那么大。

So we actually saw that the difference between the random and the real explanations as a result weren't all that large.

有意思。

Interesting.

所以这几乎就像你的用户没有把解释当作特别的东西，它只是另一个数据点。

So it's almost as though your users didn't treat the explanation as special, that it was just yet another data point.

基本上，就是这样。

Basically, that's exactly right.

他们在其他那些品质上，比如可信度、有用性或透明度，有没有表达出什么不同？

Did they express any differences in terms of those other qualities like trustworthiness or usefulness or transparency?

我希望可解释的答案在某个方面能有点优势。

I would hope the explainable answers have some edge somewhere here.

就像我提到的，他们稍微倾向于真实的解释，但这一趋势在统计上并不显著。

So like I mentioned, there's a small trend for them to prefer the real explanations, but it's a nonstatistically significant trend.

这当然可能是因为我们的样本量只有30人。

That could, of course, be because our sample size was only 30 people.

在像这样的用户研究中，当然很难获得非常大的样本量。

In a user study like this, it's very difficult, of course, to get a very large sample size.

但只有30人，或者说n等于30，要得到统计上显著的结果是相当困难的。

But with 30 people or an n of 30, it's quite difficult to get statistically significant results.

所以，总的来说，我们看到他们稍微偏好真实的解释，但我们不能确定情况就是这样，因为这也可能，当然，是随机因素造成的。

So, generally, we saw that they slightly preferred the real explanations, but we can't say for certain that that's the case because it could also, of of course, be down to random chance.

嗯，我也想深入了解一下推荐过程。

Well, I'd love to zoom in on the recommendation process as well.

你之前提到过有知识图谱参与。

You'd mentioned earlier having a knowledge graph involved.

知识图谱里有什么样的知识？

What sort of knowledge is in the knowledge graph?

我们构建知识图谱的常见方式基本上是基于表格数据生成一个初始图谱。

The common way that we create the knowledge graph is basically to generate a starting graph based on tabular data.

所以我们有招聘机构存储的表格数据。

So we have the tabular data that is stored by the recruitment agency.

那就是他们掌握的关于候选人的所有信息。

So that is everything that they have on a candidate.

包括之前的工作经验、技能，还有可能的位置，基本上什么都有。

So it's anything from their previous work experience, the skills that they have, could be their location, everything basically.

然后我们定义一个本体，也就是对知识图谱应该长什么样的定义，以及不同关系可以是什么样的。

And then we define an ontology, so a definition for what the knowledge graph should look like, so what different relationships can look like.

然后我们进行推理来获得新的关系。

And then we do inference to get new relationships.

比如说，如果你做过数据科学家的工作，那么就可以推断出这个人肯定也有统计学的技能。

For example, if you have worked as a data scientist, then you can infer that that person must also have the skill statistics.

虽然不一定总是这样，但作为一个例子，这是你可以推断出来的。

Might not always be the case, but as an example, that's something that you could infer.

所以知识图谱就是这样生成的。

So that's how the knowledge graphs are generated.

我们，我们拿表格数据，然后对它进行推理。

We, we take the tabular data and then, do inference on it.

然后推荐本身呢。

And then the recommendation itself.

我知道你基本上是在匹配，我猜，就是候选人和职位的配对。

I know you've you're essentially matching, I guess, pairs really of a candidate and a vacancy.

这从某种意义上说很直观，但你是怎么在数据中表示它的呢？

And that's intuitive in a sense, but how do you represent that in the data?

在数据中，这其实不是问题，因为你只需要一对ID，就是候选人ID和职位ID，然后还有它们的真实值。

So in the data, it's not necessarily an issue because you just have a pair of IDs, so the candidate ID and the vacancy ID and then their ground truth value.

所以，是的，从零到任意值。

So, yeah, zero to whatever.

但对于系统本身来说，这有点棘手，因为你真的不能同时双向进行。

But for the system itself, it's a bit tricky because you can't really do it both ways at the same time, really.

所以我们做的是，我们有一个从候选人到职位的知识图谱，还有一个从职位到候选人的知识图谱，因为图是有向的。

So what we did was we have a knowledge graph going from the candidate to the vacancy and one going from the vacancy to the candidate because graphs are directed.

所以一条边从一个方向走可能不会从另一个方向走。

So an edge going one way might not be going the other way as well.

然后我们基本上同时使用这两个图，从两边预测匹配程度如何。

And then we basically took both of those graphs at the same time and made a prediction on how good the match was from both sides.

然后，是的，基本上将这两个预测结合起来，做出最终预测，看他们双向匹配得有多好。

And then, yeah, basically joined those two predictions together to make a final prediction how well do they match both ways.

你能多谈谈你有哪些可用的特征吗？

Could you talk a little bit more about the features you have available?

这真的取决于公司。

That really depends on the company.

每家公司都有自己认为重要的特征选择。

Each company has their own selection of features that they find important.

说实话，我很难说出一套通用的规则。

Honestly, I I wouldn't be able to say much for what the general rules are.

我知道和我合作的公司用了哪些数据，但我不确定他们是否愿意让我现在开始透露他们存储的用户数据。

I know what is used in the companies that I work with, but I'm not sure if they would like if I now start telling about all of the data that they store on the users.

但这些数据范围很广。

But it's very broad.

任何与推荐相关的数据都可能被用到。

It can be anything that can be relevant for a recommendation.

也许我们可以把很多内容归为活动数据，一笔带过。

Maybe we can sweep a lot of that under the rug saying there's activity data.

点击行为显然很相关。

Clicks are clearly relevant.

它们能反映你的意图。

They convey your intent.

你至少得把这些都记录下来。

You'd want at least those tracked.

然后你还有简历。

Then you've got a resume as well.

你是把它拆成嵌入向量，还是说图里还会放别的东西？

Do you chop it up into embeddings, or what else goes into the the graph?

好的。

Okay.

在我主要处理的场景里，通常点击数据不多。

So in the case that I work with, mainly, there isn't usually a lot of click data.

基本都是人工标注的，所以实际上是招聘人员的决定。

It's usually all human annotated, so it's actually recruiter decisions.

然后不同的级别可以是，对，就像我提到的，比如某人有没有面试，最后被录用了，或者面试后被拒绝了，诸如此类的。

And then the different levels can be, yeah, like I mentioned, whether someone had a job interview, actually got accepted, or had a job interview and then got rejected, whatever, stuff like that.

但都是人工标注的。

But it is human annotated.

而且，是的，简历和职位空缺怎么处理也取决于公司。

And, yeah, how CVs and vacancies are are dealt with depends on the company as well.

所以有时候，就直接用纯文本。

So sometimes, it's just straight up the plain text.

他们从简历中提取文本。

They extract the the text from the CV.

但有时候也会用某种嵌入，从TFIDF到基于BERT的Transformer模型嵌入都有。

But sometimes it's also a a type of embedding, anything from TFIDF to BERT based transformer model embedding.

在知识图谱中，我们最初的做法是，为简历本身创建一个节点，并把嵌入值存储在图中的那个节点里。

And in the knowledge graph, how we did it initially was actually we created a node for the CV itself and stored the embedding value in that node in the graph.

但我们发现，人们有时会因此感到困惑，因为简历中可能有些内容并不一定在图谱的其他部分体现。

But what we found was actually that people kind of sometimes got confused as a result because there might then be something in the CV that isn't necessarily represented in the rest of the graph.

所以，如果某些内容没有存储在结构化数据中，但在简历中提到了，那就会出现不匹配。

So if something isn't stored in the structured data but is mentioned in the CV, then there will be a mismatch.

所以，实际上，我现在正在写的一篇论文，是和哥本哈根IT大学的人合作的。

So actually, a paper that I'm working on right now, it's a collaboration with people from the IT University Copenhagen.

在那里，我们实际上在研究如何根据简历和职位列表自动生成知识图谱。

There, we're actually looking into automatically generating knowledge graphs based on CVs and job listings.

所以，在那里，同样，一个大型语言模型会查看简历或职位列表。

So there, again, an LLM takes a look at the CV or the job listing.

然后根据全文，它会生成一个边列表，并构建一个图谱。

And based on the the full text, it generates a list of edges and does a a graph.

我猜技能在其中扮演了重要角色，它们处于中间位置，连接着拥有特定技能的候选人和要求特定技能的职位。

I guess skills are heavily involved, and they're sort of in the middle there between a candidate who can have certain skills and a job which requires certain skills.

这是通过某种分类体系编码的，还是训练过程中的某些机制自动解决了这个问题？

Is that sort of encoded in a taxonomy or does something about the training process figure it out on its own?

两者都有。

A bit of both.

实际上，有很多分类体系都涉及技能，还有职位分类，这样就能对不同职位类型进行清晰的层级划分。

There's multiple taxonomies actually that work with skills, also with job classification so that there's a nice hierarchical overview of different job types.

比如，有欧洲标准职业分类数据集，也就是ESCO数据集。

But you have the European standard classification of occupations dataset, the ESCO dataset.

而且在那里，我记得有将近14000种不同的技能，每种都有定义，还和不同的工作相关联，也就是说哪些工作对这些技能有要求。

And in there, they I think it's near 14,000 different skills that all have a definition and also are linked to different jobs, so for which jobs those skills are important.

但这绝对也是可以学习的东西。

But it's also definitely something that can be learned.

公司数据集里的技能不一定全都在ESCO数据集里。

Not all of the skills that are in the company dataset might be in the ESCO dataset.

所以在这些情况下，模型会自己学习哪些技能对哪些工作重要。

So in those cases, the model would itself learn which skills are relevant for which jobs.

嗯，你之前分享过关于文本解释和图形解释的细节。

Well, earlier you were sharing details about the textual explanations and also the graph one.

我们跳过了图形解释这第三种情况，我想现在你已经描述得够多了，用户可能能想象出来。

We skipped over that third case of the graph explanation, which I think now you've described enough users can maybe picture it.

对于看到图形解释的人来说，用户体验是怎样的？

What was the user experience like there for someone seeing one of the graph explanations?

我们发现，这很大程度上取决于个人的背景。

So what we saw was that it was pretty dependent on the background that the person had.

所以尤其是公司代表们对基于图形的解释更积极一些，主要是因为如果你在人力资源部门工作，过去很可能也用过各种可视化工具。

So especially the company representatives were a bit more positive about the graph based explanations, mainly because if you work in HR, they're also pretty high that you've worked with different visualizations in the past.

所以特别是那些有更多——可能不是统计方面的，但一般数据可视化经验的人，他们很快就能掌握，而那些非常关注人的方面的招聘人员，他们往往看着就挺不知所措的。

So especially people with a bit more, maybe not statistical, but general data visualization experience, they were able to get the hang of it pretty quickly, whereas the people who were the recruiters who were very much focused on the human aspect of things, they often looked at it and were pretty overwhelmed.

在稍微解释一下之后，他们通常会说，哦，对。

After explaining it to them a bit, they would usually say like, oh, yeah.

好的。

Okay.

我觉得我能看到它的用处，但我还是用文字解释吧。

I I think I could see the use of that, but I'll just stick to the the textual explanations.

基于图形的解释可能会引起很大分歧。

The graph based explanations can be pretty divisive.

但那些确实很快理解的人，他们也会说他们挺喜欢的，因为它一眼就能提供很多信息。

But the people who did actually understand them pretty quickly, they did also say that they liked them quite a lot because it does provide a lot of information at a glance.

因为不用读整段文字，你只要跟着粗线走，就像图中重要的路径，就能很好地理解模型内部发生了什么。

Because instead of having to read a full text, you can just follow the thick lines, like the important paths through the graph and have a pretty good understanding of what is going on inside the model.

那你的参与者的反馈是怎么影响你未来的工作的？

Well, how has the feedback from your participants shaped your future work?

有一件事很大程度上是基于我们之前工作里的发现，就是我之前提到过的，在源文本——也就是简历和职位描述——和解释文本以及图表本身之间存在一些不一致。

One thing that was very much based on what we found in previous work was one thing that I mentioned where there was a bit of a mismatch between what was written in the the source text, so the the CVs and the job listings, and the explanation text and the the graphs themselves.

所以这是我们目前想要解决的问题，就是让这两者之间更加一致。

So that is something that we're aiming to tackle right now, that there there's more of an alignment between the two.

这是我们之前研究中学到的第一件事，也是我们在未来的研究中会落实的。

That was the first thing that we learned from our previous research that we implemented in future research.

另一个就是，非专业用户非常强烈地偏好文本解释。

And the other one is basically that lay users very strongly prefer textual explanations.

所以求职者和招聘人员对文本解释有很强的偏好。

So the job seekers and the recruiters had a pretty strong preference for textual explanations.

所以我们真正想弄清楚的是，如何生成一个好的文本解释，因为在我们第一篇论文里，我们实际上也采访了人，有一个人，具体说是一位招聘人员，他说，哦不。

So what we actually wanted to figure out is then also how do you generate a good textual explanation because one person in our first paper, actually, we interviewed people as well, and there are one person, one recruiter specifically said, oh, no.

我喜欢解释里有很多文字，因为这样如果我给人打电话，我有很多话题可以聊，我们可以就所有事情进行愉快的交谈。

I like it when there's a lot of text in the explanation because then if I call someone up, I have a lot of things to talk about, and we can have a nice conversation about everything.

然后第二位招聘人员其实说，不。

And the second recruiter actually said, nah.

就像，三句话，或者总共三段。

Like, three sentences, maybe three paragraphs in total.

类似这样的内容就足够了。

Something like that is is more than enough.

剩下的我自己会搞明白。

I'll figure out the rest myself.

所以我们基本上想弄清楚是什么造成了这种差异。

So we wanted to figure out what creates that difference, basically.

比如，什么样的性格特征会让一个人更喜欢某一种类型的文本。

Like, what personality traits make a person prefer one type of text over another.

所以在最近的一篇论文中，我们研究的基本上是将性格特征与文本解释偏好进行比较。

So in a a recent paper as well, what we looked into was basically comparing personality traits to textual explanation preferences.

所以我们为解释设定了不同的设计维度。

So we had different design dimensions for the the explanation.

所以就是长度，是连续文本还是项目符号列表，是非常详细，比如具体说这个贡献了67%之类的，还是就说贡献很大，是正式还是非正式，还有是说服性的还是决策支持导向的。

So their length, whether they were running text or bulleted lists, whether they were very detailed, so actually saying this contributed 67% or whatever, or just this contributed a lot, whether it was formal or informal, and then also having it be persuasive or decision support oriented.

然后基本上就是给用户做了些小规模的人格测试，然后弄清楚人格特质是怎么影响解释偏好的。

And then basically did, like, small personality tests with users and then figured out how do personalities affect explanation preferences.

你有没有一个愿景，就是你的工作，你知道的，智慧从中而来，招聘网站也学习并采用你提出的一些技术或想法。

Do you have a vision for how your work as it gets you know, the wisdom comes from it and job boards learn and they implement some of the techniques you propose or ideas you've had.

你有没有一个愿景，就是我们正在朝着什么方向演变？

Do you have a vision for what we're evolving towards?

真正的目标是什么？

What's the real goal here?

所以基本上就是希望。

So it's basically the hope.

总的来说，AI不会被用来取代招聘人员，而是专门用来支持招聘人员匹配求职者和职位，同时也让求职者的求职过程不那么痛苦，因为浏览成百上千个职位可能非常累人，但这也是一个很大的决定。

AI in general, it won't be used to replace recruiters, but it will be used specifically to support recruiters in their task of matching job seekers and jobs, and then also making the job hunt for job seekers a little bit less of a a grueling task because going through hundreds of jobs can be very tiresome, but it's also a very big decision.

如果申请工作时选错了，会有很多沉没成本。

It's there's a lot of sunk cost involved if you make a a wrong choice when it comes to applying to a job.

我的意思是，只要你能收到回复。

I mean, as long as you actually hear back.

但如果你经历了整个面试过程，然后发现，哦，其实这份工作并不适合我，那显然是个大问题。

But if you go through the full interviewing process and then figure out, oh, actually, this job wasn't really for me, that's obviously a big issue.

所以我们想找到一个平衡点，让求职者能快速浏览大量工作，并且对为什么匹配有很好的理解，这样我仍然可以个人思考，比如，哦，好吧。

So we want to find a nice balance where job seekers can quickly go through a lot of jobs and have a pretty good understanding of why it would be a good match so that I can still personally think like, oh, okay.

我觉得这些论点对我很重要，所以我认为这可能值得追求，或者我应该把资源集中在其他地方。

I think these arguments are important to me, so I think this might be worth pursuing, or I should focus my resources elsewhere.

现在有很多不同的求职门户网站，它们因国家和地区的不同而有所差异。

So there's a lot of different job portals out there and they vary by country and by region.

我们俩可能能说出十几个来。

And we could probably name a dozen between us.

它们都有某种共同的核心功能，基本上就是某种搜索，通常是基于位置的搜索，可能还有一些筛选器或功能，然后你会得到一个按某种理想顺序排列的工作列表。

They all have some sort of common core there that essentially do some sort of search, usually a location based search, maybe some filters or features, and then you get a list of jobs that are in some ideal order for us.

你觉得这种格式是最优的吗，还是我们需要一些革命性的改变？

Do you think that format is optimal, or do we need something a little revolutionary here?

我觉得这里确实有个问题，如果你只是输入一个关键词，我也不知道。

I mean, one issue there indeed, if you just type in a keyword, I don't know.

你输入，比如，软件工程师之类的，你得到的就是最无聊的列表。

You type in, yeah, software engineer or whatever, you get the most bland list imaginable.

你只会得到所有匹配软件工程师的职位，然后可能还有你所在地区的，你可以稍微调整一下，比如薪资范围或者教育水平。

You just get everything that matches software engineer and then probably indeed in your area, and you can, like, fine tune it a little bit maybe on, like, the salary range that you're looking for in the education level.

但我认为更理想的方式，虽然更难，当然也更耗计算资源，是能够上传你的简历，然后根据你的完整简历得到个性化的推荐，这样比如，在你的简历里，如果你有一个个人简介部分，你提到你是什么样的人，比如你是否有团队精神之类的，这些也会被考虑进去，而不仅仅是你硬性的资历。

But what I consider to be a lot more optimal, albeit more difficult and, of course, compute intensive, is to actually be able to upload your your resume and have personalized recommendations based on your full resume so that it also for example, in your resume when you have a little personal section where you talk about what type of if you're a team player, stuff like that, that that's also taken into consideration and not just your very hard qualifications, basically.

你接下来有什么打算？

What's next for you?

我现在是博士的第四年，所以我应该很快就要准备收尾了。

So I'm currently in the fourth year of my PhD, so I should be looking to wrap things up soon.

我提到的关于自动化知识图谱构建的研究还在进行中。

So the research I mentioned about automated knowledge graph construction is still a work in progress.

我们现在已经能够用不同的语言模型和提示策略生成不同的知识图谱，但我们还需要评估它们实际效果如何，生成的图谱质量有多高。

We now manage to generate the different knowledge graphs using different LLMs and prompting strategies and whatnot, but we still have to evaluate how well they actually work, how high is the quality of the the graphs that are being generated.

我们还想进一步研究公平性的问题。

And we also want to look into a bit more fairness.

具体来说，就是性别公平和地域公平。

Specifically, want to look into gender fairness and location fairness.

所以当前的研究我们也会关注这些方面。

So we're also gonna be looking into that for the current study.

然后，如果一切顺利，我们能找到一家公司合作完成最终论文的话，我们基本上就想做一个真实世界的测试。

And then hopefully, if we manage to find a company that we can work with for the final paper, what we want to do is do a real world test, basically.

也就是说，不是模拟场景，也不是角色扮演，而是用真正在找工作的求职者的大数据集。

So instead of having a mock up situation, basically, so a role play situation, we want to be able to use a larger data set of actual job seekers who are actively looking for work.

当然，公司方和招聘方也一样。

And then, of course, the same goes for companies and recruiters.

嗯，这确实有很多令人兴奋的机会。

Well, exciting opportunities there for sure.

听众们可以在哪里关注到你呢？

Where can listeners follow you online?

我其实不太用社交媒体。

I'm not very active on social media at all.

大家可以在领英上找到我。

They can find me on LinkedIn.

我有时候会更新，比如参加会议的时候，发一些动态之类的。

I sometimes update when I'm at a a conference, stuff like that.

当然，还有谷歌学术，以及其他能找到论文的地方。

And, of course, on Google Scholar and wherever else you can find publications.

在那里也能找到我。

There, you can also find me.

我们会在节目笔记里放上链接，方便想继续关注的听众。

We'll have links in the show notes for listeners who wanna follow-up.

罗恩，非常感谢你抽空来分享你的工作。

Roan, thank you so much for taking the time to come on and share your work.

谢谢你的邀请。

Thank you for having me.