From the journal

Emotion recognition in a Moroccan AI call-centre product sold to France

The EU AI Act prohibited practices rules make a proposed employee score the first decision in a sale. A Moroccan supplier offers a French financial-services call centre transcription, conversation-topic analysis and a score labelled employee stress or enthusiasm. The customer wants coaching and performance ranking. If the score infers employees' emotions from their voice characteristics, the supplier should remove that functionality from the French offer before supply. The customer must not deploy it for that purpose. (Regulation (EU) 2024/1689, Articles 2(1) and 5(1)(f).)

10 min read

A narrower product can retain transcription and service-topic analysis, subject to data protection and employment requirements. Its specification must prevent those functions from becoming a route back to employee emotion inference. Text-based performance assessment also needs separate scrutiny.

  • Determine what the score measures and how. Its label does not establish whether it infers an emotion from biometric characteristics, reports spoken words or measures a physical state.
  • Remove biometric employee emotion inference used for coaching or ranking. Human review, contractual disclaimers and processing in Morocco do not cure the prohibited use.
  • Keep a defined set of transcript and service-analysis functions. Exclude individual performance rankings from the reduced release unless separately assessed. Align the product, demonstration, contract and customer instructions with that boundary.

Start with the inputs and the inference

The legal question concerns the function behind the score. Ask which data the model receives, which intermediate features it creates, whose state it assesses and what its output claims to establish.

Article 3(39) defines emotion recognition by reference to emotions or intentions inferred from biometric data. Article 5(1)(f) does not repeat that qualification. The Commission interprets the workplace prohibition consistently with the biometric definition and excludes sentiment analysis based only on written text. This is a material interpretation in non-binding guidance, rather than an express text-only exemption in Article 5. (AI Act, Article 3(34), (39), recitals 18 and 44; Commission Guidelines C(2025) 5052 final, paragraphs 244–245 and 250–252.)

Voice pitch, rhythm or other technically processed vocal characteristics can supply biometric information for this purpose. The AI Act definition does not require identification of the speaker. A model that uses these characteristics to infer enthusiasm, anxiety or emotional distress can therefore fall within the prohibition even without a voice-identification feature.

Ordinary transcription records words. Topic analysis can identify a billing dispute or a request to close an account. A transcript can also record that a speaker said they felt stressed, without claiming independently to measure their emotional state. These functions differ from inferring an employee's internal state from vocal features.

The distinction must survive the whole processing pipeline. Converting audio into a text field labelled “angry”, then asking another model to score that text, does not make the original biometric inference disappear. Inspect acoustic features, annotations, prompts and model outputs, including those hidden from the customer interface.

Physical states also require care. The legislation distinguishes pain and fatigue from emotions; merely detecting a raised voice differs from inferring anger. A score labelled stress cannot be classified from its name alone. Establish whether its target is a physical condition, an emotional state or an attributed statement. (AI Act, recital 18; Commission Guidelines, paragraphs 249 and 265.)

Article 5(1)(f) covers placing the relevant system on the market, putting it into service for the prohibited purpose and using it. If the Moroccan supplier develops or commissions the product and supplies it under its own name, it is the provider. The French employer using it is the deployer. EU market supply and deployment bring those actors within scope; third-country operations can also be covered where their outputs are used in the EU. (AI Act, Articles 2(1)(a)–(c), 3(3)–(4), 5(1)(f).)

A manager's review occurs after the system has made the emotional inference. Describing that output as coaching, engagement or service quality does not change its substance. Employee agreement also supplies no additional exception to Article 5.

The Commission does discuss private call-centre training without sharing with human resources, assessment or promotion consequences, or effects on the employment relationship. That narrow example cannot support the customer's proposed performance ranking. It should not become a coaching label attached to an employer-accessible score. (Commission Guidelines, paragraphs 248 and 254.)

The statutory exception concerns medical or safety reasons. Ordinary wellbeing monitoring, motivation and performance management do not establish it. On the facts described, the supplier has no basis to invoke that exception. (AI Act, Article 5(1)(f); Commission Guidelines, paragraphs 256–263.)

The ban has applied since 2 February 2025. The 2026 amendment did not postpone it. Neither a free operational pilot nor an offshore contract provides a general exemption. (AI Act, Articles 2(8), 3(10), and 113, third paragraph, point (a); Regulation (EU) 2026/1744, Article 1(39)–(40).)

Keep, remove and redesign the functions

The reduced offer should make the following choices explicit:

  • Keep faithful speech transcription, with a way to correct errors and trace a passage to its source. Remove emotional annotations derived from voice characteristics.
  • Keep conversation topics and factual service events, such as an issue raised, information supplied or a transfer requested. Define each output and its operational purpose.
  • Redesign stated sentiment as an attributed statement with context. Report that someone expressed dissatisfaction rather than presenting a measured emotional state.
  • Remove employee enthusiasm, anxiety or emotional-stress scoring derived from voice characteristics, including equivalent outputs available through an API, export or custom prompt.
  • Separate individual employee scoring and ranking from the reduced release. A text-based helpfulness or script-compliance score may still evaluate worker performance, even if it escapes the biometric emotion prohibition.

These are product-design recommendations applying the legal distinctions above. They do not establish that every retained feature is lawful in every deployment. Aggregating prohibited employee emotion scores afterwards does not cure their creation. Removing employees' names also does not make identifiable records anonymous.

The preferred initial scope is service analysis without individual employee evaluation: accurate transcripts, issues discussed and genuinely non-identifying operational summaries. If the customer insists on rankings, assess that additional purpose separately before agreeing to supply it.

Annex III point 4(b) covers AI used to monitor or evaluate workers' performance or behaviour. Its relevance extends beyond decisions to dismiss or promote. Article 6(3) provides limited exceptions subject to its conditions, but an Annex III system that profiles people remains high-risk. A human reviewer is not a general exemption. (AI Act, Articles 3(52), 6(2)–(4), Annex III, point 4(b).)

The timetable also matters. Regulation (EU) 2026/1744 deferred Chapter III, Sections 1–3, apart from Article 6(5), to 2 December 2027 for Annex III systems. Existing systems also require an Article 111(2) transition assessment, including their first market or service date and later significant design changes. These rules preserve the Article 5 prohibition. They do not suspend GDPR or French employment law. (Regulation (EU) 2026/1744, Article 1(39)(a), (40)(b).)

This workforce evaluation question differs from the recruitment uses considered in AI recruitment from Morocco and the conditions for a French launch. The intended use of each retained function needs its own classification.

Transparency also remains a separate issue. An emotion-recognition notice cannot legitimise a prohibited workplace use. Other retained features may create their own disclosure or content-marking duties, depending on their operation and applicable exceptions. The Article 50 transparency guide addresses that distinct assessment. (AI Act, Article 50(1)–(3), (6).)

Establish the French deployment conditions

Removing the prohibited score leaves personal data in recordings, transcripts and identifiable service records. The French customer must establish the purpose and legal basis, limit collection and access, provide the required information and justify retention. A recording lawfully collected for one purpose is not automatically available for employee profiling. The supplier needs appropriate processor terms where it acts on the customer's instructions; any reuse for its own model training requires separate role and purpose analysis. (GDPR, Articles 5–6, 13–14, 28 and 32.)

Assess the need for a data protection impact assessment before processing. Systematic employee evaluation or monitoring can make that mandatory. If a retained scoring process leads to solely automated decisions with legal or similarly significant effects, Article 22 imposes a separate restriction, subject to its exceptions and safeguards. Neither assessment authorises a feature already prohibited by Article 5. (GDPR, Articles 22 and 35–36.)

French employment law requires proportionate measures and prior employee information about data collection and evaluation methods. Evaluation methods must be relevant to their purpose. Where the applicable consultation rules apply, ordinarily in businesses with at least 50 employees, the employer must inform and consult the social and economic committee before deciding to implement means of monitoring employee activity. A notice issued after activation is insufficient. (Code du travail, Articles L1121-1, L1222-3, L1222-4 and L2312-38.)

Set retention by purpose and applicable requirements, rather than adopting one period for every recording and score. Financial-services recording obligations, where applicable to the particular activity, do not themselves authorise emotion analysis or general workforce surveillance. If the customer is a financial entity within the Digital Operational Resilience Act, its information and communication technology procurement and contractual requirements also need assessment; the financial-services description alone does not establish scope. (GDPR, Article 5(1)(b), (e); Directive 2014/65/EU, Article 16(7); Regulation (EU) 2022/2554, Articles 2 and 28–30.)

Address data handling in Morocco as well as France

Map recordings, transcripts, backups, support access and onward disclosures. Morocco has no EU adequacy decision. A transfer to the Moroccan supplier therefore needs an applicable GDPR Chapter V route. Standard contractual clauses require the correct parties and scope, assessment of the destination's laws and practices, and effective supplementary measures where necessary. The 2021 clauses are not the automatic route where the importer is already subject to GDPR for the processing concerned. Hosting in the EU does not eliminate a transfer if a separate Moroccan recipient can remotely access the data. (GDPR, Articles 44–46; Decision (EU) 2021/914, Article 1 and Annex, clauses 14–15.)

Moroccan processing also needs assessment under Law No 09-08 and its implementing rules. Identify the controller, the relevant Moroccan processing operations and any local representative requirement. Document the lawful processing conditions, information and security arrangements, and complete the applicable declaration or authorisation process with the CNDP, Morocco's data protection authority. A French customer or an EU data-processing agreement does not remove Moroccan formalities. (Law No 09-08, Articles 2–5, 12, 21 and 23; Decree No 2-09-165, Articles 23–28.)

Treat disclosures from Morocco to France as a separate transfer question. The CNDP's published adequate-country list includes France, but that does not dispense with the underlying processing requirements or the applicable transfer notification. The route must match the actual controller and data flow. (Law No 09-08, Articles 43–44; Decree No 2-09-165, Articles 46–50; CNDP Deliberation No 236-2015, Article 1.)

Change the product, then align the sale and agreement

The supplier should implement the reduced offer in this order:

1. Record the feature decision. Product and legal owners should list inputs, transformations, outputs, affected people, recipients and intended uses. Resolve what stress and enthusiasm actually mean before approving any score.

2. Remove and test the prohibited function. Engineering should restrict models, prompts, endpoints, exports and integrations that can recreate employee emotion inference within the supplied configuration. Test indirect requests and administrator settings, including relevant languages. Retain the tested version and results.

3. Correct demonstrations and sales material. Remove emotional-scoring claims, screenshots and trial scripts. Demonstrate the approved configuration. The AI Act's intended-purpose definition expressly takes account of promotional and sales materials and statements. (Article 3(12).)

4. Attach a precise feature and use schedule to the agreement. Specify permitted outputs and users, prohibited emotional assessments and integrations, customer configuration limits and change control. Include data roles, processing locations, subprocessors, deletion, audit cooperation and a route to suspend prohibited use. A disclaimer cannot contradict the product being sold.

5. Obtain the customer's deployment evidence. Confirm the intended workflow, information and consultation steps, impact assessment where required, retention decisions and transfer arrangements. Keep any individual performance-evaluation module outside the agreed release until its separate assessment is complete.

The resulting file should connect the released software version to the approved purpose, demonstration, contractual schedule and customer instructions. An AI Act readiness assessment can organise that scope and obligation analysis. The immediate commercial decision remains concrete: remove prohibited employee emotion inference, define the narrower offer and substantiate its deployment conditions before supply.

Plan the product review

Contact Licentium to review the permitted EU use of an AI analytics product and the product and contract changes needed for a proposed French deployment.

Planning a launch somewhere new?

Prefer email? Write to hello@licentium.io. By sending, you accept our privacy policy.